# A Grasping Movement Intention Estimator for Intuitive Control of Assistive Devices

Etienne Moullet, Justin Carpentier, Christine Azevedo Coste, François Bailly

► To cite this version:

Etienne Moullet, Justin Carpentier, Christine Azevedo Coste, François Bailly. A Grasping Movement Intention Estimator for Intuitive Control of Assistive Devices. ICNR 2024 - International Conference on NeuroRehabilitation, Nov 2024, La Granja, Spain. hal-04706628

HAL Id: hal-04706628

https://hal.science/hal-04706628v1

Submitted on 24 Sep 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Grasping Movement Intention Estimator
for Intuitive Control of Assistive Devices

Etienne Moullet, Justin Carpentier, Christine Azevedo-Coste and François Bailly

**Abstract**— This study introduces i-GRIP, an innovative movement goal estimator designed to facilitate the control of assistive devices for grasping tasks in individuals with upper-limb impairments. The algorithm operates within a collaborative control paradigm, eliminating the need for specific user actions apart from naturally moving their hand toward a desired object. i-GRIP analyzes the hand's movement in an object-populated scene to determine its target and select an appropriate grip. In an experimental study involving 11 healthy participants, i-GRIP showed promising goal estimation performances and responsiveness and the potential to facilitate the daily use of assistive devices for individuals with upper-limb impairments in the future.

## I. INTRODUCTION

Upper-limb impairments, such as spinal cord injury, stroke, or amputation, can significantly impact an individual's quality of life and autonomy. These impairments can make it difficult or impossible to perform many daily activities that require grasping, such as eating, dressing, and grooming. Various approaches and assistive devices have been developed to compensate for these difficulties, including functional electrical stimulation (FES), exoskeletons, and prostheses. However, despite continuous improvements in terms of actuation speed, accuracy, and strength, these devices still face major challenges regarding their human interfaces [1]. Indeed, regardless of their control modalities (manipulating joystick/switch buttons, detection of neural activity, muscular activation or stereotypical movements, voice recognition etc. [2], [3]), a most crucial design constraint remains that controlling an assistive device should not hinder the user in the task execution.

In spinal cord injury, this constraint is particularly hard to resolve. Individuals with quadriplegia often have a significantly reduced control over their body to express their intent, and current human-machine interfaces provide limited inputs for controlling devices [3]. Current approaches often rely on state machines to alternatively set a given user's action as the control input for a given movement elicited by the device. However, this requires the user to constantly switch between modes to achieve daily tasks, which can be cognitively demanding, result in saccadic movements, and,

E. Moullet is with WILLOW, INRIA - École Normale Supérieure - PSL Research University, Paris, France (corresponding author, e-mail: etienne.moullet@inria.fr).

J. Carpentier is with WILLOW, INRIA - École Normale Supérieure - PSL Research University, Paris, France.

C. Azevedo-Coste is with CAMIN, INRIA centre d'Université Côte d'Azur, Université de Montpellier, Montpellier, France.

F. Bailly is with CAMIN, INRIA centre d'Université Côte d'Azur, Université de Montpellier, Montpellier, France.

most importantly, impact the user's ability to perform the task effectively.

In this study, we present i-GRIP, a novel movement goal estimator for controlling upper-limb assistive devices during grasping movement. i-GRIP operates within a collaborative control paradigm, requiring no specific action from the users except naturally moving their hand toward the object they wish to grasp. The algorithm performs a kinematic analysis of the hand's movement in the observed scene to identify the targeted object and an appropriate grip to grasp it. This estimated movement goal will be used to control assistive devices in future work.

## II. METHODS AND MATERIALS

### A. i-GRIP algorithm

i-GRIP is designed to work downstream of a scene observation process that provides hands and objects spatial observations. Detected hands and objects are represented in a virtual 3D duplicate scene for analysis (see Fig. 1-b). The algorithm first performs a kinematic analysis of the hands' motions to predict their near-future trajectories. Then, each detected object is treated as a potential target for a given hand. At each time step and new measurement, four confidence scores are computed for each hand-object pair based on metrics that describe the motion of a hand $h$ relatively to the object $o^j$:

* **number of ray impacts:** the number of impacts onto the object's mesh from cones of rays that are cast from near-future trajectory points in the direction of the local velocity vector,

* **distance derivative:** the time derivative of the distance between the hand's position and the position of the center of the object's mesh,

* **distance:** the distance between the hand's position and the object's mesh,

* **future distance:** the distance between the barycenter of near-future trajectory points and the object's mesh.

Next, a velocity-dependent weighted sum of these 4 confidence scores is performed, defining a global confidence score $c_{glob}(h|o^j)$. Then, the target of the hand's movement is identified as the observed object with the highest confidence score:

$$ target(h) = \arg \max_{o^j} (c_{glob}(h|o^j)) \quad (1) $$

Finally, the appropriate grip is determined by the hand's position relative to the detected objects and their shapes. In its current version, i-GRIP focuses on oblong objects that
<figure><img src="image_2.png" /><figcaption>Fig. 1. Experimental setup and pre-processing: (a) - Example of a video frame captured during a trial overlaid with green rectangles marking the detected objects and multicolored landmarks marking hands keypoints. (b) - Corresponding 3D virtual scene: Orange, yellow, red, and green objects are the rendered meshes of the detected objects. Big blue and red spheres represent the 3D positions of, respectively, left and right hands. The middle-sized red and green spheres represent, respectively, the past and expected future trajectory of the right hand. The black lines are a cone of rays expanding from the expected future trajectory of the right hand, and whose impacts are the small magenta dots on the mesh.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2. Bleach bottle from YCB set [4] (left) and a visualization of the corresponding grip selection process (right). Transparent grey volume is the bounding cylinder of the mesh. Red arrow figures the z-axis of the object's frame. The yellow zone illustrates the z-values corresponding to a palmar grip, while the green zone and outwards correspond to a pinch grip.</figcaption></figure>

may be grasped using two main grips used in daily life [5]: palmar and pinch grips. First, the mesh of each detected object $\boldsymbol{o}^j$ is bounded by a cylinder. Its axis of revolution defines the z-axis of the object's reference frame $R^j$ whose origin is placed at the center of gravity of the mesh. Then, a grip for each hand-object pair $grip(\boldsymbol{h}|\boldsymbol{o}^j)$ is determined by the z-component of the hand's position in $R^j$ (see Fig. 2). Finally, the grip appropriate for the analyzed movement is the grip corresponding to the target found with (1):

$$grip(\boldsymbol{h}) = grip(\boldsymbol{h}|\text{target}(\boldsymbol{h})) \quad (2)$$

## B. Experimental study

An experimental study approved by INRIA ethical committee (COERLE Decision 2024-01) involving eleven healthy participants was conducted to evaluate i-GRIP's performance. Participants were seated in front of a table (see Fig. 1-a) and performed 128 grasping movements under homogeneously drawn conditions among the following:

* which hand to use: left or right,

* which objects to target among 4 objects from the YCB set [4] placed on the table: a mustard bottle, a bleach bottle, a tomato can or a box of cheez'it,

* which grip to apply: pinch or palmar,

* whether to execute the grip (actually grasping the object) or simulate it (not moving their fingers during movement nor grasping the object).

Two stereoscopic cameras (OAK-D S2, Luxonis) were placed on the left and right sides of participants at shoulder level and filmed the whole scene (hands and the four objects on the table). RGB frames and depth maps were recorded and processed offline with computer vision tools (that are not part of i-GRIP, but upstream of it) to extract the hands' and objects' observations i-GRIP takes as inputs. Hands 3D positions were estimated using mediapipe [7] and depth maps. Objects 6D poses were estimated using a version of CosyPose [6] trained on the YCB dataset [4]. Video trials for

which hands or object detections were not successful enough were excluded from the study.

## III. RESULTS

i-GRIP successfully identified the target in 89.9% of the recorded movements and selected the correct grip in 94.8% of them. Targets were identified within a mean delay of 0.52s and grips within a mean delay of 0.39s, leaving mean temporal margins before the end of the movement of, respectively, 0.67s and 0.80s.

## IV. CONCLUSION

i-GRIP effectively identified the target of grasping movement and selected appropriate grips within less than half the mean duration of movements, regardless of the experimental conditions (camera placement and movement type). Further studies must assess i-GRIP's effectiveness in real-life scenarios, pathologies, and assistive devices and possibly fine-tune its parameters accordingly.

## REFERENCES

[1] Gantenbein, J., Dittli, J., Meyer, J. T., Gassert, R., & Lambercy, O. (2022). Intention detection strategies for robotic upper-limb orthoses: a scoping review considering usability, daily life application, and user evaluation. Frontiers in neurorobotics, 16, 815693.

[2] Azevedo Coste C, William L, Fonseca L, Hiairrassary A, Andreu D, Geffrier A, ... & Guiraud, D. 2022. Activating effective functional hand movements in individuals with complete tetraplegia through neural stimulation. Scientific Reports, 12(1), 16189.

[3] Jiang N, & Farina D. 2014. Myoelectric control of upper limb pros thesis: current status, challenges and recent advances. Front Neuroeng, 7(4), 7-9.

[4] Calli, B., Singh, A., Walsman, A., Srinivasa, S., Abbeel, P., & Dollar, A. M. (2015, July). The YCB object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR) (pp. 510-517). IEEE.

[5] Feix, T., Romero, J., Schmiedmayer, H. B., Dollar, A. M., & Kragic, D. (2015). The grasp taxonomy of human grasp types. IEEE Transactions on human-machine systems, 46(1), 66-77.

[6] Labbé Y, Carpentier J, Aubry M, Sivic J. 2020. CosyPose: Consistent Multi-view Multi-object 6D Pose Estimation. In: Vedaldi A, Bischof H, Brox T, Frahm JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12362. Springer, Cham.

[7] Zhang F, Bazarevsky V, Vakunov A, Tkachenka A, Sung G, Chang C. L., & Grundmann M. 2020. Mediapipe hands: On-device real-time hand tracking. arXiv preprint arXiv:2006.10214.