# i-GRIP, a Grasping Movement Intention Estimator for Intuitive Control of Assistive Devices

Etienne Moullet, Justin Carpentier, Christine Azevedo Coste, François Bailly

► To cite this version:

Etienne Moullet, Justin Carpentier, Christine Azevedo Coste, François Bailly. i-GRIP, a Grasping Movement Intention Estimator for Intuitive Control of Assistive Devices. Humanoids 2024 - IEEE-RAS International Conference on Humanoid Robots, IEEE-RAS, Nov 2024, Nancy, France. hal-04706577

HAL Id: hal-04706577

https://hal.science/hal-04706577v1

Submitted on 8 Oct 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# I-GRIP, a Grasping Movement Intention Estimator for Intuitive Control of Assistive Devices

Etienne Moullet<fnref n="1" />, Justin Carpentier<fnref n="1" />, Christine Azevedo-Coste<fnref n="2" /> and François Bailly<fnref n="2" />

*Abstract*— This study describes and evaluates i-GRIP, a novel movement intention estimator designed to facilitate the control of assistive devices for grasping tasks in individuals with upper limb impairments. Operating within a collaborative grasping control paradigm, the users naturally move their hand towards an object they wish to grasp and i-GRIP identifies the target of the movement and selects an appropriate grip for the assistive device to perform. In an experimental study involving 11 healthy participants, i-GRIP exhibited promising estimation performances and responsiveness. The proposed approach paves the way towards more intuitive control of grasping assistive device for individuals with upper limb impairments.

## I. INTRODUCTION

Various conditions - such as spinal cord injuries, stroke, or amputation - may hinder upper limb functions. Among them, grasping is crucial for many daily activities, and its impairment significantly impacts an individual's quality of life and autonomy. Various approaches and assistive devices aim to restore this function, including functional electrical stimulation (FES), exoskeletons, and prostheses. However, despite the continuous improvements in terms of actuation speed, accuracy, or strength, major challenges regarding user interfaces and control modalities remain [1].

Regardless of the nature of the actuators, movement assistance is generally actively controlled by the user via dedicated sensors capturing the orders: joystick/switch buttons manipulation, neural activity (e.g., for Brain Computer Interface (BCI)), muscular activation (e.g., for myoelectric control), movements (e.g., using inertial measurements units (IMU)), voice, etc. [2], [3]. The choice of a given solution, driven by the residual functional capabilities of the user, the task to assist, and the device itself, has a major influence on the design of the control scheme of the device [1]. In the endeavor of restoring such a dexterous hand function as grasping, the design process of assistive solutions faces two challenges that are particularly hard to reconcile. On the one hand, the characteristics of objects, such as their shape, weight, and texture, can vary widely, requiring a high degree of flexibility and adaptability from the assistive device. On the other hand, individuals with upper limb disabilities often have a significantly reduced control over their body and thus struggle to control these devices efficiently [3].

Current control approaches often rely on state machine frameworks to accommodate for these conflicting constraints:

the same user control input can trigger different device actions depending on the ongoing task. For example, in the case of trans-humeral amputation, the control of a prosthetic hand may be achieved by myoelectric control. Surface electrodes are placed on the upper arm muscles (the biceps and triceps) to detect their activation. The number of possible muscle contraction combinations being far inferior to the number of degrees of freedom (DOFs) of the prosthesis, the user needs to navigate between predefined modes in which a given muscular activation combination triggers, for instance, either the opening of the fingers or the extension of the wrist [4]. Consequently, besides learning a dictionary of associations between muscle activation combinations and prosthesis action, a user needs to constantly switch between modes to achieve daily tasks. Attempts to circumvent these limitations focus on exploiting compensatory movements to control prostheses [5]. Compensatory movements are intuitively elicited by the user to perform the motor task and interpreted as an input to control the prosthesis. While promising for single joint command (e.g., elbow, wrist), this approach seems difficult to adapt to adaptive tasks such as grasping.

In the context of quadriplegia, hand function may be restored through FES: current pulses are applied via electrodes placed at the skin surface or in an implanted manner to activate peripheral motor nerves and evoke muscle contractions. In this context, control strategies may rely on switch buttons [6], myoelectric control, inertial measurement units (IMU) installed on the contralateral shoulder, or even on voice control [7]. Similarly to prosthesis control, such interfaces are associated to an important cognitive load on the user, in addition to lack of fluidity, resulting in sequential motions due to the state machine approach. Stereo-vision and EMG have been used to drive a hand prosthesis in a semi-autonomous mode [8], but required the user to select the target of his upcoming movement through visual feedback and muscle activation, hence disrupting the control flow. Alternative approaches, such as BCI, aim at directly detecting movement intent without relying on the interpretation of musculoskeletal actions [9]. Nevertheless, still to this day, these technologies often rely on sequential command paradigms or state machines [10] and thus present the same limitations as previously stated. In summary, while upper limb assistive technologies intrinsic performances may be considered satisfactory, existing command modalities often require the user to perform specific, sequential actions to control their device. Alas, these stereotypical motions result at best in an increased cognitive load exerted on the user,

<footnote n="1">¹E. Moullet and J. Carpentier are with WILLOW, INRIA Paris, Département Informatique de l'ENS, Paris, France (corresponding author, e-mail: etienne.moullet@inria.fr).</footnote>

<footnote n="2">²C. Azevedo-Coste and F. Bailly are with CAMIN, INRIA centre d'Université Côte d'Azur, Université de Montpellier, Montpellier, France.</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1. i-GRIP integration in a grasping assistive solution: (A) - in the generic case; (B) - in the demonstrator built for our experiment (see III-A)</figcaption></figure>

and at worst in saccadic movements that may hinder the
realisation of their tasks in daily life.

In this paper, we present i-GRIP, a novel collaborative and
generic grasping movement intention estimator, which adapts
to the user's behavior without requiring any dedicated or
stereotypical action. The contributions of this paper follow
its structure. First, we present the i-GRIP algorithm and its
workflow. Then, we describe a practical implementation of i-
GRIP, coupled with computer vision tools, and an experiment
we ran to characterize its performances. Finally, we discuss
our results and their implications.

## II. I-GRIP ALGORITHM

As previously exposed, a wide range of deficiencies may
affect the upper limbs. A corresponding set of devices and
sensors have been developed to assist with or restore grasping
function. This work aimed to devise a real-time collaborative
generic grasping movement intention estimator.

### A. General design principles

I-GRIP was designed to sit between the sensory and actu-
ation functions to integrate it into various assistive solutions.
From continuous observations of a *scene* (a human upper
body and surrounding objects) i-GRIP gathers information
about the initiation of a grasping movement by the user
and transmits it to a dedicated, downstream process in
charge of controlling an assistive device which terminates
it autonomously (see Fig.1(A)). From these very general
 workflow principles, design choices were made regarding
hands observation, objects observation, i-GRIP's outputs and
general objectives.

First, while real-time upper limb observations may rely
on various sensing techniques, providing different levels of
information [11], [12], hand positions were chosen as the
upper limb observations input of our algorithm. Indeed,
they may be derived from the measurements of diverse
kind of sensors and are thus compatible with different user
impairments. Moreover, such low complexity measurements
were deemed less susceptible to be perturbed by the actions
of the assistive device (as full hand joint poses would be,
for instance). Next, surrounding objects observations inputs

<figure><img src="image_3.png" /><figcaption>Fig. 2. Mustard bottle from YCB set [17] (left) and a visualization of the corresponding grip selection process (right). Transparent grey volume is the bounding cylinder of the mesh. The red arrow figures the z-axis of the object's frame. The blue zone illustrates z-values corresponding to a palmar grip, while the magenta zone and outwards correspond to a pinch grip.</figcaption></figure>

were chosen to be their 6d poses in order to properly leverage
their shapes. Then, in order to accommodate for diverse
assistive devices, i-GRIP output was meant to be high level
information about the initiation of a grasping movement by
the user: the target of its movement and an appropriate
grip to grasp it. Finally, i-GRIP algorithm was conceived
to interpret natural movements from the user, requiring little
to no adaptation. For this purpose, the grip selection process
was developed to be easily understandable by the user and
facilitate the anticipation of the device's actions (see II-H).

### B. Internal workflow overview

I-GRIP is able to track the movements of several hands
at the same time, but their analysis is entirely parallelized
and no bimanual task is taken into account. We thus present
in the following and in Fig. 3 the processing steps for a
single hand. Whenever a new hand or objects observation
is sent to i-GRIP, a full loop of computation is ran. It
first performs kinematic analyzes of the hands motions to
predict their near future trajectories. Next, every detected
object is treated as a potential target and a confidence score
is computed for each hand-object pair. Then, the object
carrying the highest confidence score is identified as the
target of the movement. Finally, the position of the hand
relative to the object and its shape allows for the selection
of an appropriate grip. Fig. 3 illustrates the data flow and
computation steps performed by i-GRIP, which are further
detailed in the following subsections.

Please note that every formula presented in the rest of this
section corresponds to a single loop of i-GRIP. The time
variable was not included for the sake of notations clarity.

### C. Scene observations pre-processing

As previously stated, i-GRIP is designed to work down-
stream of two scene observation processes that provide hands
and objects observations as inputs. More precisely, those
inputs consist in:

* a list of hands detected in the observed scene $\mathcal{H} = \{h^i, i \in \mathbb{N}\}$, embedding their 3d positions $p^i$ and measurement timestamps $t^i$,

* a list of objects detected in the observed scene $\mathcal{O} = \{\mathcal{o}^j, j \in \mathbb{N}\}$, embedding their 6d poses (3d positions
<figure><img src="image_4.png" /><figcaption>Fig. 3. Data workflow and computations inside i-GRIP for one hand and two objects.</figcaption></figure>

$p^j$ and 3d orientations $\theta^j$) as well as their 3d mesh models $M^j$ and measurement timestamps $t^j$.

In a preliminary step, $p^i$, $p^j$ and $\theta^j$ are filtered using speed-based low-pass filter, in order to mitigate the impact of the noise in the received measurements. Then, a kinematic analysis of hands' movements is performed. We keep track of the recent past trajectory of a hand $h^i$ in a sliding window $P^i = [(p^i_{-k}, \tau_k^i)]$, $k \in [0, n_f]$, where $\tau_k = t^i - t^i_k$ is the time elapsed between the last available measurement and the measurement of $p^i_{-k}$, and where $n_f$ is a fixed size adapted to the frequency of acquisition of inputs. We next perform a weighted second order polynomial regression on this window, with weights defined as

$$w_k = \frac{1}{1 + \tau_k} \qquad (1)$$

Then, we compute a list of future trajectory points $P^i_{future}$ by extrapolating the computed polynomial to near-future timestamps. These variables are used to achieve target identification and grip selection.

### D. Object metrics evaluation

A 3D virtual scene (see Fig. 4-B) incorporates the processed hand $h^i$ as a 3D point $p^i$ and each processed object $\mathbf{o}^j$ as its mesh $M^j$ placed according to its pose $(p^j, \theta^j)$. For a given hand $h^i \in \mathcal{H}$, every object $\mathbf{o}^j \in \mathcal{O}$ detected in the scene is treated as a potential target and processed separately. For each pair $(h^i|\mathbf{o}^j)$, four different metrics are computed:

* **ray impacts number:** from predicted future points $P^i_{future}$ we cast cones of rays whose axes are co-linear to the local velocity vector, and whose angles depend on the local scalar velocity (see Fig. 4). The metric $n^i_{impacts}$ then consists in the number of rays that intercept the object's mesh.

* **distance derivative:** we define $d^i_{center} = \|p^i - c^j\|$ as the distance between the hand's position and the position of the center of the object's mesh $c^j$. We define this metric as its time derivative $d^i_{der} = \frac{d}{dt} d^i_{center}$

* **distance:** we define this metric as the distance between the hand's position and the closest node on the object's mesh $d^i_{mesh} = \min_{n^i_l \in M^j} (\|p^i - n^j_l\|)$.

* **future distance:** we denote $p^i_{future,m}$ as the barycenter of the predicted future points $P^i_{future}$. We define this metric as its distance to the object's mesh $d^i_f = \min_{n^i_l \in M^j} (\|p^i_{f,m} - n^j_l\|)$.

### E. Object metrics confidence scores

At this step of the algorithm, we put the metrics defined above back in perspective of the whole scene: taking into account the hand's absolute motion and the other objects' positions, we compute four confidence scores that evaluate the likelihood of an object to be the target of the movement according to the corresponding metric.

* **ray impacts number:** we denote $n^i_{tot}$ as the total amount of impacts over all detected objects and define the impacts confidence score as

$$c_{impacts}(h^i|\mathbf{o}^j) = \frac{n^i_{impacts}}{n^i_{tot}} \qquad (2)$$

* **distance derivative:** we denote $v^i$ as the hand's scalar velocity and define the distance derivative confidence score as

$$c_{d\_der}(h^i|\mathbf{o}^j) = \frac{d^i_{der}}{v^i} \qquad (3)$$

* **distance:** we denote $d_{max}$ as the maximum distance between all pairs of potential targets, and define the distance confidence score as

$$c_{dist}(h^i|\mathbf{o}^j) = 1 - \max \left( \frac{d^i_{mesh}}{d_{max}}, 1 \right) \qquad (4)$$

* **future distance:** similarly to (4), we define the corresponding confidence score as

$$c_{fut\_dist}(h^i|\mathbf{o}^j) = 1 - \max \left( \frac{d^i_f}{d_{max}}, 1 \right) \qquad (5)$$
<figure><img src="image_5.png" /><figcaption>Fig. 4. Recorded scene from the experiment and its analysis by i-GRIP: (A) - Example of a video frame captured during a trial overlaid with green rectangles marking the detected objects and multicolored landmarks marking hands key-points. (B) - Corresponding 3D virtual scene: Orange, yellow, red, and green objects are the rendered meshes of the detected objects. Big blue and red spheres represent the 3D positions of, respectively, left and right hands. The middle-sized red and green spheres represent, respectively, the past and expected future trajectory of the right hand. Black lines are a cones of rays expanding from the hands' expected future trajectories, whose impacts on the objects' meshes are the small magenta dots.</figcaption></figure>

### F. Object global confidence score

We assign to each metric $m \in M = \{\text{impacts}, \text{d\_der}, \text{dist}, \text{fut\_dist}\}$ a corresponding weight $w_m$. We define variable weights $w_0$ and $w_1$, denoting that they apply respectively to zero and first order derivative metrics, so that $w_{\text{distance}} = w_{\text{future\_distance}} = w_0$ and $w_{\text{impacts}} = w_{\text{distance\_derivative}} = w_1$. We use the fact that during reaching tasks human hand's scalar velocity consistently follows a well-known "bell-shaped curve" [13] (see Fig. 6(A) for an illustration from our data). We detect whether the hand is before velocity peak or after by comparing the current scalar velocity to the recent-past maximum scalar velocity $v_{\text{max\_win}} = \max(V^i)$ where $V^i = [v^i_{-k}]$, $k \in [0, n_f]$ is a sliding window of fixed size $n_f$ of the hand's scalar velocity. We compute $w_0$ as

$$w_0 = \frac{v^i}{v_{\text{max\_win}}} \qquad (6)$$

and $w_1$ as

$$w_1 = \begin{cases} 2 & \text{if } v^i > v_{\text{max\_win}} \\ 1 & \text{if } v^i \le v_{\text{max\_win}} \end{cases} \qquad (7)$$

We then compute a global target confidence as the weighted sum of these metrics confidences:

$$c_{\text{glob}}(\mathbf{h}^i | \mathbf{o}^j) = \frac{\sum_{m \in M} w_m c_m(\mathbf{h}^i | \mathbf{o}^j)}{\sum_{m \in M} w_m} \qquad (8)$$

### G. Target identification

When the previously described steps have been performed for every detected object $\mathbf{o}^j \in \mathcal{O}$, we synthesize and compare their respective global confidences. The most probable target for the motion of hand $\mathbf{h}^i$ is then defined as the potential target with highest global confidence:

$$\text{target}(\mathbf{h}^i) = \arg \max_{\mathbf{o}^j} (c_{\text{glob}}(\mathbf{h}^i | \mathbf{o}^j)) \qquad (9)$$

For the sake of comparison (see IV), we also define target as it would be found considering each metric separately for $m \in M$:

$$\text{target}_m(\mathbf{h}^i) = \arg \max_{\mathbf{o}^j} (c_m(\mathbf{h}^i | \mathbf{o}^j)) \qquad (10)$$

### H. Grip selection

In its current version, i-GRIP is focused on oblong objects, that may be grasped using two of the main grips used in daily life [14]: palmar and pinch grips. For each object, we define a reference frame $R^j$ whose origin is the center of gravity of its mesh, and oriented such as the z-axis matches the axis of the bounding cylinder of the mesh. We denote $L^j$ the length of this cylinder. During the movement of a hand $\mathbf{h}^i$ towards an object $\mathbf{o}^j$, its position is expressed in the object's frame as $\mathbf{p}^{i,j}$, and its z-component is compared to $L^j$ to determine the appropriate grip as

$$\text{grip}(\mathbf{h}^i | \mathbf{o}^j) = \begin{cases} \text{pinch} & \text{if } |z^{i,j}| \ge 0.85 * L^j \\ \text{palmar} & \text{if } |z^{i,j}| < 0.85 * L^j \end{cases} \qquad (11)$$

Fig. 2 gives a visualization of this selection process. The most probable appropriate grip for a given hand is then defined as the grip corresponding to the most probable target in (9):

$$\text{grip}(\mathbf{h}^i) = \text{grip}(\mathbf{h}^i | \text{target}(\mathbf{h}^i)) \qquad (12)$$

## III. EXPERIMENTAL EVALUATION

As a reminder (see II and Fig. 1(A)), i-GRIP is not a controller, but rather provides target and grip information to a downstream process in charge of controlling an assistive device. Thus, the assessment of i-GRIP was restricted to its outputs and an experiment was conducted to gather a dataset of hand trajectories and their corresponding "ground truths", consisting in the targets the movements were directed to and the grips that were employed.

### A. Observation processes experimental implementation

Even though no assistive device was included in the experiment, i-GRIP is not a standalone solution and expects hands positions and objects poses as inputs. An implementation of the observing processes presented in Fig. 1(A) were thus required to build a demonstrator and evaluate i-GRIP. In order to approach real-life conditions and even though data analysis would be done offline, only real-time solutions were considered for data acquisition. Prioritizing cost, ease of use, installation and universality, stereoscopic RGB-d cameras (OAK-D S2, Luxonis) were chosen as the primary observation system.

In coherence with the chosen sensors, Mediapipe [15] and CosyPose [16] were used to extract respectively the positions of hands and poses of objects present in the filmed scene. Mediapipe [15], was used on RGB frames to get 2D hand landmarks. Depth maps were then leveraged to compute 3D coordinates of each landmark. Finally, hands 3D positions were measured at the center of the palm. CosyPose [16] was fed RGB frames and allowed to detect objects and estimate their 6D poses. The version that was used was specially
<figure><img src="image_6.png" /><figcaption>Fig. 5. Processing success rates for each participant under different conditions. Rows (A), (B) and (C) show respectively video trials validation, target detection and grip identification success rates. Columns (i), (ii) and (iii) correspond respectively to all trials, trials regrouped according to relative camera placements and trials regrouped according to grasping movement types. Horizontal colored lines represent the average rates over all trials for a given condition.</figcaption></figure>

trained on the YCB set [17], from which objects were picked
for the experiment (see III-B). Fig. 1(B) illustrates the data
 workflow of the built demonstrator.

B. *Experimental setup and procedure*

An experimental study approved by INRIA ethical
committee (COERLE Decision 2024-01) involving eleven
healthy participants was conducted to evaluate i-GRIP al-
gorithm performances. Participants were seated in front of
a table. Four daily life objects were selected from the YCB
set [17]: a mustard bottle, a bleach bottle, a tomato can or
a box of cheez'it. They were randomly placed between the
participant's hands resting position and a computer screen
on which were displayed instructions (see Fig. 4(A)). These
instructions informed the participant on the four settings that
characterized the upcoming trial he had to perform:

* *hand*: whether to use the left or right hand,

* *target*: which object to aim at among the 4 objects on the table

* *grip*: whether to apply a pinch or palmar grip,

* *movement type*: whether to execute the grip (actually grasping the object) or simulate it (approaching the object without moving their fingers during movement or grasping the object).

Four repetitions of each possible combination of these set-
tings, leading to 128 trials, were randomly shuffled for each
participant. Two RGB-d cameras (OAK-D S2, Luxonis) were
placed on the left and right sides of the participant, at
shoulder level, and filmed both of their hands and every
object on the table at 30 frames per second (FPS). The two

resulting points of view doubled the number of video trials
to analyse.

C. *Recordings pre-processing*

Recordings were cut to match the beginning and end of
each trial, checked to exclude trials where participants didn't
perform the required combination.

Preliminary testing of the observation tools described in
III-A exhibited inconsistent performances on videos and
that hands and/or objects detections were not successful on
all frames. Being based on a kinematic properties, i-GRIP
requires a minimal temporal continuity in the inputs it is fed
to allow for a relevant analysis. Trials pre-processings results
were thus analysed to verify if detections were consistent
enough throughout their videos. A first validation condi-
tion verified if hands and objects first detections happened
early enough during the trial. A second validation condition
verified if, once detected, hands and objects detections did
not fail for too many consecutive frames. Trials that met
both conditions were deemed valid, while the others were
excluded from the following analysis.

D. *Trial evaluation*

The i-GRIP workflow (see II) was applied to each recorded
RGB-d frame, providing the corresponding target and grip
predictions sequences, that were compared to the ground
truth of the trial instructions.

In order to evaluate target identifications throughout a
whole valid video trial, *k*<sub>target</sub> denoted the index of the first
frame of the longest sequence of frames for which target
identification was successful, and $k_f$ denoted the index of the end of movement frame. $n_{target}$ denoted the amount of frames between $k_{target}$ and $k_f$ for which target identification was successful and the success ratio $r_{target} = \frac{n_{target}}{k_f - k_{target}}$ was computed. A trial was deemed successful in the regard of target identification if $r_{target} > 70\%$.

Similarly to target identification, $k_{grip}$ identified the index of the first frame of longest sequence of frames for which target identification was successful, and $k_f$, the index of the end of movement frame. $n_{grip}$ denoted the amount of frames between $k_{grip}$ and $k_f$ for which target identification was successful and the success ratio $r_{grip} = \frac{n_{grip}}{k_f - k_{grip}}$ was computed. A trial was deemed successful in the regard of grip selection if $r_{grip} > 70\%$.

Temporal delay and margin were computed respectively as the time elapsed between the movement onset and the target detection, and between the target detection and the end of the movement. Similar computation was performed for grip selection.

## IV. RESULTS

Hands and objects observation processes upstream of i-GRIP were applied on a total of 74378 frames and were fully successful on 66499 of them (89.4%). One participant could not pursue the experiment until the end, and the data of four trials were corrupted, resulting in a total of 1359 performed trials. Out of the corresponding 2718 recorded video trials, 1553 were deemed valid (Sec. III-C). Row A in Fig. 5 shows the validation rates across participants for all trials under different conditions. All results displayed in the following subsections only relate to valid trials. Fig. 6 shows the temporal evolution of i-GRIP's metrics for a valid and representative trial.

### A. Target identification

Target identification following the global confidence defined in (9) was successful on 61.9% of the frames of valid video trials. Frame-by-frame success rates of individual metrics defined in (2), (3), (4) and (5) are given in the first column of Table I.

Target was successfully identified on 89.9% of all valid video trials. Trial success rates of the individual metrics are given in the second column of Table I. When computed for each participant over every recording devices, the average success rate was 90.1% with a standard deviation of 7.5% (see Fig. 5(B) for the target identification success rate across participants and different experimental conditions). Video trial target identification success rate reached:

* 90.5% with the camera placed on the ipsilateral side of the moving hand, and 89.4% with the one placed on the contralateral side,

* 92.9% when the grip was executed at the end of the movement, and 86.8% when it was not.

Targets were correctly identified within an average delay of 0.52s, resulting in an average temporal margin until the end of the movements of 0.67s. Table I gives the target

<figure><img src="image_7.png" /><figcaption>Fig. 6. Temporal data of a typical video trial from participant P2, recorded from the camera in contralateral position. The task was to reach for the tomato can with their right hand and execute a palmar grip. (A) shows the hand's velocity profile. (B) to (F) show the temporal evolution of the target metrics confidences defined in (2), (3), (4), (5) and (8). Black arrows show the moments where each metric found the correct target. (G) and (H) illustrate the resulting identified target and selected grip over time.</figcaption></figure>
detection delays and margins for every confidence in the third and fourth columns respectively.

## B. Grip selection

Grip was successfully selected on 94.8% of all video trials. When computed for each participant over every recording device, the average success rate was 94.5% with a standard deviation of 4.3% (see Fig. 5(C) for the grip selection success rate across participants and different experimental conditions). Grip selection success rate reached:

* 95.5% with the camera placed on the ipsilateral side of the moving hand, and 94.2% with the one placed on the contralateral side,

* 94.7% when the grasp was executed at the end of the movement, and 94.9% when it was not.

Grips were correctly selected within an average delay of 0.39s, resulting in an average temporal margin until the end of the movements of 0.80s.

## V. DISCUSSION

In this paper, we presented i-GRIP, an algorithm conceived to estimate grasping movement intention. I-GRIP was designed to be integrated into a wide variety of assistive solutions and apparatuses (see Fig. 1). Its expected inputs were limited to estimations of the positions of hands and poses of objects observed in a monitored scene. Its outputs, to be transmitted to a controlled upper limb assistive device, consisted in the identification of the object targeted by an ongoing movement and an appropriate grip to grasp it. As i-GRIP treats hands separately, we described the computational loop workflow for a single hand, and showed its illustration for two objects in Fig. 3. First, a kinematic analysis of a hand's absolute trajectory was performed, to predict its near future motion. Then, every detected object was treated as a potential target, and an analysis of the hand's motion relative to each of them was conducted: four metrics and their corresponding confidence scores were computed in (2), (3), (4) and (5), and were then synthesized into a global confidence score in (8), evaluating the overall likelihood of an object being the target of the movement. In the meantime, the shape of each object was leveraged to select an appropriate grip according to the hand's position relative to them. Finally, the target output was identified as the object presenting the highest global confidence score in (9) and the grip output was set as the grip embedded by the target output in (12).

<table><caption>TABLE I
METRICS TARGET IDENTIFICATION ACCURACY (SUCCESS RATES FOR SINGLE FRAMES AND WHOLE TRIALS) AND REACTIVITY (DELAYS AND MARGINS OVER TRIALS)</caption><thead><tr><td>Metric</td><td>Frames</td><td>Trials</td><td>Delay (s)</td><td>Margin (s)</td></tr></thead><tbody><tr><td>Global</td><td>61.9%</td><td>89.9%</td><td>0.52</td><td>0.67</td></tr><tr><td>Distance derivative</td><td>56.8%</td><td>72.0%</td><td>0.48</td><td>0.72</td></tr><tr><td>Impacts</td><td>15.7%</td><td>34.7%</td><td>0.97</td><td>0.25</td></tr><tr><td>Distance</td><td>53.6%</td><td>88.2%</td><td>0.59</td><td>0.60</td></tr><tr><td>Future distance</td><td>59.6%</td><td>88.7%</td><td>0.52</td><td>0.58</td></tr></tbody></table>

## A. Overall performances

I-GRIP's success rates for target identification and grip selection over trials, computed over all participants, trial settings and recording devices, reached respectively 89.9% and 94.8%. The success rates presented for each participant in Fig. 5(B,i) and (C,i) and their reported corresponding mean values and standard deviations show inter-participant variability, suggesting that improvement could be achieved in hand-tailoring i-GRIP specifically for a user.

Additionally, it may be noticed in Fig. 5 and the reported results that the grip selection success rate is greater than the target identification success rate for each experimental condition, which is also the case with all video trials combined. Similarly, reported average delays and margins computed over all participants, trial settings and recording devices, show that grip selection was quicker than target identification. This means that while the target was misidentified, the grip associated to it was correct which may be explained by the similar oblong shape the objects shared. As a consequence, adaptations might be necessary when expanding i-GRIP to different shapes or grip types.

## B. I-GRIP workflow analysis

Fig. 6 shows the evolution during a typical trial of the confidence scores in (2), (3), (4), (5) and (8) for the four detected objects, as well as i-GRIP's corresponding outputs. One can notice that every metric confidence scores fails to identify the target at the beginning of the movement. In fact, Table I shows that they are only able to identify a target after some delay (see arrows in Fig. 6 for a visualization in a single trial). This can be explained by the fact that they all rely on the observation of the hand's trajectory, and cannot anticipate the user's intent before it is translated into a distinguishable motion. Yet, one can notice that the metrics complement each other: "velocity-related" metrics (i.e impacts and distance derivative) tend to react faster than "distance-related" metrics (i.e distance and future distance), but the latter takeover in terms of confidence toward the end of the movement.

Moreover, interestingly, as shown in Table I, individual metrics under-performed the global confidence, both frame-by-frame and for video trials. This means that while an object might not bear the maximum metric confidence score, the combination of confidence scores into the global one allows i-GRIP to identify it as the correct target. Additionally, Table I illustrates that the global confidence achieves timings close to the best performing metric on this matter (i.e distance derivative), all while being superior in terms of success rates in target identification and grip selection.

## C. Limitations

The employed object detection tool was constrained to a given set of known objects (see III-A) and its limited computation speed narrowed this study to the grasping of immobile objects. Moreover, observation tools inconsistent efficiency (see III-C) led to the exclusion of a significant amount of trials, which may raise concerns about the handling of failed detections in real-life implementation. Nevertheless, one may
be confident that the computer vision research field, being actively driven by robotics, will soon provide tools enhanced in universality, speed and accuracy. In addition, fine tuning hand detection on a user would be relevant in terms of specificity and reliability. In any case, even if these issues are related to processes upstream of i-GRIP, adapted safeguards would need to be implemented to prevent any action if hands are lost for a too long period.

This experiment was meant as a proof of concept for i-GRIP and was, as such, restricted to able-bodied participants. While the simulated movement type (see III-B) was meant to mimic impaired motion, no specific instructions were given regarding movements velocity. Fig. 5(iii) shows that i-GRIP performed slightly worse on simulated grasping for target identification but equally for grip selection, giving hope in its applicability for impaired users. Moreover, it is most likely that impaired users would perform slower movements, suggesting that i-GRIP's margins would be greater than those reported in Table I, giving time to an assistive device to trigger its actions during motion. Yet, these very actions could possibly result in different kinematics than the ones analysed in this work [18], and further studies involving impaired participants using real assistive devices will be required.

Finally, this work didn't set any confidence threshold on target selection from which the device grasping action would be triggered. Setting this value, along with tuning other i-GRIP's parameters would allow for a personalized compromise between reactivity and reliability, according to a user's preferences, pathology and assistive device.

## D. Additional observations

As previously stated, the used computer vision tools exhibited inconsistent performances. Although it was not the main focus of this study, in the absence of a straightforward argument about the homogeneity of these inconsistencies in function of camera placement, its influence on trial validation was evaluated. Video trial validation rates presented in Fig. 5(A) show that a device placement contralateral to the moving hand seems to lead to slightly more efficient hand and object detections throughout trials. Nonetheless, these results suggest that the video trials selected for i-GRIP's evaluation were overall homogeneously representative of the trials diversity.

The reported overall success rates for target identification and grip selection show little to no influence of camera placement over i-GRIP performances. This suggests that i-GRIP may be fed with inputs from both hands extracted from a single recording device, which can be placed arbitrarily according to any other design criterion.

## VI. CONCLUSIONS

This paper stands as a proof of concept for a collaborative and generic grasping movement intention estimator. Further studies will be needed to assess i-GRIP in real-life use cases with users with upper limb motor deficiencies and for diverse conditions and assistive devices. Although our goal was to

keep i-GRIP as generic as possible, its many parameters could be fine-tuned to enhance and tailor its performances to specific situations and users' preferences.

## REFERENCES

[1] Gantenbein, J., Dittli, J., Meyer, J. T., Gassert, R., & Lambercy, O. (2022). Intention detection strategies for robotic upper-limb orthoses: a scoping review considering usability, daily life application, and user evaluation. Frontiers in neurorobotics, 16, 815693.

[2] Azevedo Coste C, William L, Fonseca L, Hiairrassary A, Andreu D, Geffrier A, ... & Guiraud, D. 2022. Activating effective functional hand movements in individuals with complete tetraplegia through neural stimulation. Scientific Reports, 12(1), 16189.

[3] Jiang N, & Farina D. 2014. Myoelectric control of upper-limb prosthesis: current status, challenges and recent advances. Front Neuroeng, 7(4), 7-9.

[4] Light, C. M., Chappell, P. H., Hudgins, B., & Engelhart, K. (2002). Intelligent multifunction myoelectric control of hand prostheses. Journal of medical engineering & technology, 26(4), 139-146.

[5] Legrand, M., Jarrassé, N., Marchand, C., Richer, F., Touillet, A., Martinet, N., ... & Morel, G. (2022). Controlling upper-limb prostheses with body compensations. In Wearable Robotics: Challenges and Trends: Proceedings of the 5th International Symposium on Wearable Robotics, WeRob2020, and of WearRAcon Europe 2020, October 13–16, 2020 (pp. 101-106). Springer International Publishing.

[6] Trotobas, C., Ferreira, F. M. R. M., de Faria Moraes, M. R., Valladao, A. M., Martins, H. R., Fattal, C., & Azevedo Coste, C. A. (2024). Combining Functional Electrical Stimulation (FES) to Elicit Hand Movements and a Mechanical Orthosis to Passively Maintain Wrist and Fingers Position in Individuals With Tetraplegia: A Feasibility Test. IEEE Transactions on Medical Robotics and Bionics.

[7] Tigra, W., Navarro, B., Cherubini, A., Gorron, X., Gelis, A., Fattal, C., ... & Azevedo Coste, C. A. (2016). A novel EMG interface for individuals with tetraplegia to pilot robot hand grasping. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 26(2), 291-298.

[8] Markovic, M., Dosen, S., Cipriani, C., Popovic, D., & Farina, D. (2014). Stereovision and augmented reality for closed-loop control of grasping in hand prostheses. Journal of neural engineering, 11(4), 046001.

[9] Nicolas-Alonso, L. F., & Gomez-Gil, J. (2012). Brain computer interfaces: a review. Sensors, 12(2), 1211-1279.

[10] Padfield, N., Camilleri, K., Camilleri, T., Fabri, S., & Bugeja, M. (2022). A comprehensive review of endogenous EEG-based BCIs for dynamic device control. Sensors, 22(15), 5802.

[11] Bailly, F., Ceglia, A., Michaud, B., Rouleau, D. M., & Begon, M. (2021). Real-time and dynamically consistent estimation of muscle forces using a moving horizon emg-marker tracking algorithm—application to upper limb biomechanics. Frontiers in bioengineering and biotechnology, 9, 642742.

[12] Yahya, M., Shah, J. A., Kadir, K. A., Yusof, Z. M., Khan, S., & Warsi, A. (2019). Motion capture sensing techniques used in human upper limb motion: A review. Sensor Review, 39(4), 504-511.

[13] Flash, T., & Hogan, N. (1985). The coordination of arm movements: an experimentally confirmed mathematical model. Journal of neuroscience, 5(7), 1688-1703.

[14] Feix, T., Romero, J., Schmiedmayer, H. B., Dollar, A. M., & Kragic, D. (2015). The grasp taxonomy of human grasp types. IEEE Transactions on human-machine systems, 46(1), 66-77.

[15] Zhang F, Bazarevsky V, Vakunov A, Tkachenka A, Sung G, Chang C. L., & Grundmann M. 2020. Mediapipe hands: On-device real-time hand tracking. arXiv preprint arXiv:2006.10214.

[16] Labbé Y, Carpentier J, Aubry M, Sivic J. 2020. CosyPose: Consistent Multi-view Multi-object 6D Pose Estimation. In: Vedaldi A, Bischof H, Brox T, Frahm JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12362. Springer, Cham.

[17] Calli, B., Singh, A., Walsman, A., Srinivasa, S., Abbeel, P., & Dollar, A. M. (2015, July). The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR) (pp. 510-517). IEEE.

[18] Moulet, E., Roby-Brami, A., & Guigon, E. (2022). What is the nature of motor adaptation to dynamic perturbations?. PLOS Computational Biology, 18(8), e1010470.