# Depth-sensor-based shared control assistance for mobility and object manipulation: toward long-term home-use of BCI-controlled assistive robotic devices

Angélina Bellicha, Lucas Struber, François Pasteau, Violaine Juillard, Louise

Devigne, Serpil Karakas, Stephan Chabardes, Marie Babel, Guillaume

Charvet

► To cite this version:

Angélina Bellicha, Lucas Struber, François Pasteau, Violaine Juillard, Louise Devigne, et al. Depth-sensor-based shared control assistance for mobility and object manipulation: toward long-term home-use of BCI-controlled assistive robotic devices. Journal of Neural Engineering, 2025, 22 (1), pp.016045. 10.1088/1741-2552/adae36. hal-04972378

HAL Id: hal-04972378

https://hal.science/hal-04972378v1

Submitted on 3 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

CrossMark

OPEN ACCESS

RECEIVED

8 August 2024

REVISED

9 January 2025

ACCEPTED FOR PUBLICATION

24 January 2025

PUBLISHED

14 February 2025

Original Content from
this work may be used
under the terms of the
Creative Commons
Attribution 4.0 licence.

Any further distribution
of this work must
maintain attribution to
the author(s) and the title
of the work, journal
citation and DOI.

<figure><img src="image_3.png" /></figure>

PAPER

# Depth-sensor-based shared control assistance for mobility and object manipulation: toward long-term home-use of BCI-controlled assistive robotic devices

Angélina Bellicha<fnref n="1" />,*, Lucas Struber<fnref n="1" />,*, François Pasteau<fnref n="2" />, Violaine Juillard<fnref n="1" />, Louise Devigne<fnref n="2" /> ID, Serpil Karakas<fnref n="1" />, Stephan Chabardes<fnref n="1" />,<fnref n="3" />, Marie Babel<fnref n="2" /> ID and Guillaume Charvet<fnref n="1" /> ID

E-mail: angelinabellicha@hotmail.fr and lucas.struber@cea.fr

**Keywords:** shared control, BCI, reach-and-grasp, wheelchair driving, clinical evaluation

Supplementary material for this article is available [online](#)

## Abstract

**Objective.** Assistive robots can be developed to restore or provide more autonomy for individuals with motor impairments. In particular, power wheelchairs can compensate lower-limb impairments, while robotic manipulators can compensate upper-limbs impairments. Recent studies have shown that Brain-Computer Interfaces (BCI) can be used to operate this type of devices. However, activities of daily living and long-term use in real-life contexts such as home require robustness and adaptability to complex, changing and cluttered environments which can be problematic given the neural signals that do not always allow a safe and efficient use. This article describes assist-as-needed sensor-based shared control (SC) methods relying on the blending of BCI and depth-sensor-based control. **Approach.** The proposed assistance targets the BCI-teleoperation of effectors for tasks that answer mobility and manipulation needs in a at-home context. The assistance provided by the proposed methods was evaluated through a wheelchair mobility and reach-and-grasp laboratory-based experiments in a controlled environment, as part of a clinical trial with a quadriplegic patient implanted with a wireless 64-channel ElectroCorticoGram recording implant named WIMAGINE. **Main results.** Results showed that the proposed methods can assist BCI users in both tasks. Indeed, the time to perform the tasks and the number of changes of mental tasks were reduced. Moreover, unwanted actions, such as wheelchair collisions with the environment, and gripper opening that could result in the fall of the object were avoided. **Significance.** The proposed methods are steps toward at-home use of BCI-teleoperated assistive robots. Indeed, the proposed SC methods improved the performance of the two assistive devices.

Clinical trial, registration number: NCT02550522.

## 1. Introduction

The number of individuals suffering from chronic motor disabilities due to spinal cord injuries is around 5 million worldwide [1]. Assistive technologies aim to provide technological solution to contribute to the restoration or increase of the autonomy of individuals with disabilities. In particular, individuals facing important motor deficits such as people with quadriplegia can use mobility assistance devices such as

power wheelchairs or upper-limb assistive robots. These devices requires the user to initiate the task by mobilizing sufficient capacity to control the device in a secure and acceptable manner. However, despite the numerous adaptation alternatives for giving control to the user, individuals with severe impairments such as people with quadriplegia generally experience major cognitive challenges and fatigue in using these interfaces that rely on a complex and burdensome manual control. Furthermore, these devices are

<footnote n="1">¹ University Grenoble Alpes, CEA, Leti, Climatec, Grenoble F-38000, France</footnote>

<footnote n="2">² University Rennes, INSA Rennes, Inria, CNRS, IRISA - UMR 6074, Rennes F-35000, France</footnote>

<footnote n="3">³ University Grenoble Alpes, CHU Grenoble Alpes, Climatec, Grenoble F-38000, France</footnote>

<footnote>* Authors to whom any correspondence should be addressed.</footnote>
targeted to be used on a long-term basis in an uncon-
trolled environment such as home. Indeed, the home
can be a cluttered environment with changing objects
and people. Thus, to be compatible with long term
home use, an assistive robot must: (1) be easy to use
and acceptable by the user, (2) be able to navigate
in cluttered, unknown and changing environments
where some objects are to be manipulated while oth-
ers need to be avoided, and (3) ensure upholding of
strict safety measures to avoid harm to people and to
the device.

In the last decades, the use of Brain-Computer
Interfaces (BCI) has shown promise to empower
people with physical impairments to perform
Activities of Daily Living (ADLs). However, oper-
ating robotic assistive devices through BCI control
remains challenging, as it requires a precise control
over a high number of degrees a freedom (DoF)
with short delays, and is sometimes associated to a
high cognitive load [2] implying fatigue for the user
and the impossibility or difficulty to perform con-
current tasks. Using non-invasive recordings, only
Steady-State Visual Evoked Potentials (SSVEP) BCI
showed sufficient performances in terms of decoding
and DoF to safely control robotic assistive devices
[3, 4], motor imagery BCI being restricted to 1 or
2 DoF with limited performances [5, 6]. However,
SSVEP requires a screen for the user to focus on
which renders it complicated or impossible to use in
interaction with the environment. Furthermore, in
terms of practical use, EEG devices are usually com-
plicated and long to setup requiring exterior assist-
ance. In order to control enough DoF, implantable
recording devices such as microelectrodes arrays or
Electrocorticography (ECoG) implants can be used
[7, 8]. While the use of microelectrodes arrays allow
precise control over a high number of DoF, a long-
term use is usually impossible given the stability of
signals and models that require regular updates and
because electrodes become unusable after a short
period of time [9]. On the contrary, ECoG implant
showed their ability to provide stable decoding mod-
els over long periods of time with a high number of
DoF [10].

Shared control (SC) is a control method used in
various robotic applications in which a user and a
robotic system collaborate. In assistive robotics, con-
trol is generally shared between the user command
(retrieved from a control interface) with a sensor-
based control relying on features detected by the
sensors equipped on the robot [11]. The use of SC
methods in assistive robotics is critical to ensure the
acceptability of devices as it allows a user in-the-loop
control. Studies have shown that devices that require
the participation of the user are more accepted than
devices that do not [12]. SC applied to an assistive
robot can provide task assistance as well as safety with
respect to the surrounding environment.

In the context of wheelchair driving assistance,
SC has already been implemented to provide collision
avoidance navigation assistance [13].

SC methods can be used to assist reach-and-grasp
tasks. The sensor-based autonomous control relies on
the identification of the intent of the user in the envir-
onment captured by sensors. In the literature, sensor-
based SC methods applied to reach-and-grasp assist-
ance usually rely on RGB-D cameras rigidly placed
in the environment [14–16]. The use of a camera
fixed in the environment poses several issues e.g. in
case of mobile robotic manipulators, e.g. reach-and-
grasp manipulators mounted on a wheelchair. First
and foremost, it requires to know precisely the pos-
ition and orientation of the camera in space relative
to the robot (transformation matrix of the camera to
the robot) which is not possible to have with preci-
sion if the robot is moving. Moreover, robustness of
the assistance to a change of position of the object
can be prevented if the position is computed before
any movement [17]. Arm occlusion and occlusion
from objects in the environment can render the com-
putation of the position of objects or sensing of the
environment in real-time impossible. Some studies
in the literature have proposed eye-in-hand sensor-
based servoing, with the sensor mounted on the end-
effector [5, 18, 19]. Eye-in-hand sensor-based servo-
ing allows the robotic manipulator to be on a mobile
base such as a wheelchair and answers most occlu-
sion problems. For eye-in-hand servoing, the size of
the sensor can complicate the integration on the end-
effector. Sensor-based servoing requires the extrac-
tion of visual features. In many cases, the objects
are known and their cloud representation is stored
in a library of objects or they are tagged. Tags are
sometimes used to provide necessary information to
grasp the objects [4, 18, 20, 21] but the placement
of tags and the encoding of the information such as
the orientation of the gripper are most likely cum-
bersome in a daily-life context. The identification of
objects using template matching methods [14, 22,
23] requires a known-in-advance library of objects.
In an at-home context where objects are varied and
unknown, the use of template matching methods can
be complicated.

The object targeted by the user, the object to
grasp, needs to be isolated from the possibly mul-
tiple extracted objects, it can be the closest to the
end-effector [5, 20, 23] or the most likely based on
bayesian inference [22, 24] or chosen using a BCI
input [25–27]. Different methods of authority trans-
fer exist meaning that the amount of control the user
and the autonomous part has can vary. In some meth-
ods, the user only selects the object and the movement
of the robot is autonomous [25, 28, 29]. In others,
the control is switched from the user to the sensor-
based control [14, 18, 20, 30]. Finally, in other meth-
ods, the BCI and sensor-based control are blended
with varying or constant weighs. The idea behind the blending of the autonomous and BCI control is to benefit from the advantages of both user and sensor-based control. This fusion also allows to cede as little control as possible to the autonomous control in order to make the device more acceptable by users [12]. The weighting of the fusion of BCI and sensor-based cues can depend on different factors including the distance to the object [5, 7, 14] or the confidence in the object selection by the intent detection algorithm [22, 23, 31].

In this article, we propose a SC framework to assist motor impaired individuals at home. As mobility and object manipulation are key components of autonomy, we focus on how to assist the BCI control of a power wheelchair and of a robotic manipulator. This framework relies on the chronic WIMAGINE ECoG implants and on depth-sensor based servoing. The WIMAGINE implant has the advantage of being less invasive that micro-arrays [32] and that it presents signal stability through time [33]. It was also shown that it allows long-term model stability with up to 8 DoF [10]. This makes it particularly suitable for long-term use. Moreover, the only required setup is the installation of a head-set that can be easily done by a non-expert individual. For power wheelchair driving, the proposed sensor-based servoing uses 48 depth-sensors mounted on the wheelchair allowing an obstacle avoidance assistance. For reach-and-grasp tasks, the proposed sensor-based servoing uses two 8-by-8 depth-sensors mounted on the end-effector of the manipulator allowing an eye-in-hand servoing. The proposed method requires no *a priori* knowledge of the object and a low-cost computation-wise visual feature extraction. We propose to isolate the object to grasp based on distance. The fusion of BCI and sensor-based velocities is based on the distance to the object considering that the closest the end-effector is to an object, the more likely it is that it was the object to grasp.

# 2. Material and methods

## 2.1. Overview of the system

Sensor-based SC assistance has been embedded in two robotic platforms that can be controlled through the same BCI: a power wheelchair with an obstacle avoidance assistance and a robotic manipulator for assisted reach-and-grasp. An overview of the system is presented in figure 1.

## 2.2. BCI system

### 2.2.1. ECoG recordings

The BCI used to teleoperate both robotic platforms is based on the chronic ECoG WIMAGINE neuroprosthesis [32]. ECoG signal recordings were sampled at 586 Hz and were low and high pass filtered in a bandwidth from 0.5 Hz to 300 Hz using an analog

bandpass filter and, after 12 bit digitalization, using a digital low pass FIR filter with a cutoff frequency 293 Hz. The data were then radiotransmitted to a custom designed base station connected to a computer. The propagation delay from the electrodes to the PC platform due to signals acquisition remained under 100 ms. During the experimental sessions presented in this article, only 32 electrodes were recorded in a checkerboard-like pattern because of limited data rates in the radio link. ECoG signals were finally transferred in real-time to the decoding software that allows the prediction of the subject's discrete mental states.

### 2.2.2. Neural features extractions

Neural features extraction procedure is described in details in [10]. After replacing missing points by previous values to cope with loss of data in the raw ECoG, 1 s-long epochs of neural signals with a 100 ms sliding step, were mapped to the temporal frequency space using a complex continuous wavelet transform (CCWT) (Morlet) with a frequency range from 10 to 150 Hz (10 Hz step). The absolute value of the CCWT coefficients was then decimated along the temporal modality to obtain a 10-timepoints description of the epoch for each frequency band and each channel, resulting in the temporal-frequency-spatial neural feature tensor $X_t \in R^{10*15*32}$.

### 2.2.3. Model training (HMM)

Hidden Markov Models (HMM) combining emission and transition probabilities were trained and used for online classification from neural features [10]. The emission probability was computed using REW-NPLS with one-hot encoded class labels, post-processed by softmax function [34]. The transition probability matrix was estimated by counting the number of transitions in the training set. The class prior was established to ensure equal probability distribution among classes. The HMM decoder outputs states prediction for each 1 s-long epochs of signals, allowing a 10 Hz prediction rate. The mental state with the highest probability is then sent to one of the two robotic platforms. Only one BCI command is provided by time step. During online training, the model was updated every 15 s and the current model was used for prediction.

## 2.3. Power wheelchair

### 2.3.1. Augmented wheelchair platform

An off-the-shelf You-Q back-wheel drive Alex Qlass power wheelchair was used to conduct the clinical trials. The power wheelchair used in the clinical trial is presented in figure 2(a).

### 2.3.2. BCI teleoperation of the power wheelchair

Each mental state is associated to a displacement in a direction or to an immobility (idle) command.
<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

The user has thus control over 2 DoF of the power wheelchair (forward-left, forward-right, forward and backward). A forward motion was associated to the left and right motion as the wheelchair is a non-holonomic mobile base. The BCI commands are sent to the wheelchair by bluetooth, introducing around 400 ms of latency and then used to control the velocity of the wheelchair.

### 2.3.3. Sensor-based SC for wheelchair driving

The sensor-based SC implemented to assist the control of a power wheelchair has been presented in [13]. Furthermore, Time-of-Flight (ToF) infrared sensors from STMicroelectronics (VL53L1CX) are used for the sensing of the environment. The SC method allows a smooth obstacle avoidance, indeed, it allows to stop the power wheelchair on a frontal arrival in front of an obstacle and to circumvent obstacles in case of an angled arrival [35]. The SC law is based on the definition of two distinct areas (allowed/bidden areas) in the wheelchair velocity domain. An example is presented in figure 2(b). All computations are performed in less than 10 ms on an embedded microprocessor.

## 2.4. Robotic manipulator

### 2.4.1. *Reach-and-grasp robotic platform*

We used a Jaco robotic manipulator from Kinova, a 6-DoF robotic manipulator equipped with a three-finger gripper. This robotic manipulator, that can be easily mounted on power wheelchairs, is used in the rehabilitation field by people to perform ADLs requiring grasping tasks. The robot can be operated with a joystick but other control interfaces can be used, a BCI in our case. To operate the robot in our BCI-control grasping assistance framework, we used the kinova-ros package with ROS Noetic running on Ubuntu 20.04.

### 2.4.2. BCI teleoperation of the robotic manipulator

Each mental state is associated to a displacement in a direction, to an immobility (idle) command or a gripper command. The user has control over 2 DoF of the robotic manipulator, the translations of the end-effector in the plane at fixed height (forward-backward, left-right) and the opening/closing of the gripper. The BCI commands are sent to the robotic manipulator by TCP-IP, with around 200 ms of latency between the sending of the command and
<figure><img src="image_6.png" /><figcaption>Figure 3. (a) Jaco robot with mounted sensors on the gripper. The object used in the experiment is in front of the gripper. A depth-image of the object from the sensor $d_1$ is represented. (b) The reference frames of the sensors $d_i$ ($R_{di}$) and of the end-effector $R_e$ are represented on a schematic representation of the gripper. The first view is from under the gripper and the second is from the side and from under it. The translations between the sensor and end-effector reference frames are also represented ($t_{d_1}$ and $t_{d_2}$). (c) Representation of the different SC phases that rely on the object-end-effector distance. The threshold distances of the SC ($d_{shared}$) and autonomous ($d_{auto}$) areas are represented alongside the variation of the weighting factor $\alpha$ over the distance to the object.</figcaption></figure>

the beginning of the movement. The patient does not have control over the orientation of the gripper nor the vertical position; the orientation of the gripper is thus fixed reducing the working space of the robot. To alleviate the BCI control, only one mental task was used to open and close the gripper. The gripper is opened if it was previously closed and closed if previously opened. As the patient can only teleoperate the end-effector in a plane of fixed height, a sensor-based servoing using the sensor under the gripper is implemented to control the position of the gripper in the vertical direction. The altitude is automatically increased (≈ 30 cm) after the closing of the gripper to avoid scraping the table with the object and automatically brought down prior the opening of the gripper. The BCI movement commands, positional increments, are used to control in velocity the position of the end-effector in a 100 Hz servoing loop using a proportional gain. The positional increment size ($\delta$ = ±0.02m) and the proportional gain value ($K_p$ = 3.0) have been chosen empirically to allow a precise positioning of the end-effector and to provide a useful visual feedback to the patient.

### 2.4.3. Sensor-based control of the robotic manipulator
To perform the reaching task, we propose an eye-in-hand sensor-based servoing. The proposed solution does not relies on a priori knowledge of the object and can be used on moving platforms such as a power wheelchair.

#### 2.4.3.1. Sensors

The eye-in-hand sensor-based servoing is implemented using 2 multi-array depth sensors mounted on the end-effector of the robot. The sensors used are Time-of-Fight VL53L5CX infrared sensors from

STMicroelectronics, they provide a 8 × 8 depth image at 15 Hz. Figure 3(a) depicts the mounting of the two sensors, inside (called $d_1$) and under (called $d_2$) the gripper. As objects are grasped by the side, the sensor inside the gripper is placed so it faces the object to grasp. A depth-image of the sensor inside the gripper ($d_1$) when facing the object is included in figure 3(a).

The reference frames of the sensors as well as the reference frame of the end-effector are represented on figure 3(b). According to figure 3(b), we get:

$$d_i \mathbf{t}_e = \begin{bmatrix} 0 & 0 & t_{d_i} \end{bmatrix}^T \quad \text{with } i \in \{1, 2\} \quad (1)$$

$$d_1 \mathbf{R}_e = \begin{bmatrix} -1 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \quad \text{and}$$

$$d_2 \mathbf{R}_e = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 0 & -1 \\ 0 & -1 & 0 \end{bmatrix}. \quad (2)$$

Only 3 DoFs of the robot are controlled by the sensor-based servoing, the three translation velocities. In order to not disturb the control of the end-effector by the user, rotations of the end-effector are not permitted. We thus have:

$$e \mathbf{J}_e = \begin{bmatrix} 1 & 0 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 & 0 \end{bmatrix}^T. \quad (3)$$

#### 2.4.3.2. Visual features extraction

We propose a sensor-based servoing approach based on a set of visual features from both sensor $d_1$ and $d_2$. The sensor-based servoing is in the end-effector reference frame.
Let $\mathbf{b}_i = [b_{ix} \ b_{iy} \ b_{iz}]$ with $i \in \{1, 2\}$ be the barycenter of the extracted cluster from the depth sensor $d_i$ image with $b_{ix}$ and $b_{iy}$ respectively the x-coordinate and y-coordinate of the barycenter $b_i$ and $b_{iz}$ the mean depth value of the extracted cluster from the depth sensor $d_i$ image.

We propose a sensor-based servoing task that regulates a set $\mathbf{s}$ of visual features extracted from $b_1$ and $b_2$ so that $\mathbf{s} = (b_{1x}, b_{1z}, b_{2z})$.

In order to extract the visual features $\mathbf{s}$, the depth images from the sensors are filtered using a passthrough filter to filter out all points with distance out of our range of interest with $[z_{\min}, z_{\max}] = [0.02 \text{ m}, 0.5 \text{ m}]$ as cutoff distances. Following this filtering, an euclidean clustering algorithm using a Kd-tree structure is used to extract clusters with an area between $[a_{\min}, a_{\max}] = [3 \text{ pxl}, 64 \text{ pxl}]$ with a depth tolerance of $z_{\text{margin}} = 0.03 \text{ m}$. After the clustering process, the barycenter $b_j^i = [b_{jx}^i \ b_{jy}^i \ b_{jz}^i]$ of each cluster $c_i$ from depth sensor $d_j$ is computed. Then for each sensor $d_j$, $b_j = b_j^k$ such that $b_{jz}^k = i \ b_{jz}^i$.

### 2.4.3.3. Sensor-based servoing

The sensor-based servoing is used to place the end-effector close to an object represented by the set of visual features $\mathbf{s}$ in the depth-images of the sensors $d_1$ and $d_2$. The desired distance to the object and to the plane were chosen empirically to ensure a stable grasp given the gripper geometry and the gripper height.

* let $\mathbf{v}_s$ be the velocity computed by the sensor-based servoing of the end effector in its own frame,

* let $\mathbf{s}_1 = (b_{1x}, b_{1z})$ and $\mathbf{s}_2 = b_{2z}$ such that $\mathbf{s} = (\mathbf{s}_1, \mathbf{s}_2)$,

* let $\mathbf{s}^* = (\mathbf{s}_1^*, \mathbf{s}_2^*)$ be the desired visual features with $\mathbf{s}_1^* = (0, d_{\text{object}}^*)$ in the image of sensor $d_1$ and $\mathbf{s}_2^* = d_{\text{plane}}^*$ in the image of sensor $d_2$ with $d_{\text{grasp}}^* = 0.06 \text{ m}$ and $d_{\text{plane}}^* = 0.05 \text{ m}$.

* let $\mathbf{e}_{s_1} = s_i - s_i^*$ with $i \in \{1, 2\}$ be the error between the desired and current visual features $s_i^*$ and $s_i$,

* let $\lambda_1$ and $\lambda_2$ be the sensor-based servoing gains, $\lambda_1 = \lambda_2 = 2$.

* let $L_{s_1}$ (resp. $L_{s_2}$) be the interaction matrix of the visual feature set $s_1$ (resp. $s_2$),

* let $d_1^i \mathbf{W}_e$ and $d_2^i \mathbf{W}_e$ be the velocity screw transformation that links the camera velocity (expressed in $d_1$ and $d_2$) to the end effector velocity screw (expressed in $e$).

We use the well-known interaction matrices of image point coordinates and the interaction matrix related to the coordinate Z of a 3D point given by [36]:

$$ 
\begin{aligned}
\mathbf{L}_{s_1} &= \begin{bmatrix} \frac{1}{b_{1z}} & 0 & \frac{b_{1x}}{b_{1z}} & b_{1x}b_{1y} & -(1+b_{1x}^2) & b_{1y} \\ 0 & 0 & -1 & b_{1y}b_{1z} & b_{1x}b_{1z} & 0 \end{bmatrix} \\ 
\mathbf{L}_{s_2} &= \begin{bmatrix} 0 & 0 & -1 & b_{2y}b_{2z} & b_{2x}b_{2z} & 0 \end{bmatrix}.
\end{aligned} 
\quad (4) 
$$

We set

$$ 
\begin{aligned}
\mathbf{J}_{s_i} &= \mathbf{L}_{s_i} \cdot \mathbf{d}_i \mathbf{W}_e \cdot \mathbf{e}_i \mathbf{J}_e && \text{with} \\ 
\mathbf{d}_i \mathbf{W}_e &= \begin{bmatrix} \mathbf{d}_i \mathbf{R}_e & [\mathbf{d}_i \mathbf{t}_e]_\times & \mathbf{d}_i \mathbf{R}_e \\ \mathbf{0}_{3 \times 3} & \mathbf{d}_i \mathbf{R}_e & \mathbf{0} \end{bmatrix} && (5) 
\end{aligned} 
$$

with $[\mathbf{d}_i \mathbf{t}_e]_\times$ the skew matrix expression.

Using equations (4) and (5), we can express $J_{s_1}$ and $J_{s_2}$ as

$$ \mathbf{J}_{s_1} = \begin{bmatrix} J_{x1} & 0 & J_{z1} \\ 0 & 0 & 1 \end{bmatrix} \quad \text{and} \quad \mathbf{J}_{s_2} = \begin{bmatrix} 0 & J_{y2} & 0 \end{bmatrix}. \quad (6) $$

with $J_{x1}$, $J_{z1}$ and $J_{y2}$ being scalar values. As shown by equation (6), the visual feature sets $s_1$ and $s_2$ are correlated to different degrees of freedom of the end effector. We can then apply the following control law:

$$ \mathbf{v}_s = -(\lambda_1 \mathbf{J}_{s_1}^+ \mathbf{e}_{s_1} + \lambda_2 \mathbf{J}_{s_2}^+ \mathbf{e}_{s_2}). \quad (7) $$

### 2.4.4. SC of the robotic manipulator

The SC assistance is meant to assist the user to reach-and-grasp objects but is not meant to replace the user. The autonomous sensor-based control is progressively and continuously fused with the BCI control of the end-effector of the robotic manipulator in the end-effector reference frame.

* let $\alpha$ be the weighting factor of the fusion of the teleoperation BCI velocities and the sensor-based velocities.

* let $\mathbf{v}_e$, $\mathbf{v}_u$ and $\mathbf{v}_s$ be respectively the velocity of the end-effector after fusion, the velocity of the teleoperation based on BCI user inputs and the sensor-based velocities.

We have

$$ \mathbf{v}_e = (1 - \alpha) \mathbf{v}_u + \alpha \mathbf{v}_s. \quad (8) $$

The weighting factor $\alpha$ of the fusion follows a sigmoid function as in [7] or [22] depending on the distance of the end-effector to the object with $\alpha \in [0, 1]$. The variation of $\alpha$ over distance to the object written in equation (9) is represented in figure 3(c). When $\alpha = 0$, the user has full control over the position of the end-effector. When $\alpha = 1$ the end-effector is fully controlled by the sensor-based servoing. For $\alpha \in ]0, 1]$, the control is shared between user BCI control and sensor-based servoing. The distance of the end-effector to the object for which $\alpha = 0$ and $\alpha = 1$ are respectively $d_{\text{shared}} = 0.25 \text{ m}$ and $d_{\text{auto}} = 0.15 \text{ m}$. This means that when the end-effector is far from the object, the patient has total control of the position of the end-effector and the closest the end-effector is to the object, the more weight is given to the sensor-based control. When the end-effector is really close to
the object, the control is completely autonomous. The gripper is closed when the desired position is reached. Figure 3(c) represents the different phases of the SC.

$$ \alpha = \begin{cases} \frac{0}{1+\exp(K(d_{\text{obj}}-\text{shift}))} & \text{if } d_{\text{obj}} \ge d_{\text{shared}} \\ \frac{1}{1} & \text{if } d_{\text{auto}} < d_{\text{obj}} < d_{\text{shared}} \\ 1 & \text{if } d_{\text{obj}} \le d_{\text{auto}} \end{cases} \quad (9) $$

with:

* $d_{\text{obj}}$ the distance to the object

* $d_{\text{shared}}$ the threshold distance of the SC area

* $d_{\text{auto}}$ the threshold distance of the autonomous control area

* shift = $\frac{d_{\text{shared}}-d_{\text{auto}}}{2}$

* K a gain chosen so the weighting gain $\alpha$ would be equal to 1 when $d_{\text{object}} \le d_{\text{auto}}$ and 0 for $d_{\text{object}} \ge d_{\text{shared}}$.

## 2.5. Clinical evaluation

### 2.5.1. Subject and clinical trial

The proposed experiment involves a user implanted with the WIMAGINE ECoG implant enabling the recording of brain signals from the motor and sensory cortices [37]. The data was recorded and analyzed as a part of the 'BCI and Tetraplegia' (ClinicalTrials.gov identifier: NCT02550522) clinical trial, which was approved by the Agency for the Safety of Medicines and Health Products (Agence nationale de sécurité du médicament et des produits de santé-ANSM) with the registration number: 2015-A00650-49 and the ethical Committee for the Protection of Individuals (Comité de Protection des Personnes-CPP) with the registration number: 15-CHUG-19. The patient gave his inform consent to participate to the study. The patient who participated in this experiment is a 33 year-old male with traumatic quadriplegia caused by accident 14 years before this study. He was implanted bilaterally with two WIMAGINE implants in November 2019 and was since trained using a custom-made BCI platform to control multiple real and virtual effectors. Before implantation, the patient underwent fMRI and MEG exams to ensure that he was able to produce measurable motor imagery in the sensorimotor cortices. Due to an hardware failure in one of the implant, only the left implant of the patient was used.

### 2.5.2. BCI decoding

#### 2.5.2.1. Mental tasks

The mental tasks used by the patient to control the wheelchair and the robotic manipulator were identical except for the grasp added for the robotic manipulator. During idle state periods, the patient was not asked to perform any specific tasks but to take a break in the experiment, and was allowed to speak

and move freely. For left and right motion, he performed motor imagery of the flexion of the left and right wrists. Forward motion was triggered by motor imagery of walking and backward motion by shoulder (shrugging) motor imagery. Motor imagery of power grasp (all fingers flexion) was used for grasping.

#### 2.5.2.2. Online training

Online BCI models that were used in this clinical evaluation were used for several sessions. For each effector, the model was trained from scratch during the first experimental session, and was updated if necessary at the beginning of each following session. Model training and updating were done in a virtual environment without controlling the real effector. This environment allowed to display cues to the patient, for each of which he associated the corresponding mental task. States predictions were also displayed into the environment so as to provide a feedback to the patient regarding the use of the current model. During training and updating, ECoG data and cue labels were used to (re)calibrate the decoder (figure 4(a)). At the beginning of each session except for the first one, the need for model updating was evaluated by testing the previous session model. This was done by presenting at least one cue per state to the patient, without updating the model from these data. If performances were not satisfactory enough to control the effector, the model was updated and the update was stopped when performances reached an acceptable threshold. The viability of control performances was visually controlled by the experimenter. At the end of the training, the model was tested again to evaluate the gain of performances allowed by the update.

#### 2.5.2.3. Models and training evaluation

Training phases were evaluated offline in terms of gain of performances and training duration. To do so, for each state, we computed its average predicted probability over time when it was cued to the patient. This was done before and after the possible training phase. To evaluate the long-term viability of the model, the number of cues and the duration necessary to train or update the model were calculated for each state. Finally, we also computed the balanced accuracy between classes and the confusion matrix after the training phase to assess the models that were used to control the effectors. This was done by counting at each time step (0.1 s) the state with the highest decoding probability (predicted class) and the cue (true class). One should note that balanced accuracy and confusion matrices were computed after correction of the latency between cues and states predictions, which was computed by cross-correlation. This latency corresponds to the mean delay between a cue and the instant the associated state prediction becomes the
<figure><img src="image_7.png" /><figcaption>**Figure 4.** Schematic representation of the clinical evaluation with the patient. (a) Model training phase in which the subject was cued with instruction of direction or idle, and performed associated mental tasks. Subsequent labeled ECoG data were used to train or update the model. Current model was applied to the ECoG data to predict the state. Prediction of the cued state was displayed on the screen providing visual feedback to the subject. Once the model was considered by the experimenter to be good enough, it was fixed and saved before use with the effectors. (b) Control of the effectors. During experiments with the effectors, the model was used to decode in real-time the mental tasks of the subject. Decoded states were sent to the wheelchair or the robotic manipulator as control input. For wheelchair driving evaluation, the subject had to follow a circuit materialized by rigid obstacles. Regarding reach-and-grasp task, the subject had to controlled the robot to grasp an object in a specific position (two possible positions) and release it in a release area. Both experiments were performed with and without the assistance of SC.</figcaption></figure>

most probable. It comes from the reaction time of the patient as well as a decoding delay which is due to the fact that each prediction is based on the last second of signal, then including at the beginning periods before the patient triggered his motor imagery.

### 2.5.3. Experimental setup and procedure for the wheelchair driving task

#### 2.5.3.1. Experimental setup and conditions

A circuit of roughly 45 m² with static obstacles was setup to evaluate the SC method. The circuit required the patient to perform difficult maneuvers in narrow spaces simulating a home-like environment. The experimental setup is represented in figure 4(b). The assistance can be activated or deactivated, we have thus two experimental conditions named SC standing for with shared control and NSC standing for without SC.

#### 2.5.3.2. Experimental procedure

The entire experiment was divided into 5 sessions of approximately 2 h. Eighteen trials with SC and 17 without shared control (NSC) were randomly distributed over the 5 sessions. An experimenter could

at any moment take control of the wheelchair from the patient in case of safety issues or in case of an impossibility to maneuver it. The patient was not informed of the activation/deactivation of the collision avoidance solution.

#### 2.5.3.3. Data pre-processing and metrics used to evaluate the task performances

Different metrics were used to evaluate the impact of SC on the power wheelchair driving task: the time to complete the circuit, the number of collisions of the wheelchair with the obstacles, the number of times the experimenter took control away from the patient (that generally occurred after a collision) and the number of changes of mental tasks. The number of changes of mental tasks was measured as the number of changes between predicted states with the highest probability. It can thus include changes that are due to decoder errors not intended by the patient. However since the same model was used for SC and NSC trials, it is very unlikely that a difference between conditions is not due to real changes in patient motor imagery. All statistic analyses were performed using
Python (statsmodels package). The normality of the data was tested (Shapiro-Wilk test). *t*-tests with SC (2 levels: SC and NSC) as factors were used with $α = 0.05$ for normal distributions and Mann-Whitney U tests for non-normal data. Outliers were removed if they exceeded 3*IQR* (InterQuartile Range). The number of outliers removed is indicated for each metric.

### 2.5.4. Experimental setup and procedure for the reach-and-grasp task

#### 2.5.4.1. Experimental setup and conditions

The goal was to assess the assistance provided for a reach-and-grasp task. While we aim to use the robotic manipulator mounted on a wheelchair, the robotic manipulator was mounted on the edge of a table for practical reasons as only the assistance for reach-grasp task is presented. A foam pin, portrayed in figure 3(a), was placed on the table in one of two possible positions. A cardboard box, placed on the table, was used as a releasing spot for the object. The experiment setup is represented in figure 4(b). The assistance can be activated or not. The patient had control of the gripper only for trials without assistance. For trials with the assistance, the gripper is controlled by the assistance and thus BCI gripper commands are ignored. We have four experimental conditions named SC-Pos1, SC-Pos2, NSC-Pos1 and NSC-Pos2 with: SC and NSC standing respectively for with SC and NSC and Pos1 and Pos2 standing for the two object positions.

#### 2.5.4.2. Experimental procedure

The entire experiment was divided into 7 sessions of roughly 2 h each. It took around 14 h with never more than 2 sessions per day. Trials were presented to the patient in randomized blocks of 5 identical trials. While the patient could ask for breaks at any time, all blocks started in a session were finished inside the same session. 58 successful trials have been recorded: 15 repetitions of the NSC-Pos1, NSC-Pos2, SC-Pos-Pos1 and 13 repetitions of SC-Pos2. The patient was not informed of the activation/deactivation of the reach-and-grasp assistance.

#### 2.5.4.3. Data pre-processing and metrics used to evaluate the tasks performances

Only completed and successful trials were used for the analysis. Different metrics were used to evaluate the impact of SC on the reach-and-grasp task: the time to perform the task and the time to successfully grasp the object, the number of changes of mental tasks and the number of grasps (opening and closing of the gripper). The start and the end of a trial are respectively the first mental task recorded and the final releasing of the object. The first successful grasp of the object is used to compute the time to grasp the object. The number of grasps was studied as the assistance prevented unwanted opening and closing of the gripper. All statistical analyses were performed

using Python (statsmodels package). 2-way ANOVA with SC (2 levels: with and without) and position (2 levels: position 1 and position 2) as factors were used with $α = 0.05$. The normality of the data was tested (Shapiro-Wilk test) and a logarithmic normalization was used in case of rejection of the normality criteria. Outliers were removed if they exceeded 3*IQR*. The number of outliers removed is indicated for each metric.

## 3. Results

### 3.1. Evaluation of the wheelchair driving assistance

#### 3.1.1. BCI models evaluation

##### 3.1.1.1. Training efficiency

Figure 5(a) shows the average probability of prediction of a state when it was cued to the subject before and after training as well as time and number of cues necessary to (re)train the model, for each session. After the model creation, the decoded state probability was equal to 0.60 ± 0.06. Then, as expected, updating the decoder allowed to improve decoded state probability for each session (before: 0.45 ± 0.17, 0.55 ± 0.10, 0.51 ± 0.19, 0.45 ± 0.08; after: 0.74 ± 0.08, 0.81 ± 0.05, 0.69 ± 0.04, 0.55 ± 0.07; for sessions 2 to 5 respectively). Considering that there are 5 states, the chance level of this probability was 0.2. Regarding the time and the number of cues necessary to create the model from scratch, it took 245.9 ± 25.3 s and 13.8 ± 3.0 cues per state during first session (i.e. around 20 min and 69 cues in total). In the next sessions, the updating time and cues remained reasonably low (time: 0.0 ± 0.0 s, 28.4 ± 12.7 s, 58.3 ± 12.5 s, 0.0 ± 0.0 s; cues: 0.0 ± 0.0, 2.2 ± 1.1, 6.2 ± 2.3, 0.0 ± 0.0; for sessions 2 to 5 respectively). Note that in sessions 2 and 5, the decoded state probability improved between before training and after training while the model remained fixed. This indicates that just presenting cues to the patient without updating the model allowed him to retrieve the right motor imagery to control the model.

##### 3.1.1.2. Model performances

Figure 5(b) shows the average confusion matrix of the models used with the wheelchair, indicating that states were well predicted in 69.7% to 80.0% of the time samples. Some confusion was present with backward predicted as forward (15.2%) and left predicted as right (19.0%). Other important confusions were the prediction of a moving state as idle. Globally, balanced accuracy was equal to 74.5% ± 5.1% over sessions (chance level 20%). Also, the system patient plus model induced 1.98 ± 0.7s latency in the decoding.

##### 3.1.2. Task performances

###### 3.1.2.1. Circuit duration

Figure 5(c) shows the time to perform the complete circuit. The time is significantly reduced with SC (effect of SC, *p* = 0.019, *t*-test). The average time in
<figure><img src="image_8.png" /><figcaption>(b)</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(c)</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>**Figure 5.** Wheelchair driving assistance evaluation: (a) Top: BCI training efficiency, evaluated through the average probability over time of predicting a state when it was cued, before (red) and after (green) training at each session. Each bar and errobar represent respectively the mean and standard deviation across states. Bottom: Average time (orange) and number of cues (blue) per state that were necessary to train or update the model at each session. Each point and errobar represent respectively the mean and standard deviation across states. (b) BCI model performances. Left: Average confusion matrix across sessions of the model used for wheelchair control (after training). Right: balanced accuracy of the model (mean + SD across sessions) and latency of the system patient + decoder (mean + SD across sessions). (c) Wheelchair driving task performances with the time to complete the circuit in seconds, the number of collisions, the number of control losses and the number of changes of mental tasks with (yellow) and without (purple) SC. Each plot represents the individual points of the different repetitions of the circuit, and the boxplot showing median, lower/upper quartile and the most extreme points that are inside 1.5\*interquartile range. \*p < 0.05, \*\*\*p < 0.001 in t-tests or Mann–Whitney U tests.</figcaption></figure>

seconds to complete the circuit is 278.3 ± 88.2 with SC and 361.8 ± 110.7 NSC.

### 3.1.2.2. Number of collisions

Figure 5(c) shows the number of collisions per trial. With SC, only two collisions occurred amongst 18 trials compared to 96 over the 17 trials NSC. The number of collisions is significantly reduced with SC (effect of SC *p* < 0.001, Mann–Whitney U test). The median number of collisions is of 0 (IQR 0.0–0.0) with SC and 5.0 (IQR 3.75–8.0) NSC). Two outliers were removed from the analysis.

### 3.1.2.3. Number of control losses

Figure 5(c) shows the number of times per trial the experimenter had to take control over the patient for safety reasons or due to an impossibility for the patient to maneuver the wheelchair. With SC, only two losses of control by the patient occurred among 18 trials compared to 35 over the 17 trials NSC. The number of control losses is significantly reduced with SC (effect of SC *p* < 0.001, Mann–Whitney U test). The median number of control losses is of 0 (IQR 0.0–0.0).

with SC and 2.0 (IQR 0.75–3.0) NSC). Two outliers were removed from the analysis.

### 3.1.2.4. Number of changes of mental tasks

Figure 5(c) shows the number of changes of mental tasks per trial for all trials. Although there was a trend towards a reduced number of changes of mental tasks with SC, this was not significant (effect of SC, *p* = 0.08, *t*-test). The average number of changes of mental tasks is 78.8 ± 29.5 with SC and 98.5 ± 30.9 NSC. Due to lack of data, three trials NSC and one with were not used in the analysis.

## 3.2. Evaluation of the reach-and-grasp assistance

### 3.2.1. BCI performances

#### 3.2.1.1. Training efficiency

Figure 6(a) shows the average probability of prediction of a state when it was cued to the subject before and after training as well as the time and the number of cues necessary to (re)train the model, for each session. Note that in this experiment, a ‘model’ session was dedicated to model creation and the robotic manipulator was not used during this session. After
<figure><img src="image_11.png" /><figcaption>(b)</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(c)</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>**Figure 6.** Reach-and-grasp assistance evaluation: (a) Top: BCI training efficiency, evaluated through the average probability over time of predicting a state when it was cued, before (red) and after (green) training at each session. Each bar and errobar represent respectively the mean and standard deviation across states. Bottom: Average time (orange) and number of cues (blue) per state that were necessary to train or update the model at each session. Each point and errobar represent respectively the mean and standard deviation across states. (b) BCI model performances. Left: Average confusion matrix across sessions of the model used for robotic manipulator control (after training). Right: balanced accuracy of the model (mean + SD across sessions) and latency of the system patient + decoder (mean + SD across sessions). (c) Reach-and-grasp task performances with the time to perform the task and the time to grasp the object in seconds, the number of changes of mental tasks and the number of grasps with (yellow) and without (purple) SC. Each plot represents the individual points of the different repetitions of the circuit, and the boxplot showing median, lower/upper quartile and the most extreme points that are inside 1.5\*interquartile range.\* *p* < 0.05, \*\*\* *p* < 0.001 in 2-way ANOVAs (with log-normalization when data distribution was not normal).</figcaption></figure>

the model creation, the decoded state probability was equal to 0.64 ± 0.06. Then, as expected, updating the decoder allowed to improve decoded state probability for each session (before: 0.58 ± 0.10, 0.51 ± 0.10, 0.41 ± 0.11, 0.59 ± 0.08, 0.71 ± 0.08; after: 0.74 ± 0.05, 0.60 ± 0.08, 0.75 ± 0.07, 0.61 ± 0.05, 0.73 ± 0.07; for sessions 1 to 5 respectively). Considering that there are 6 states, the chance level of this probability was about 0.17. Regarding the time and the number of cues necessary to create the model from scratch, it took 167.7 ± 25 s and 18.8 ± 7.5 cues per state before the first session (i.e. around 17 min and 113 cues in total). In the next sessions, the updating time and cues remained reasonably low (time: 21.4 ± 6.4 s, 0.0 ± 0.0 s, 49.6 ± 6.0 s, 4.7 ± 2.2 s, 0.0 ± 0.0 s; cues: 3.0 ± 1.3, 0.0 ± 0.0, 5.8 ± 2.3, 1.0 ± 0.5, 0.0 ± 0.0; for sessions 4 to 5 respectively).

### 3.2.1.2. *Model performances*

Figure 6(b) shows the average confusion matrix of the models used with the robotic manipulator, indicating that the states were well predicted in 61.8% to

82.4% of the time samples. Almost all important confusions were the prediction of a moving state as idle, which can slow the patient in performing the task but not take him away from his target. Globally, the balanced accuracy was equal to 72.4% ± 3.5% over sessions (chance level 16.7%). Also, the system patient plus model induced 1.7 ± 0.1 latency in the decoding.

### 3.2.2. *Task performances*

#### 3.2.2.1. *Time to perform the task*

Figure 6(c) shows the time to perform the reach-and-grasp of the object, its displacement and its release. This time is significantly reduced with SC (effect of SC, *p* < 0.001). There is no effect of position on the impact of SC (no significant interaction *p* = 0.08). The time to perform the task was not impacted by the position of the object (no effect of Position *p* = 0.34). The data was log-normalized before statistical analysis as the normality hypothesis was rejected. The median time in seconds to perform the task is of 106.4 (IQR 91.1 – 153.1) with SC and 157.6 (IQR 136.6 – 219.9) without.
### 3.2.2.2. *Time to grasp the object*

Figure 6(c) shows the time to perform the first successful and possibly only grasp of the object. This time is significantly reduced with SC (effect of SC, $p = 0.028$). While there is no effect of position on the impact of SC (no significant interaction $p = 0.31$) the impact of SC seems greater for position 2. The time required to grasp the object was not impacted by the position of the object (no effect of Position $p = 0.68$). The average time in seconds is of $51.9 \pm 40.1$ with SC and $75.3 \pm 38.2$ NSC. One outlier was removed for this analysis.

### 3.2.2.3. *Number of changes of mental tasks*

Figure 6(c) shows the number of mental task changes for each condition. The number of mental task changes is significantly different between SC and NSC conditions (effect of SC $p < 0.001$). This number is similarly reduced for both positions (no interaction $p = 0.86$) and is not impacted by the position of the object (no effect of Position, $p = 0.52$). The data was log-normalized as the normality hypothesis was rejected. The median number of changes of mental tasks is of 30.0 (IQR 22.0 – 38.5) with SC and 41.5 (IQR 32.0 – 57.0) NSC.

### 3.2.2.4. *Number of grasps*

As the gripper is closed to grasp the object and opened to release it, the minimum and ideal number of grasps is 2. The number of grasps was greater than 2 for 23 trials amongst the 30 trials without assistance. Figure 6(c) shows the number of grasps per condition. The number of grasps is greater NSC ($p < 0.001$) but is not significantly impacted by the position (no effect of Position $p = 0.10$ nor interaction $p = 0.09$). The average number of grasps is of $2.0 \pm 0.0$ with SC and $4.8 \pm 2.7$ NSC.

## 4. Discussion

We proposed a BCI-depth-sensor-based SC framework to provide assistance for individuals with motor deficits at home. We focused our study on a mobility task, power wheelchair driving, and a reach-and-grasp task using a robotic manipulator. This framework relies on the WIMAGINE ECoG implant and on depth-sensor based servoing. We showed that SC improved the user's performances for both tasks in a laboratory-based controlled environment.

### 4.1. Power wheelchair driving assistance

For the power wheelchair driving task, an obstacle avoidance assistance was implemented to assist the user in the BCI-control of the power wheelchair. The SC algorithm blends the BCI control with a set of constraints deduced from the depth-sensors. This SC method had been tested in previous clinical trials using a joystick as interface but had not been tested yet with a BCI [35].

In our study, we showed that the performances of the patient are improved with SC whether it is in terms of efficiency or safety. Indeed, the number of collisions and the number of times the experimenter had to take the control away from the patient are significantly lower with SC. These losses of control happened when the wheelchair collided with a wall or when the patient was unable to maneuver the wheelchair out of a complicated position. Both metrics thus reflect that the safety is improved with SC. Furthermore, the reduced number of control losses shows that the patient is more autonomous with SC than without it as an expert needs to be present in the absence of SC. Furthermore, the time to perform the task was reduced, showing that increased safety and autonomy did not come at the cost of efficiency. Unlike other solutions proposed in the literature [38], we did not modify the wheelchair electronics or chassis, our solution can be used on all power wheelchairs on sale using a kit.

One main limitation of our study is the nature of the obstacles that were white walls starting from the ground. The assistance proved useful for the avoidance of these obstacles, however they do not represent well all obstacles present in a home environment like chairs or table legs. Furthermore, the sensors mounted on our solution cannot be used with all kinds of materials like glass or other reflective materials that hinder a correct use of these infra-red ToF sensors. The use of other sensors could be envisioned to improve our solution. Another limitation of our study is that for safety and medical reasons, the patient was not in the wheelchair that he controlled but in his own on the side of the circuit. While the patient could see the entirety of the circuit, it is likely that it complicated a bit the BCI-control of the wheelchair as he had to do mental spatial transformations to choose the appropriate BCI commands. Non-colocation is known to hinder control [39]. Although mounting a first-person view camera into the wheelchair and control it via a monitor could have provided this colocation, it could also have reduced the peripheral and depth vision of the patient and added a supplementary delay. Since the BCI decoding latency was not very different in wheelchair control than in robotic arm control, we believe that control complexity was comparable in both experiments.

### 4.2. Reach-and-grasp assistance

A sensor-based SC method was implemented to assist the BCI-control of the robotic manipulator for a reach-and-grasp task. The SC algorithm blends the BCI control with an eye-in-hand depth-sensor-based autonomous control.

We have shown that SC helped the patient based on several criteria. First, we showed that the time required to grasp the object and to complete the entire task was reduced with the assistance. Secondly, we showed that the number of changes of mental tasks
was reduced with SC. This decrease implies a reduced cognitive load, with SC, which is important for the acceptability of the device as a high cognitive load implies fatigue and the impossibility or difficulty to perform concurrent cognitive tasks. A study with concurrent cognitive tasks could be interesting to shed more light on the cognitive load associated to the use of our solution. As for the number of gripper commands necessary to grasp the object and release it, it was reduced with SC. Unwanted releases of the object that are avoided with the assistance are time and energy-consuming and may be followed by falls of the object.

For the sensor-based servoing, we used the barycenter coordinates of the cluster closest to the end-effector as visual features. No identification of the object captured is performed, the method is thus low-cost in terms of computation and does not require any *a priori* knowledge of the object. The proposed method allows thus to grasp unknown and new objects contrary to other SC approaches that use libraries of objects templates [7, 14, 23] or tags on the objects [4, 18]. In our study, we chose to use only one object to evaluate the task in order not to increase the number of trials as the experiment already lasted around 14/h. Selecting the closest object as target also allows to change the targeted object by moving the end-effector in front of another object as the visual features are updated constantly contrary to most studies who use SSVEP to select the object to be grasped [25-27]. However, the absence of object recognition implies that obstacles or limbs are not differentiated from objects to grasp which could lead to safety issues. As the object is represented as a 3D-point, the shape or main orientation of the object is not captured and the orientation of the gripper is not controlled to best grasp the object given its shape. The extraction of different features from the sensors or a shape recognition as in [40] or [41] could be investigated to allow a better grasping orientation and to increase the safe use of this method. The fusion of BCI and sensor based velocities is a function of the distance of the end-effector to the object with a constant update of the targeted object. As long as there is no major direction change from the user, this function allows a progressive authority transfer from the user to the autonomous control. This transfer is not abrupt contrary to studies where the control is switched between the user and the autonomous control [14, 21, 30] or to studies with piece-wise functions [5]. This kind of SC leaves a lot of control to the user which is crucial for the acceptance of the device by users [12]. However, once close to the object, when the control is mostly sensor-based, the user can remain inactive. The distance of the end-effector to the closest object is the only factor regulating the weighting of BCI and sensor-based velocities. Once the autonomous phase is entered, the user cannot intervene and abort the grasp. Cao et al in

[23] explain that not asking for confirmation before grasping the object renders the control more natural and more similar to an ecological one. However, even before entering the autonomous area, it becomes impossible to change drastically the trajectory as the sensor-based control is given more weight after a certain distance is passed (middle of the SC area). This can be problematic if the user wants to change target close to an object or gets close to an obstacle by mistake or on the way to an object.

Only 2 degrees of freedom (forward-downward, left-right) are controlled by the patient. The third axis, up-down in our case, and the orientation of the gripper are not controlled. This reduces the workspace, preventing the grasping of objects or the avoidance of obstacles in the environment. The homeenvironment is usually cluttered, even with a good BCI control, the use of the robotic manipulator could benefit from an obstacle avoidance assistance to improve the safety of its use.

### 4.3. BCI performances and limitations

Each condition was repeated several times but only one patient performed the experiments. The BCI capabilities vary between individuals [42] and this variability could have an impact on the effect of the assistance. Indeed, another patient included in the clinical trial had control over 8 DoF [10]. Furthermore, while the patient could take as many breaks as he wanted, the use of a BCI interface is cognitively demanding and it is possible that fatigue impacted the results. Only one implant of the patient was used for these experiments, it is likely that it influenced the performances of the patient. Indeed, the tasks controlateral to the implant, namely grasp and right direction, are better decoded than the only one on the other side. Nevertheless, the patient presents good BCI performances, as highlighted in the results. Indeed, the accuracy is high for all mental states as can be seen on the confusion matrices. For the wheelchair driving, there are still confusions between states and the number of collisions and losses of control highlight the need of an assistance for power wheelchair driving tasks even-though the accuracy is high. As for the reach-and-grasp task, the confusion is mostly with the idle state. This promotes a safer use of the robotic manipulator since it ensures that the robotic manipulator remains still when a prediction error is done. However, this likely increases the time to perform the task. These differences between the wheelchair driving and reach-and-grasp models most likely come from a difference in the training. Indeed, to improve control performances of the subject, the transition between moving states and idle were favoured during training of the decoding model of the robotic manipulator, while this was not the case with the wheelchair. The transition probability between moving states was then lower with the robotic manipulator. It is also possible that the patient improved
his motor imagery with time as the reach-and-grasp experiment was performed later. Although the grasp mental task was correctly identified in more than 73% of the time samples, backward and idle sates were sometimes wrongly classified into grasping, leading to several unwanted grasps. This highlights the need of an assistance for reach-and-grasp tasks.

In this study, we presented SC methods to improve the autonomy of quadriplegic individuals. The tasks chosen, the control of a power wheelchair and the control of a robotic manipulator for reach-and-grasp tasks were studied separately in laboratory settings eventhough the motor imagery mental tasks were the same except for the gripper command. It would be interesting to mount the robotic manipulator on the wheelchair and to test the two of them together in a home environment.

## 5. Conclusion

This article presents a sensor-based SC framework to assist the BCI teleoperation of assistive devices for mobility and object manipulation tasks at home. The assistance for mobility is an obstacle avoidance assistance for power wheelchair driving and a reach-and-grasp assistance for object manipulation. Our study assessed the assistance provided by comparing the performance of a power wheelchair driving task and a reach-and-grasp task with and NSC assistance in a controlled environment. The reduced duration to perform the circuit with the wheelchair, the reduced number of collisions and the reduced number of time an expert took the control away from the patient show that the performance of the patient improved with SC and that it was accompanied by an increased safety. The reduced time needed to perform a reach-and-grasp task, the avoidance of unwanted opening and closing of the gripper and the reduced cognitive load show that the performance of the patient was improved with SC for a reach-and-grasp while allowing the grasping of unknown objects. The next steps are (1) to test the assistance of the power wheelchair and robotic manipulator combined and to (2) render the reach-and-grasp safer thanks to the assistance.

## Data availability statement

The data that support the findings of this study are available upon reasonable request from the authors. The data cannot be made publicly available upon publication because they contain sensitive personal information.

## Acknowledgments

This work was supported by Institut Carnot Leti—Inria (Intercarnot MOVE@HOME Project), Fonds de dotation Climatec (BCI4GRASP Project), The French Ministry of Health (Grant PHRC-15-15-0124). The

authors would like to thank STMicroelectronics for providing support and for supplying the depth sensors used in the clinical trial.

Authors confirm that consent to appear in the videos has been obtained from the participant and that he is aware of intended publication.

## Ethical statement

The research was conducted in accordance with the principles embodied in the Declaration of Helsinki and in accordance with local statutory requirements. The studies involving human participants were reviewed and approved by National Agency for the Safety of Medicines and Health Products (Agence nationale de sécurité du médicament et des produits de santé: ANSM), registration number 2015-A00650-49, and the Committee for the Protection of Individuals (Comité de Protection des Personnes-CPP), registration number 15-CHUG-19. The patients/participants provided their written informed consent to participate in this study and for publication.

## ORCID iDs

Angélina Bellicha https://orcid.org/0000-0002-6628-3002

Lucas Struber https://orcid.org/0000-0002-3890-9100

Louise Devigne https://orcid.org/0000-0002-9109-545X

Marie Babel https://orcid.org/0000-0001-6425-389X

Guillaume Charvet https://orcid.org/0000-0003-4938-9419

## References

[1] Furlan J C, Sakakibara B M, Miller W C and Krassioukov A V 2013 Global incidence and prevalence of traumatic spinal cord injury *Can. J. Neurol. Sci.* **40** 456–64

[2] Emami Z and Chau T 2020 The effects of visual distractors on cognitive load in a motor imagery brain-computer interface *Behav. Brain Res.* **378** 112240

[3] Chen X, Zhao B, Wang Y, Xu S and Gao X 2018 Control of a 7-DOF robotic arm system with an SSVEP-based BCI *Int. J. Neural Syst.* **28** 1850018

[4] Zhou Y, Yu T, Gao W, Huang W, Lu Z, Huang Q and Li Y 2023 Shared three-dimensional robotic arm control based on asynchronous BCI and computer vision *IEEE Trans. Neural Syst. Rehabil. Eng.* **31** 3163–75

[5] Xu Y, Zhang H, Cao L, Shu X and Zhang D 2022 A shared control strategy for reach and grasp of multiple objects using robot vision and noninvasive brain-computer interface *IEEE Trans. Autom. Sci. Eng.* **19** 360–72

[6] Zhang W, Sun F, Wu H, Tan C and Ma Y 2019 Asynchronous brain-computer interface shared control of robotic grasping *Tsinghua Sci. Technol.* **24** 360–70

[7] Downey J E, Weiss J M, Muelling K, Venkatraman A, Valois J S, Hebert M, Bagnell J A, Schwartz A B and Collinger J L 2016 Blending of brain-machine interface and vision-guided autonomous robotics improves
neuroprosthetic arm performance during grasping *J. NeuroEng. Rehabil.* **13** 28

[8] Muelling K, Venkatraman A, Valois J S, Downey J E, Weiss J, Javdani S, Hebert M, Schwartz A B, Collinger J L and Bagnell J A 2017 Autonomy infused teleoperation with application to brain computer interface controlled manipulation *Auton. Robots* **41** 1401–22

[9] Rousche P J and Normann R A 1998 Chronic recording capability of the utah intracortical electrode array in cat sensory cortex *J. Neurosci. Methods* **82** 1–15

[10] Moly A et al 2022 An adaptive closed-loop ECoG decoder for long-term and stable bimanual control of an exoskeleton by a tetraplegic *J. Neural Eng.* **19** 026021

[11] Abbink D A, Carlson T, Mulder M, De Winter J C, Aminravan F, Gibo T L and Boer E R 2018 A topology of shared control systems-finding common ground in diversity *IEEE Trans. Human-Mach. Syst.* **48** 509–25

[12] Argall B D 2018 Autonomy in rehabilitation robotics: an intersection *Technical Report*

[13] Devigne L, Narayanan V K, Pasteau F and Babel M 2016 Low complex sensor-based shared control for power wheelchair navigation *IEEE Int. Conf. on Intelligent Robots and Systems* vol 2016 pp 5434–9

[14] Xu Y, Ding C, Shu X, Gui K, Bez sudnova Y, Sheng X and Zhang D 2019 Shared control of a robotic arm using non-invasive brain-computer interface and computer vision guidance *Robot. Auton. Syst.* **115** 121–9

[15] Jain S, Farshchiansadegh A, Broad A, Abdollahi F, Mussa-Ivaldi F and Argall B D 2015 *Assistive Robotic Manipulation Through Shared Autonomy and a Body-Machine Interface*. (Piscataway, NJ: IEEE)

[16] Wang Y, Xu G, Song A, Xu B, Li H, Hu C and Zeng H 2018 Continuous shared control for robotic arm reaching driven by a hybrid gaze-brain machine interface *IEEE Int. Conf. on Intelligent Robots and Systems* pp 4462–7

[17] Johnson G D, Waytowich N R, Cox D J and Krusienski D J 2010 Extending the discrete selection capabilities of the P300 speller to goal-oriented robotic arm control *2010 3rd IEEE RAS and EMBS Int. Conf. on Biomedical Robotics and Biomechatronics, BioRob 2010* pp 572–5

[18] Ding D, Styler B, Chung C-S and Houriet A 2022 Development of a vision-guided shared-control system for assistive robotic manipulators *Sensors* **22** 4351

[19] Kim H K, Biggs S J, Schloerb D W, Carmena J M, Lebedev M A, Nicolelis M A and Srinivasan M A 2006 Continuous shared control for stabilizing reaching and grasping with brain-machine interfaces *IEEE Trans. Biomed. Eng.* **53** 1164–73

[20] Qasim M and Ismael O Y 2022 Shared control of a robot arm using BCI and computer vision *J. Eur. des Syst. Automat.* **55** 139–46

[21] Gillini G, Lillo P D and Arrichiello F 2021 An assistive shared control architecture for a robotic arm using EEG-based BCI with motor imagery *IEEE Int. Conf. on Intelligent Robots and Systems* pp 4132–7

[22] Muelling K, Venkatraman A, Valois J S, Downey J, Weiss J, Javdani S, Hebert M, Schwartz A B, Collinger J L and Bagnell J A 2015 Autonomy infused teleoperation with application to BCI manipulation (arXiv: 1503.05451)

[23] Cao L, Li G, Xu Y, Zhang H, Shu X and Zhang D 2021 A brain-actuated robotic arm system using non-invasive hybrid brain-computer interface and shared control strategy *J. Neural Eng.* **18** 046045

[24] Jain S and Argall B D 2018 Recursive Bayesian human intent recognition in shared-control robotics *IEEE Int. Conf. on Intelligent Robots and Systems* pp 3905–12

[25] Chen X, Zhao B, Wang Y and Gao X 2019 Combination of high-frequency SSVEP-based BCI and computer vision for controlling a robotic arm *J. Neural Eng.* **16** 026012

[26] Yang C, Wu H, Li Z, He W, Wang N and Su C Y 2018 Mind control of a robotic arm with visual fusion technology *IEEE Trans. Ind. Inf.* **14** 3822–30

[27] Tang J and Zhou Z 2017 A shared-control based BCI system: for a robotic arm control *1st Int. Conf. on Electronics Instrumentation and Information Systems, EIIS 2017* vol 2018 pp 1–5

[28] Du S, Wang F, Zhou G, Li J, Yang L and Wang D 2021 Vision-based robotic manipulation of intelligent wheelchair with human-computer shared control *Proc. of the 33rd Chinese Control and Decision Conf., CCDC 2021* pp 3252–7

[29] Sun F, Zhang W, Chen J, Wu H, Tan C and Su W 2019 Fused fuzzy Petri nets: a shared control method for brain-computer interface systems *IEEE Trans. Cog. Developmental Syst.* **11** 188–99

[30] Katyal K D et al 2014 A collaborative BCI approach to autonomous control of a prosthetic limb system *IEEE Int. Conf. on Systems, Man and Cybernetics*

[31] Gopinath D, Jain S and Argall B D 2017 Human-in-the-loop optimization of shared autonomy in assistive robotics *IEEE Robot. Autom. Lett.* **2** 247–54

[32] Mestais C S, Charvet G, Sauter-Starace F, Foerster M, Ratel D and Benabid A L 2015 WIMAGINE: wireless 64-channel ECoG recording implant for long term clinical applications *IEEE Trans. Neural Syst. Rehabil. Eng.* **23** 10–21

[33] Larzabal C, Bonnet S, Costecalde T, Auboiroux V, Charvet G, Chabardes S, Aksenova T and Sauter-Starace F 2021 Long-term stability of the chronic epidural wireless recorder WIMAGINE in tetraplegic patients *J. Neural Eng.* **18** 056026

[34] Eliseyev A, Auboiroux V, Costecalde T, Langar L, Charvet G, Mestais C, Aksenova T and Benabid A-L 2017 Recursive exponentially weighted N-way partial least squares regression with recursive-validation of hyper-parameters in brain-computer interface applications *Sci. Rep.* **7** 1–15

[35] Fraudet B, Leblong E, Piette P, Nicolas B, Devigne L, Babel M, Pasteau F, Routhier F and Gallien P 2023 SWADAPT2: benefits of a collision avoidance assistance for powered wheelchair users in driving difficulty *Disability Rehabil. Assist. Technol.* **19** 1907–15

[36] Chaumette F and Hutchinson S 2006 Visual servo control, part I: basic approaches *IEEE Robot. Autom. Mag.* **13** 82–90

[37] Benabid A L et al 2019 An exoskeleton controlled by an epidural wireless brain-machine interface in a tetraplegic patient: a proof-of-concept demonstration *Lancet Neurol.* **18** 1112–22

[38] Sivakanthan S, Candiotti J L, Sundaram A S, Duvall J A, Joseph J, Sergeant G, Cooper R, Satpute S, Turner R L and Cooper R A 2022 Mini-review: robotic wheelchair taxonomy and readiness Sivashankar *Neursci. Lett.* **772** 136482

[39] Saint-Aubert J, Heurley L P, Morgado N, Régnier S and Haliyo S 2020 Pre-Calibrated visuo-haptic co-location improves execution in virtual environments *IEEE Trans. Haptics* **13** 588–99

[40] Jain S and Argall B D 2016 Grasp detection for assistive robotic manipulation *Proc. - IEEE Int. Conf. on Robotics and Automation* vol 2016 pp 2015–21

[41] Zhang W, Sun F, Liu C, Tan C, Liu S and Su W 2017 A hybrid EEG-based BCI for robot grasp controlling *2017 IEEE Int. Conf. on Systems, Man and Cybernetics, SMC 2017*, 2017-Janua (Institute of Electrical and Electronics Engineers Inc) pp 3278–83

[42] Vidaurre C and Blankertz B 2010 Towards a cure for BCI illiteracy *Brain Topography* **23** 194–8