# Assistive Robotic Technologies for Next-Generation Smart Wheelchairs: Codesign and Modularity to Improve Users' Quality of Life

Fabio Morbidi, Louise Devigne, Catalin Stefan Teodorescu, Bastien Fraudet, Emilie Leblong, Tom Carlson, Marie Babel, Sarah Delmas, Guillaume Caron, Francois Pasteau, et al.

► To cite this version:

Fabio Morbidi, Louise Devigne, Catalin Stefan Teodorescu, Bastien Fraudet, Emilie Leblong, et al. "Assistive Robotic Technologies for Next-Generation Smart Wheelchairs: Codesign and Modularity to Improve Users' Quality of Life." IEEE Robotics and Automation Magazine, 2023, 30 (1), pp.24-35. 10.1109/MRA.2022.3178965. [hal-03709574](https://hal.science/hal-03709574)

HAL Id: hal-03709574

https://hal.science/hal-03709574

Submitted on 20 Sep 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Assistive Robotic Technologies for Next-Generation Smart Wheelchairs: Codesign and Modularity to Improve Users' Quality of Life

By Fabio Morbidi, Louise Devigne, Catalin Stefan Teodorescu, Bastien Fraudet, Émilie Leblong, Tom Carlson, Marie Babel, Guillaume Caron, Sarah Delmas, François Pasteau, Guillaume Vailland, Valérie Gouranton, Sylvain Guégan, Ronan Le Breton, and Nicolas Ragot

See affiliations pp. 13-14

This article describes the robotic assistive technologies developed for users of electrically powered wheelchairs, within the framework of the European Union's Interreg ADAPT (Assistive Devices for Empowering Disabled People Through Robotic Technologies) project. In particular, special attention is devoted to the integration of advanced sensing modalities and the design of new shared control algorithms. In response to the clinical needs identified by our medical partners, two novel smart wheelchairs with complementary capabilities and a virtual reality (VR)-based wheelchair simulator have been developed. These systems have been validated via extensive experimental campaigns in France and the United Kingdom.

## Introduction

### Motivation and Original Contributions

Assistive robotics is playing an increasingly important role in our aging society. In fact, robotic technologies are gaining

ground in medical applications for the design of new rehabilitation devices and personal mobility aids. In particular, electrically powered wheelchairs are among the most popular and powerful personal mobility aids in use today [1]. However, driving a power wheelchair safely requires the use of residual motor skills as well as sufficient cognitive and visuospatial abilities. Unfortunately, a significant number of people with disabilities are unable to operate a wheelchair on their own due to unsafe driving.

According to the World Health Organization, in 2018, 75 million people worldwide needed a wheelchair, but only 5–15% of those in need had access to one [31]. To attain equitable access to assisted mobility, it is therefore imperative to design new technical aids to compensate for any deficiencies while relying on the skills of each individual. Robotic assistance for driving a power wheelchair is hence an indispensable tool for people's independence. Based on this observation, scientists and clinicians have jointly addressed the issue of technical assistance and its place in the rehabilitation process. The first cause of the abandonment of electric wheelchairs is the risk of collision, which can affect the users and their environment. A flexible trajectory correction that

<footnote>Digital Object Identifier 10.1109/MRA.2022.3178965</footnote>

<footnote>Date of current version: 24 June 2022</footnote>
can be adapted to the needs and habits of the users is then necessary. The design of such a device requires the implementation of *shared control* solutions [2]–[4] to both respect the user’s intention and achieve an acceptable behavior. To engage in shared control enables adjustment for noisy and unpredictable signals as well. Moreover, in the rehabilitation process, it is important that the users understand how the help is provided so that they can correct gestures and behavior on their own. Assisted driving can thus be employed to hone the user’s perception of the surrounding environment and situations encountered to raise awareness of the level of assistance provided. In this context, *multisensory feedback* can be usefully coupled with the shared control system to offload some of the control burden.

Finally, learning to drive a power wheelchair can be a frustrating experience, which is discouraging for people whose impairments overly affect their ability to maneuver safely. If the training, despite the aids provided, is not successful, people may thus be prevented from using the wheelchair. Conversely, with repeated sessions that tackle progressively more challenging scenarios, improvements can often be observed. However, health-care institutions and medical device companies work under strict time and budgetary constraints such that they do not always have the resources to extend the learning process. In addition, the risks taken during the driving sessions often dissuade the accompanying teams from continuing the experience. For all these reasons, *VR-based driving simulators* have recently garnered attention as a viable alternative for offline learning of wheelchair control [5]. For example, a wheelchair user can repeat the same training

circuit, under exactly the same conditions, as many times as the clinician deems it necessary. This saves time and resources while maintaining safety and improving objective outcomes.

This article presents the main results of our research program contributing to the development of a robotic wheelchair with built-in assistive features, which responds to the specific needs of actual users in their everyday life. All the aspects of this program are covered, from omnidirectional vision and haptic communication to the design of a VR-based driving simulator along with a suite of sensor fusion and shared control algorithms for two complementary smart wheelchairs. The research leading to these results has received funding from the Interreg VA France (Channel) England ADAPT project (https://adapt-project.com/english). The original consortium included 14 partners from French and English research laboratories and medical institutions. The goal of the project, driven by the real needs of occupational therapists and specialists in rehabilitation medicine, was to design, develop, and evaluate innovative assistive technologies.

The *bottom-up*, *human-centric*, and *collaborative* approach advocated in this article has the advantage of providing flexible solutions that adapt to a broad class of user impairments and types of environments (indoor/outdoor and structured/unstructured). The preferences and priorities have been identified by our clinical partners, and they have been translated into a range of functional requirements and technical specifications (see Figure 1 for an excerpt). On this basis, we have devised five standardized obstacle courses of growing complexity, which have been used during our clinical trials [6]. They cover a fairly large spectrum of maneuvers and real-life

<figure><img src="image_2.png" /><figcaption>**Figure 1.** The functional analysis. Thanks to the collaboration with our medical partners, user needs and preferences have been translated into a set of functional requirements. Each parent function has a number of subfunctions, some of which already have well-known solutions (e.g., "F1a*: positive obstacle detection"), whereas others remain open research questions, and thus they have been explored in greater depth within the ADAPT project (e.g., "F1b: negative obstacle detection"). The (a) functional requirements for the smart wheelchair and (b) additional functional requirements for the simulator.</figcaption></figure>
situations, such as corridor following, entering and reversing out of an elevator, moving up a slope, descending a curb ramp, and so on.

## Related Work

We are aware that we are not the first to adopt a *codesign principle* to guide the development of new assistive robotic technologies (see, e.g., [7] in a recent issue of this magazine). Over the past decade, numerous smart wheelchairs have been proposed to target different types of usage [8], [9] and different categories of patients [10], [11]. However, these works are mainly concerned with the transfer of sensing technologies and control algorithms originally developed in mobile robotics. Other research groups have dealt with *specific usability* [12], *ergonomic* [13], and *safety and accessibility issues* [14]. On the other hand, while VR-based wheelchair simulators are known to offer new opportunities for training thanks to their flexibility, safety, and guaranteed repeatability [15], we are still far from a realistic and comfortable experience for the user, with a high sense of presence and low levels of cybersickness.

Hence, to this day, there still exists a significant gap between the expectations of wheelchair users and off-the-shelf assistive devices. The ambition of the ADAPT project was to bridge this gap in the literature and take a step forward toward an ecosystem of modular, strap-on assistive solutions tailored to meet the individual requirements of the end users. Through the prism of our personal experience in the field, our aim herein is to provide a concise description of these

solutions and assess the progress made so far. For further details on the technical aspects, the interested reader is referred to our previous publications in the "References" section.

## Instrumented Power Wheelchairs

The hardware architecture put forward in the ADAPT project is characterized by diversity in terms of sensing and onboard computation (different sensor specifications, types of microcontrollers, and so on). On the other hand, the software architecture is unified, and it relies on Robot Operating System (ROS) as middleware. This "lingua franca" enables the interchange of multiple hardware and software components, which can be tested and shared among project partners before being integrated during the clinical trials.

In what follows, we present the wheeled platforms developed in France and the United Kingdom and the wheelchair simulator.

## Wheeled Platforms

The Aspire Create group at University College London (UCL) has developed a smart wheeled platform by instrumenting a Sunrise Medical Quickie Salsa M² power wheelchair with custom and off-the-shelf electronics (see Figure 2). The midwheel-drive platform has six wheels with independent suspensions, is endowed with a curb-climbing ability for heights up to 7 cm, measures 61 cm at the widest point, and has 60-Ah batteries that can propel it up to 10 km/h. An inertial measurement unit (IMU) SparkFun 9DoF

Razor, which includes a three-axis accelerometer, gyroscope, and magnetometer, has been installed under the driver's seat. Industrial wheel encoders (Kubler 500 ppr) together with 3D-printed pulleys have been placed in the narrow space between the main drive wheels and the chassis of the wheelchair to obtain measurements from odometry. Twelve ultrasonic sensors (SRF08) have been installed in four custom housings in the corners of the chassis of the wheelchair. Each housing contains three ultrasonic sensors covering a theoretical angle of 135°, where obstacles can be detected. Electric current sensors and voltage measurements are used to monitor the electric power flow through the two motors. Finally, a single-board computer (Raspberry Pi 3B+) acts as the ROS master, using a publisher-subscriber model. We refer the reader to [16] for more details on all these components, including the schematics of the hardware architecture.

<figure><img src="image_3.png" /><figcaption>**Figure 2.** A side view of the instrumented wheelchair with added bespoke electronics, developed at UCL.</figcaption></figure>
The wheeled platform developed at the Institut National des Sciences Appliquées (INSA) Rennes also builds on the Quickie Salsa M² wheelchair. It is equipped with 48 time-of-flight (ToF) sensors organized in seven modules distributed along its perimeter: six modules of six sensors are located on each side and under the footplates, and one module of 12 sensors is installed behind the backrest (see Figure 3). The ST VL53L1X sensors have the following technical specifications: distance measurement up to 4 m, ranging frequency up to 50 Hz, typical full field of view of 27°, and size of 4.9 × 2.5 × 1.56 mm³. Their measurements have been used to directly detect positive obstacles (doors, walls, and so on) around the wheelchair or to infer the presence of negative obstacles (potholes, inclines, drop-offs/steps, and so forth). The range measurements are also combined with the visual information coming from an overhead omnidirectional camera (see the “Advanced Sensing: Omnidirectional View” section for more details).

## Wheelchair Simulator

INSA Rennes has also been involved in the design of an immersive wheelchair simulator, which has been manufactured by CL Corp. (www.clcorporation.com). The simulator consists of a mechanical platform equipped with an adjustable wheelchair seat and wheelchair electronics. The mechanical platform relies on a D-Box system (five actuators and associated electronics), and it has been designed to be as close as possible to the standard dimensions of a power wheelchair to enhance the immersive experience. The actuators provide four degrees of freedom, pitch, roll, yaw, and heave. The platform can accommodate the seat and electronic modules of any commercial power wheelchair (in our case, we used those

of the Quickie Salsa M²). As a result, the user can control the simulator with standard interfaces, such as a joystick. Moreover, the same velocity and acceleration driving profiles as those provided by the real wheelchair can be delivered. The communication between the virtual environment and simulator is ensured by ROS, which makes it readily compatible with any existing VR engine. The simulator provides a first-person perspective and currently integrates vestibular feedback to reproduce the motion sensations experienced on a real wheelchair [17], but it does not take anticipatory action to predict the user's behavior (e.g., the platform does not tilt in advance of when the driver is about to negotiate a curve).

Our 3D test environments include an indoor, maze-like obstacle course [18] conceived by our clinical partners and a full-scale model of a city square [17]. The two environments have been created with the Unity Real-Time Development Platform, and they can be displayed using different interfaces: a standard monitor, a head-mounted display (as in [5]), and a pair of 3D glasses in an immersive room (www.irisa.fr/immersia), as shown in Figure 4.

<figure><img src="image_4.png" /><figcaption>**Figure 3.** The wheeled platform developed at INSA Rennes (top view). The ID and location of the 48 ToF sensors mounted on the wheelchair are shown. The seven sensor modules are depicted in green.</figcaption></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /><figcaption>**Figure 4.** A wheelchair simulator tested by a volunteer in immersive conditions. In (a), the user wears a head-mounted display, and in (b), the user wears 3D glasses in *Immersia*, a virtual-reality research platform at IRISA/Inria Rennes.</figcaption></figure>
Advanced Sensing: Omnidirectional Vision

Twin-fisheye cameras are compact visual sensors that capture
high-resolution, 360° images and videos. The classical design
(known as a “symmetrical dual fisheye lens”) includes two
fisheye lenses pointing in opposite directions and two prisms
that direct the light rays to two photosensitive elements [see

Figure 5(a)]. The dual-lens panorama design was introduced by Ricoh, in 2013 (Theta series), and it has been adopted by several other camera manufacturers over the past 10 years, e.g., in the Insta360, Samsung Gear360, Adventure 360, Nikon KeyMission 360, and Garmin Virb 360.

A twin-fisheye camera (Ricoh Theta S) installed on a mast overhead behind the user is the “Swiss Army knife” of sensors on INSA’s smart wheelchair. In fact, it is used for *driving assistance* (together with an array of ToF sensors) and *3D scene reconstruction* (for use in the wheelchair simulator or in an image-based localization module). In what follows, we provide further details on these two functionalities, which are relevant to navigate unknown, indoor, GPS-denied environments and to train novice wheelchair users.

The University of Picardie Jules Verne (UPJV) and INSA
Rennes have recently codeveloped SpheriCol [19], a new
driving assistance system for power wheelchairs (see Fig-
ure 6). Similar to the parking assistant of modern cars, Spheri-
Col improves situation awareness by overlaying color-coded
range measurements from a ring of ToF sensors (see Figure 3)
on a stream of 360° images of the surrounding environment
provided by the Ricoh Theta camera.

The sequence of images captured by the twin-fisheye cam-
era during the displacement of the wheelchair is also used for
offline 3D scene reconstruction by spherical photogramme-
try. In this way, a digital “twin” of the real environment (with
the same appearance and proportions) can be easily generat-
ed. For spherical photogrammetry, the dual-fisheye images
from the Ricoh Theta are transformed into equirectangular
images and fed into Agisoft Metashape, which yields dense,
colored 3D point clouds with colors of photographic quality
[see Figure 5(b)].

If the camera pose relative to each image of the sequence is
known, the resulting 3D reconstruction tends to be more
accurate, and the computational cost is significantly reduced.
In a classical data processing pipeline, Metashape relies on
GNSS (Global Navigation Satellite System) measurements,
which are typically available outdoors but not indoors. To
address this issue in indoor environments, we first generated
a sparse 3D model with the associated camera poses using
OpenVSLAM [20], an off-the-shelf software package for visu-
al simultaneous localization and mapping. These poses are
then given as input to Metashape. The trajectory of the wheel-
chair estimated with OpenVSLAM and an external motion
capture system (see the “Driving Assistance” section) were
used to assess the quality of the driving assistance provided by
SpheriCol [21].

OpenVSLAM may fail if the interframe motion between
successive images in a sequence is large, which might result,
for example, from an abrupt change in the joystick position.
Hence, it cannot be directly used online to assist wheelchair
users. In [22], we overcame this limitation by proposing a
new, accurate direct visual gyroscope, which copes with large
interframe motions. Based on the mixture of photometric
potentials, it takes the spherical images from a twin-fisheye
camera as input and provides an estimate of its 3D

<figure><img src="image_7.png" /><figcaption>**Figure 5.** (a) A dual-fisheye image captured by a Ricoh Theta S camera. (b) An image-based 3D reconstruction of the obstacle course at Pôle Saint-Hélier (a physical medicine and rehabilitation center in Rennes), obtained with Metashape. The ceiling has been removed to provide visibility of the room interior.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>**Figure 6.** The basic system components of SpheriCol. The (a) twin-fisheye camera, (b) wheelchair equipped with a ring of ToF sensors, and (c) user interface displaying a panoramic image of the surrounding environment, with colored distance markers overlaid.</figcaption></figure>
orientation with respect to a reference image (typically the initial one). In our experiments, we observed no performance degradation for reference images captured up to several tens of meters away and for rotational displacements of a few tens of degrees. To quantitatively evaluate the performance of our visual gyroscope and other state-of-the-art vision-based ego-motion estimation algorithms, a data set of omnidirectional images captured by catadioptric and twin-fisheye cameras mounted on different robotic platforms, PanoraMIS [23], has been made publicly available. In particular, Sequence 7 of the data set (1.35 km, in whole) comes with an accurate ground truth provided by an Adept Mobile-Robots Seekur Jr. robot (with integrated and external IMUs, GPS measurements, and wheel odometry). This robot was chosen since its footprint is comparable to that of a standard power wheelchair.

## Shared Control

Shared control is a concept involving collaboration between a human and a machine [2], [3]. The human expresses an intention that the machine facilitates and implements in an optimal way. The assist-as-needed paradigm provides assistance only when required, providing the user with as much control authority as possible. This concept is a key emerging technology with wide applicability in medical robotics.

Two modular and complementary shared control strategies have been proposed in the ADAPT project to cater for the wide variety of user needs: the first strategy is *sensor based* and the second is *model based*, as illustrated in Figure 7. Both strategies are compatible with new vibration-based human-machine interfaces, as detailed in the “Human-Machine Interaction and Haptic Feedback” section.

## Sensor-Based Shared Control

The sensor-based shared control method developed at INSA Rennes relies on a simple geometric algorithm that can be easily implemented on low-cost embedded devices with limited computational resources [24]. The algorithm makes use of measurements from any type of range sensor on the wheelchair and leverages the distance constraints to compute two areas in the velocity space, which correspond to the input velocities of the wheelchair that are safe (allowed area) or unsafe (forbidden area). The shared control blends the user's input and algorithm's output to ensure safe and smooth wheelchair navigation. This obstacle avoidance solution is robust: the user has full authority over the wheelchair when the input is safe and benefits from progressive assistance during difficult maneuvers (e.g., reversing out of an elevator).

## Model-Based Shared Control

While the dynamics of a power wheelchair can be precisely characterized and have been widely used for control design in the literature, it is challenging to combine the capabilities of a machine (as described by its dynamic model) with the limited unpredictable information coming from 1) the environment and 2) the human user (e.g., the joystick interface is a projection of the user's intention). To compensate for the limited and incomplete information available from real-time (online) measurements, recent research has explored stochastic models.

One way of implementing a stochastic model is to use probabilistic shared control [25]. However, this technique incurs considerable computational cost to generate possible wheelchair trajectories, which may preclude its use in real-time applications. To circumvent this limitation, in [26], UCI's group proposed to use stochastic dynamic programming

<figure><img src="image_9.png" /><figcaption>**Figure 7.** The shared control working principle. Using information from the environment and driver, restrictions (red-shaded area) are created in the joystick plane, yielding safe linear and angular velocities ($v_{joy}, \omega_{joy}$) for the wheelchair.</figcaption></figure>
(SDP), which takes all the computation burden offline. The outcome is a lookup table that can be readily used online by the assist-as-needed algorithm. More specifically, in [26], a model-based control architecture has been introduced to solve the obstacle avoidance problem. It consists of four blocks, where some are deterministic and others are stochastic.

First, the *wheelchair dynamics* block includes the physical equations of motion of a two-wheeled differential drive vehicle (for the experimental model identification, see [16]). Second, the *environment* block is used to model the static obstacles, with the vehicle having limited knowledge of the global map. In fact, a local map around the wheelchair is built based on the distance measurements coming from an array of sensors (e.g., ultrasonic or laser sensors). Third, the *driver intention* block includes stochastic models of driver intention (e.g., an *expert driver* capable of maneuvering the wheelchair at high speed yet seldomly hitting obstacles, a *"blind" driver* for which the probability of hitting obstacles or avoiding them is the same, and a *naughty child* who intentionally advances at high speed toward the obstacles with the intention to hit them as a learning experience). Fourth, the *supervisory control* block computes optimal assist-as-needed actions specifically tailored to each driver model, which comes in the form of multidimensional lookup tables.

## Human-Machine Interaction and Haptic Feedback

The ADAPT project gave the French and English teams a great opportunity to design innovative human-machine interfaces. Among other devices, haptic interfaces have been conceived to assist users with wheelchair's navigation.

Two types of systems have been tested: a *haptic joystick* [24] and a *vibrotactile armband* [27] (see Figure 8). Both

devices can be easily interfaced with the control system of any consumer-grade wheelchair. While driving the wheelchair, a reactive force is applied by the haptic joystick to the hand of the user. By offering resistance in the direction of an obstacle, the user is thus indirectly informed about the safe trajectory to follow. However, the haptic joystick remains a simple decision support system which does not replace the driver, who is in full control of the wheelchair at all times. The armband can be worn anywhere on the upper and lower limbs, depending on the user's sensory capabilities. The armband is composed of four evenly spaced vibrotactile actuators, powered by a lithium-ion battery and controlled by an embedded wireless electronic board. The armband is inexpensive and provides intuitive commands (information about the path to follow or about the presence of obstacles, in the form of a direction with respect to the current orientation of the wheelchair). As a result, users do not need long training sessions.

The sensor-based shared control algorithm developed at INSA Rennes is compatible with the haptic feedback provided by the haptic joystick and the vibrotactile armband. The feedback is computed by processing range measurements coming from the wheelchair, and it supports the user during spatial navigation tasks. The haptic feedback can be employed in conjunction with the algorithm in [24] (progressive assistance while approaching an obstacle) or standalone (i.e., the control is not delegated, and the user has full authority over the wheelchair).

## Field Tests and Clinical Trials

### Clinical Evaluation

Experiments and regular roundtable sessions with patients, robotics experts, occupational therapists, and specialists in rehabilitation medicine have played a key role throughout the ADAPT project. In fact, if the former were necessary to validate the robotic solutions developed, the latter were essential to ensure that the specific needs of the patients were satisfactorily met. The research ethics committee approved the clinical studies, and informed consent was obtained from all participants. The codesign principle has been a guiding line through the project, and the comments and suggestions of the end users have been extremely helpful to improve their experience (e.g., by adjusting the height of a sensor, providing additional feedback, and delivering smoother acceleration profiles). The experimental protocols defined by the clinicians and roboticists have been adapted to fit users' experience (including novices, expert users, and people with disabilities who were not allowed to operate a power wheelchair), and scenarios of growing complexity have been proposed to the various participants during different sessions. The clinical trials turned out to be of paramount importance for mechanical/electronic testing and medical validation.

### Driving Assistance

The driving assistance solutions developed in the ADAPT project have been tested during two clinical trials: SWADAPT1

<figure><img src="image_10.png" /><figcaption>**Figure 8.** A volunteer wears a vibrotactile armband developed at INSA Rennes on his right upper arm while driving UCL's wheelchair.</figcaption></figure>
(NCT04072536) and SWADAPT2 (NCT04259151); see Figure 9(a) and (c). Subjects with neurological disorders participated in these clinical studies. The main objective was to assess their driving performance with and without assistance. To this end, we measured the number of collisions and total time to completion in three standardized circuits of increasing difficulty. The SWADAPT1 clinical trial involved 25 users with expert wheelchair driving skills. The results of this trial indicate that the proposed assistance solutions are accurate, risk averse, and safe, with a high degree of acceptability. Moreover, even if the participants were already expert drivers, the study has shown that the use of the assistance module statistically significantly reduced the number of collisions during complex maneuvers [6]. The protocol followed during the SWADAPT2 clinical trial was similar to that of SWADAPT1, but 28 users with driving difficulties took part in it. The results show a significant reduction in the number of collisions. Notably, the more challenging the obstacle courses are, the more useful the assistance is perceived to be. The benefits of assistance in terms of usage and self-confidence have been clearly demonstrated in SWADAPT2.

SpheriCol (see the “Advanced Sensing: Omnidirectional Vision” section) has been successfully tested with patients with cognitive disorders [19] and 17 able-bodied participants [21]. The circuit in Figure 9(b) was built in a large indoor environment (a gymnasium) and equipped with an overhead Qualisys motion capture system (eight Miquis cameras) to track the wheelchair during its displacement and obtain precise ground truth measurements for evaluation purposes. As depicted in Figure 10, which reports a statistical analysis of the answers to the questionnaire handed out to the 17 volunteers, SpheriCol received neutral to positive satisfaction and encouraging usability results from the majority of the participants. In particular, Figure 10(d) reports the percentage of the time SpheriCol was used by the participants in the test circuit. In addition, even though the sample size remains relatively small, 44% of the users engaged in our study stated that the video stream was one of the major strengths of the driving assistant, 16% appreciated the distance information provided by the colored markers, and 20% deemed the system helpful for reversing the wheelchair and for risk management (collision avoidance).

## Haptic Feedback

The haptic feedback has been evaluated with able-bodied participants, and the clinical trials with patients were still in progress at the time of writing. The joystick has been tested to provide a proof of concept, and the results of this study have been recently presented in [24]. On the other hand, the wearable haptic armband has been assessed with healthy participants in UCL's PAMELA (Pedestrian Accessibility Movement Environment Laboratory) facility. PAMELA is equipped with a modular platform that can be used to replicate gentle slopes (around 10°) and negative obstacles (with a roughly 30-cm drop). We constructed the circuit shown in Figure 11, which consists of static components (e.g., a door, a narrow

passageway, and an elevator) that have been identified as relevant for this case study by clinicians [6]. The circuit is composed of lightweight cardboard sheets to ensure participants' safety in case of collisions. The absolute position of the moving wheelchair is estimated with a vision system based on multiple cameras attached to the ceiling. The encouraging results of these trials, have been recently presented in [27].

## Training in VR

The wheelchair simulator described in the “Wheelchair Simulator” section provides a high-fidelity, 3D immersive

<figure><img src="image_11.png" /><figcaption>(a)</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(b)</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>(c)</figcaption></figure>

**Figure 9.** The test circuits considered during the clinical trials at (a) and (b) INSA Rennes, and at (c) Pôle Saint-Hélier.
environment, and it offers the possibility to repeat the same circuit multiple times under identical experimental conditions. The user gets the impression of driving a real wheelchair, and safe navigation is guaranteed at all times. Shorter training sessions are thus necessary, and a wider array of (indoor/outdoor, obstacle-free/cluttered) environments and real-life conditions (variable light conditions, moving pedestrians) can be tested.

Driving a real wheelchair could be dangerous for people with disabilities, requiring extensive training sessions to acquire the ability to move safely. The goal of the clinical trial SIMADAPT1 (NCT04171973) was to verify whether the performance observed on a real circuit was comparable to the one experienced on the wheelchair simulator. To this end, the wheelchair users were asked to complete the three obstacle courses considered in SWADAPT1 and SWADAPT2 (see the “Driving Assistance” section) in the real world and in the VR environment. In total, 29 expert drivers with neurological degenerative disorders were screened by clinicians to take part in this study. The results show that there is no statistically significant difference between the real world and VR (Kruskal-Wallis test). Participants’ quality of experience (QoE) was measured via a USE (Usefulness, Satisfaction, and Ease of Use) questionnaire with 30 questions grouped into four criteria and rated on a seven-point Likert scale [28], as reported in Figure 12. In addition, if the cognitive load is generally higher in VR, the VR/real-world cognitive load ratio decreases as the difficulty of the circuits tested by

the users increases. In VR, the patients experienced a high sense of presence, and the level of cybersickness remained very low in the three circuits. In particular, the collected data indicate that using the simulator during a training phase could drastically reduce damage to the environment (walls, doors, and furniture) and driving accidents [18].

The objective of the clinical trial SIMADAPT2 (NCT04894981) was to evaluate the impact of the immersive environment on VR driving performance. Three different conditions were compared in SIMADAPT2: with a Cave Automatic Virtual Environment (Immersia at IRISA/Inria Rennes), with a head-mounted display, and with a nonimmersive TV screen (see Figure 4). Overall, 18 wheelchair users with and without driving difficulties participated in this clinical study, organized in two sessions to comply with COVID-19 restrictions. Similar to SIMADAPT1, our preliminary results consistently show a small simulated-to-real gap, strong acceptability and a feeling of safety, and better driving performance with the immersive displays. Again, our data support the idea that training with the simulator during a learning phase leads to a significant reduction of damage to property.

# Discussion: Challenges and Recommendations

## Technical Challenges

An open challenge is to guarantee that the ensemble of assistive technologies developed in the ADAPT project by the French and English partners works safely and harmoniously.

A possible way forward is to exploit redundant information. For instance, today, a growing number of accurate 3D models of indoor and outdoor environments is publicly available. These (CAD or point cloud) models could be used in conjunction with spherical photogrammetry (especially in areas that the wheelchair user has never visited before) to underpin real-time vision-based motion estimation algorithms. A first step in this direction has been taken in [29], where a new panoramic 3D pose-tracking algorithm has been shown to provide accurate estimates, even in the presence of large inter-frame motions (several meters). The algorithm relies on a representation of catadioptric images as a mixture of photometric potentials, similar to the one used for the direct visual gyroSCOPE in [22]. In future work, we plan to adapt the approach in [29] to dual-fisheye images to have the largest possible number of algorithms working

<figure><img src="image_14.png" /><figcaption>**Figure 10.** The statistical results from the questionnaire used to evaluate SpheriCol [21]. The (a) ease of learning, (b) ease of use, and (c) usefulness of SpheriCol. The (d) percentage of time that driving assistance was used by the 17 participants in the test circuit. NA: not available.</figcaption></figure>
with the same hardware onboard the wheelchair.

To guarantee safety, an assistive technology is also expected to operate as designed in any circumstances (including unfavorable conditions, such as variable lighting, rain, uneven terrain, and so on). Vision-based driving assistance systems, such as SpheriCol, might perform poorly in scenes with a large range of light intensities, i.e., in scenes where bright sunshine coexists with dark shadows, as in the transitions between indoor and outdoor environments. Unfortunately, the price to pay for compactness in consumer-grade twin-fisheye cameras is the limited dynamic range. Real-time high-dynamic-range (HDR) vision will finally make it possible to design assistive devices that work in scenes with challenging illumination conditions. UPJV's group has recently developed a new panoramic system (consisting of an orthographic camera combined with four convex mirrors and three neutral density filters), HDROmni [30], which optically extends the dynamic range of the images. Preliminary tests on a mobile robot are promising, and there are plans to apply the same optical design to SpheriCol to make it more robust to abrupt illumination changes. Another direction for future research pertains

to vision-based closed-loop control and, in particular, heading control, for which a twin-fisheye camera can be regarded as a valid alternative to conventional MEMS gyroscopes integrated into the smart wheelchair.

The SDP approach to shared control holds great potential for matching assistance to different driving styles. However, while the computational heavy lifting is carried out offline, finding an optimal policy using a naive implementation based on Bellman's principle of optimality remains a time-consuming task. Therefore, if we are to build more granular driver models that would be able to offer an even better fit between driving assistance and the user's habits, this process should be accelerated. To this end, in future versions of the shared control algorithm, we plan to adopt a *policy iteration approach*.

Finally, as far as the wheelchair simulator is concerned, we are currently considering the possibility of improving the user experience by explicitly taking *motion cues* into account (i.e., the perceptual mechanisms by which humans sense the motion of their own body with respect to the surrounding environment).

<figure><img src="image_15.png" /><figcaption>**Figure 11.** UCL's PAMELA facility. (a) The modular platform enables setting up slopes and negative obstacles. (c) The circuit, with a volunteer testing the model-based shared control algorithm proposed in [26].</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>**Figure 12.** Evaluation of the QoE of the wheelchair simulator. Mean and standard deviation of the USE score in the real world (red) and the VR environment (blue), according to four criteria [18].</figcaption></figure>

## Functional Challenges

While the technologies developed in the ADAPT project have been very successful in matching the needs of the real users
identified by our clinical partners, a number of challenging
functional requirements are still missing. For example, the
assistance provided by a smart wheelchair should always be
socially acceptable, and in future iterations of our algorithms,
we are going to include an additional layer that accommo-
dates the social dimension (proxemics).

Moreover, in real-world scenarios, user expectations and
capabilities (e.g., the level of effort or attention) are not fixed
but subtly vary over time. To address this issue, we are cur-
rently working on new methods that dynamically adapt the
level of assistance to the instantaneous needs of the user. For
that purpose, we intend to take advantage of an eye tracker
and body sensors to monitor the physiological and biochemi-
cal profile of the driver in the short and long terms (in fact,
biomarkers in saliva and sweat are known to be indicative of
performance and stress).

Recommendations

As the five-year term of the ADAPT project comes to an end,
it is certainly worthwhile here to sum up some of the key
findings and lessons learned, based on our own experience
of the terrain. These guidelines are intended for researchers
in rehabilitation and assistive robotics and for health-care
professionals. They are as follows:

* The development of a new smart wheelchair requires the concerted effort of three actors throughout the process (“codesign principle”): medical specialists, robotic researchers, and end users. A mere transfer of consolidated robotic technologies is doomed to failure.

* Simplicity, modularity, and ergonomics are fundamental design principles for smart wheelchairs, and they cannot be sacrificed in the development stage.

* Haptic interfaces are emerging assistive devices for power wheelchairs. They are minimally invasive and intuitive to use, but they still have not found their way into mainstream clinical practice today. Likewise, omnidirectional vision has not met with widespread acceptance.

* Training programs for health-care professionals to learn new assistive technologies (“train-the-trainer” sessions) are crucial to accelerate deployment toward full-scale adoption.

* The journey to the market is long and arduous (especially in the time of COVID-19). For instance, the time elapsed between the submission of the experimental protocol and approval by the local ethics committee can exceed the length of the product development phase.

Conclusion

This article provided a general overview of the innovative
assistive robotic technologies developed in the ADAPT
project. The exposition focused on the design, implemen-
tation, and experimental validation, via large-scale clinical
trials, of two complementary smart wheelchairs and a
wheelchair driving simulator based on virtual reality. This
research, carried out by an international team of roboti-
cists and medical experts, is rooted in two basic principles,
codesign and modularity, and it has the potential to

transform the everyday life of millions of wheelchair users
worldwide.

Acknowledgments

This work was carried out as part of the Interreg VA France
(Channel) England (FCE) ADAPT (Assistive Devices for
Empowering Disabled People Through Robotic Technolo-
gies) project. The Interreg FCE program is a European Terri-
torial Cooperation program that aims to fund high-quality
cooperation projects in the Channel border region between
France and England. The program is funded by the European
Regional Development Fund (ERDF). The authors would like
to thank Éric Bazin (Institut d'Electronique et des Technolo-
gies du Numérique, INSA Rennes) for his help during the
development of the hardware platforms and the Inria associ-
ate team ISI4NAVE for supporting several exchanges among
the French and English partners.

References

[1] R. C. Simpson, E. F. LoPresti, and R. A. Cooper, "How many people would benefit from a smart wheelchair?" *J. Rehabil. Res. Dev.*, vol. 45, no. 1, pp. 53–72, 2008, doi: 10.1682/JRRD.2007.01.0015.

[2] D. A. Abbink et al., "A topology of shared control systems – Finding common ground in diversity," *IEEE Trans. Human-Mach. Syst.*, vol. 48, no. 5, pp. 509–525, 2018, doi: 10.1109/THMS.2018.2791570.

[3] A. Erdogan and B. D. Argall, “The effect of robotic wheelchair control paradigm and interface on user performance, effort and preference: An experimental assessment,” *Robot. Auton. Syst.*, vol. 94, pp. 282–297, Aug. 2017, doi: 10.1016/j.robot.2017.04.013.

[4] C. X. Miller, T. Gebrekristos, M. Young, E. Montague, and B. D. Argall, "An analysis of human-robot information streams to inform dynamic autonomy allocation," in *Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst.*, 2021, pp. 1872–1878, doi: 10.1109/IROS51168.2021.9636637.

[5] M. Zolotas, J. Elsdon, and Y. Demiris, “Head-mounted augmented reality for explainable robotic wheelchair assistance,” in *Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst.*, 2018, pp. 1823–1829, doi: 10.1109/IROS.2018.8594002.

[6] É. Leblong et al., “SWADAPT1: Assessment of an electric wheelchair-driving robotic module in standardized circuits: A prospective, controlled repeated measure design pilot study,” *J. Neuroeng. Rehabil.*, vol. 18, no. 1, 2021, Art. no. 140, doi: 10.1186/s12984-021-00923-2.

[7] J. Vogel et al., "An ecosystem for heterogeneous robotic assistants in caregiving: Core functionalities and use cases," *IEEE Robot. Autom. Mag.*, vol. 28, no. 3, pp. 12–28, 2021, doi: 10.1109/MRA.2020.3032142.

[8] J. Podobnik, J. Rejc, S. Šlajpah, M. Munih, and M. Mihelj, “All-terrain wheelchair: Increasing personal mobility with a powered wheel-track hybrid wheelchair,” *IEEE Robot. Autom. Mag.*, vol. 24, no. 4, pp. 26–36, 2017, doi: 10.1109/MRA.2017.2746182.

[9] H. Ikeda, T. Toyama, D. Maki, K. Sato, and E. Nakano, "Cooperative step-climbing strategy using an autonomous wheelchair and a robot," *Robot. Auton. Syst.*, vol. 135, p. 103,670, Jan. 2021, doi: 10.1016/j.robot.2020.103670.

[10] T. How, R. H. Wang, and A. Mihailidis, “Evaluation of an intelligent wheelchair system for older adults with cognitive impairments,” *J. Neuroeng. Rehabil.*, vol. 10, no. 1, 2013, Art. no. 90, doi: 10.1186/1743-0003-10-90.
[11] P. Viswanathan et al., "Intelligent wheelchair control strategies for older adults with cognitive impairment: User attitudes, needs, and preferences," *Auton. Robot.*, vol. 41, no. 3, pp. 539-554, 2017, doi: 10.1007/s10514-016-9568-y.

[12] M. Kutbi et al., "Usability studies of an egocentric vision-based robotic wheelchair," *ACM Trans. Human-Robot Interact.*, vol. 10, no. 1, 2020, Art. no. 4, doi: 10.1145/3399434.

[13] Y. Morales, A. Watanabe, F. Ferreri, J. Even, K. Shinozawa, and N. Hagita, "Passenger discomfort map for autonomous navigation in a robotic wheelchair," *Robot. Auton. Syst.*, vol. 103, pp. 13-26, May 2018, doi: 10.1016/j.robot.2018.02.002.

[14] J. L. Candiotti et al., "A heuristic approach to overcome architectural barriers using a robotic wheelchair," *IEEE Trans. Neural Syst. Rehabil. Eng.*, vol. 27, no. 9, pp. 1846-1854, 2019, doi: 10.1109/TNSRE.2019.2934387.

[15] S. Arlati, V. Colombo, G. Ferrigno, R. Sacchetti, and M. Sacco, "Virtual reality-based wheelchair simulators: A scoping review," *Assistive Technology*, vol. 32, no. 6, pp. 294-305, 2020, doi: 10.1080/10400435.2018.1553079.

[16] C. S. Teodorescu, I. Caplan, H. Eberle, and T. Carlson, "Model-based sensor fusion and filtering for localization of a semi-autonomous robotic vehicle," in *Proc. Eur. Contr. Conf.*, 2021, pp. 1283-1290, doi: 10.23919/ECC54610.2021.9654877.

[17] G. Vailland, Y. Gaffary, L. Devigne, V. Gouranton, B. Arnaldi, and M. Babel, "Vestibular feedback on a virtual reality wheelchair driving simulator: A pilot study," in *Proc. ACM/IEEE Int. Conf. Human-Robot Interact.*, 2020, pp. 171-179, doi: 10.1145/3319502.3374825.

[18] G. Vailland et al., "VR based power wheelchair simulator: Usability evaluation through a clinically validated task with regular users," in *Proc. IEEE Virtual Reality 3D User Inter.*, 2021, pp. 420-427, doi: 10.1109/VR50410.2021.00065.

[19] S. Delmas et al., "SpheriCol: A driving assistance system for power wheelchairs based on spherical vision and range measurements," in *Proc. IEEE/SICE Int. Symp. Syst. Integr.*, 2021, pp. 505-510, doi: 10.1109/IEEECNF49454.2021.9382766.

[20] S. Sumikura, M. Shibuya, and K. Sakurada, "OpenVSLAM: A versatile visual SLAM framework," in *Proc. 27th ACM Int. Conf. Multimedia*, 2019, pp. 2292-2295, doi: 10.1145/3343031.3350539.

[21] S. Delmas, F. Morbidi, G. Caron, M. Babel, and F. Pasteau, "SpheriCol: A driving assistant for power wheelchairs based on spherical vision," *IEEE Trans. Med. Robot. Bionics*, to be published.

[22] G. Caron and F. Morbidi, "Spherical visual gyroscope for autonomous robots using the mixture of photometric potentials," in *Proc. IEEE Int. Conf. Robot. Automat.*, 2018, pp. 820-827, doi: 10.1109/ICRA.2018.8460761.

[23] H.-E. Benseddik, F. Morbidi, and G. Caron, "PanoraMIS: An ultra-wide field of view image dataset for vision-based robot-motion estimation," *Int. J. Robot. Res.*, vol. 39, no. 9, pp. 1037-1051, 2020, doi: 10.1177/0278364920915248.

[24] L. Devigne, F. Pasteau, M. Babel, V. K. Narayanan, S. Guegan, and P. Gallien, "Design of a haptic guidance solution for assisted power wheelchair navigation," in *Proc. IEEE Int. Conf. Syst., Man, Cybern.*, 2018, pp. 3231-3236, doi: 10.1109/SMC.2018.00547.

[25] C. Ehz, P. Trautman, L. Devigne, V. Bureau, M. Babel, and T. Carlson, "Probabilistic vs linear blending approaches to shared control for wheelchair driving," in *Proc. Int. Conf. Rehabil. Robot.*, 2017, pp. 835-840, doi: 10.1109/ICORR.2017.8009352.

[26] C. S. Teodorescu, B. Zhang, and T. Carlson, "A stochastic control strategy for safely driving a powered wheelchair," in *Proc. 21st IFAC World Congr.*, 2020, pp. 10,148-10,153, doi: 10.1016/j.ifacol. 2020.12.2741.

[27] L. Devigne et al., "Power wheelchair navigation assistance using wearable vibrotactile haptics," *IEEE Trans. Haptics*, vol. 13, no. 1, pp. 52-58, 2020, doi: 10.1109/TOH.2019.2963831.

[28] A. M. Lund, "Measuring usability with the USE questionnaire," *Usability Interface*, vol. 8, pp. 3-6, Jan. 2001.

[29] S. E. Guerbas, N. Crombez, G. Caron, and E. Mouaddib, "Direct 3D model-based tracking in omnidirectional images robust to large inter-frame motion," in *Proc. IEEE Int. Conf. Adv. Robot.*, 2021, pp. 505-510, doi: 10.1109/ICAR53236.2021.9659324.

[30] J. Ducrocq, G. Caron, and E. Mouaddib, "HDROmni: Optical extension of dynamic range for panoramic robot vision," *IEEE Robot. Autom. Lett.*, vol. 6, no. 2, pp. 3561-3568, 2021, doi: 10.1109/LRA.2021.3060642.

[31] World Health Organization, "Assistive technologies," World Health Organization, Geneva, Switzerland. Accessed: Jun. 1, 2022. [Online]. Available: www.who.int/news-room/fact-sheets/detail/assistive-technology.

**Fabio Morbidi**, Modeling, Information, and Systems laboratory, University of Picardie Jules Verne, Amiens, 80039, France.
Email: fabio.morbidi@u-picardie.fr.

**Louise Devigne**, Rainbow/Hybrid teams, Institut de Recherche en Informatique et Systèmes Aléatoires/Inria Rennes and Institut National des Sciences Appliquées de Rennes, Rennes, 35700, France.
Email: louise.devigne@irisa.fr.

**Catalin Stefan Teodorescu**, Aspire Create, University College London, London, WC1E 6BT, U.K., and Royal National Orthopedic Hospital, London, HA7 4LP, U.K.
Email: s.teodorescu@ucl.ac.uk.

**Bastien Fraudet**, Pôle Saint-Hélier, Rennes, 35000, France.
Email: bastien.fraudet@pole-sthelier.com.

**Émilie Leblong**, Pôle Saint-Hélier, Rennes, 35000, France.
Email: emilie.leblong@pole-sthelier.com.

**Tom Carlson**, Aspire Create, University College London, London, WC1E 6BT, U.K., and Royal National Orthopedic Hospital, London, HA7 4LP, U.K.
Email: t.carlson@ucl.ac.uk.

**Marie Babel**, Rainbow/Hybrid teams, Institut de Recherche en Informatique et Systèmes Aléatoires/Inria Rennes and Institut National des Sciences Appliquées de Rennes, Rennes, 35700, France.
Email: marie.babel@irisa.fr.

**Guillaume Caron**, Modeling, Information, and Systems laboratory, University of Picardie Jules Verne, Amiens, 80039, France, and Joint Robotics Laboratory, Centre National de la Recherche Scientifique-National Institute of Advanced Industrial Science and Technology, Tsukuba, 305-8560, Japan.
Email: guillaume.caron@u-picardie.fr.
**Sarah Delmas**, Modeling, Information, and Systems laboratory, University of Picardie Jules Verne, Amiens, 80039, France.
Email: sarah.delmas@ensta.bretagne.org.

**François Pasteau**, Rainbow/Hybrid teams, Institut de Recherche en Informatique et Systèmes Aléatoires/Inria Rennes and Institut National des Sciences Appliquées de Rennes, Rennes, 35700, France.
Email: francois.pasteau@irisa.fr.

**Guillaume Vailland**, Rainbow/Hybrid teams, Institut de Recherche en Informatique et Systèmes Aléatoires/Inria Rennes and Institut National des Sciences Appliquées de Rennes, Rennes, 35700, France.
Email: guillaume.vailland@irisa.fr.

**Valérie Gouranton**, Rainbow/Hybrid teams, Institut de Recherche en Informatique et Systèmes Aléatoires/Inria Rennes and

Institut National des Sciences Appliquées de Rennes, Rennes, 35700, France.
Email: valerie.gouranton@irisa.fr.

**Sylvain Guégan**, University of Rennes, Institut National des Sciences Appliquées de Rennes, Laboratoire de Génie Civil et Génie Mécanique, Rennes, 35000, France.
Email: sylvain.guegan@insa-rennes.fr.

**Ronan Le Breton**, University of Rennes, Institut National des Sciences Appliquées de Rennes, Laboratoire de Génie Civil et Génie Mécanique, Rennes, 35000, Rennes, France.
Email: ronan.le-breton@insa-rennes.fr.

**Nicolas Ragot**, École Supérieure d'Ingénieurs en Génie Électrique-Institut de Recherche en Systèmes Électroniques Embarqués, Rouen, 76801, France.
Email: nragot@cesi.fr.

<figure><img src="image_17.png" /></figure>