# SpheriCol: A Driving Assistant for Power Wheelchairs based on Spherical Vision

Sarah Delmas, Fabio Morbidi, Guillaume Caron, Marie Babel, François Pasteau

► To cite this version:

Sarah Delmas, Fabio Morbidi, Guillaume Caron, Marie Babel, François Pasteau. SpheriCol: A Driving Assistant for Power Wheelchairs based on Spherical Vision. IEEE Transactions on Medical Robotics and Bionics, 2023, 5 (2), pp.387-400. 10.1109/TMRB.2023.3258519. hal-04042108

HAL Id: hal-04042108

https://hal.science/hal-04042108

Submitted on 23 Mar 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# SpheriCol: A Driving Assistant for Power Wheelchairs based on Spherical Vision

Sarah Delmas<fnref n="1" />, Fabio Morbidi<fnref n="1" />,*, Guillaume Caron<fnref n="1" />,<fnref n="2" />, Marie Babel<fnref n="3" />, François Pasteau<fnref n="3" />

***Abstract***—This paper presents a new vision-based navigation assistant for power wheelchairs, called SpheriCol. Inspired by the parking assist system in cars, an omnidirectional image captured by a twin-fisheye camera is overlaid with colored distance markers from range sensors mounted on the wheelchair, to generate an augmented view of the surrounding environment. Such an image is displayed in real time to the user on a screen. To evaluate the effectiveness of the proposed system, able-bodied subjects and older adults with motor impairments have been asked to drive a consumer-grade power wheelchair equipped with SpheriCol, via a standard joystick. Our clinical trials with different obstacle courses, consistently indicate that SpheriCol is effective in improving safety and comfort, and in supporting a driver's decision during challenging but prevalent maneuvers, such as reversing out of an elevator, corridor centering, and turning on the spot.

***Index Terms***—Assistive robotics, smart wheelchair, omnidirectional vision, collision avoidance, safe navigation.

## I. INTRODUCTION

WHEELCHAIRS maintain or improve an individual's functioning, self-confidence, and independence, thereby promoting well-being. In the absence of this assistive device, people with severe motor impairments are often excluded, isolated, and locked into poverty, thus increasing the impact of disease and disability on a person, her family, and society as a whole. In the European Union, wheelchair users comprise around 1% of the population, or 7.5 million people. In the United States, the prevalence was 2.3% or 5.5 million adults (18 years and older), in 2014 [1]. According to the World Health Organization (WHO), in 2018, 75 million people worldwide needed a wheelchair, but only 5% to 15% of those in need had access to one [2].

In the last decade, assistive robotics, because of its high economic and societal value, has been at the forefront of research in robotics [3], [4], [5], [6], [7]. However, while many individuals rely on the provision of wheelchairs, they experience driving difficulties on a daily basis [8]. In fact, even the urban spaces of modern cities fail to conform to the most basic accessibility and safety principles, and several architectural barriers still exist today for those who use mobility aids outdoors: non-adapted public transport, narrow sidewalks, absence of curb cuts [9], [10], [11], [12], [13]. Likewise, wheelchair users are faced with a number of challenging tasks indoors (e.g. reverse out of an elevator, doorway traversal,

turn in place, dock under a table), and they often struggle to build a mental model of the surrounding environment [14], [15]. Access to good quality and affordable assistive technologies has been mandated by the Convention on the Rights of Persons with Disabilities of the United Nations for ten years, but the WHO estimates that there is still a shortage of health and rehabilitation staff with suitable knowledge and skills, to provide a wheelchair that meets the individual needs of a user in care. Moreover, wheelchair accessories tailored to suit a certain user's profile, remain expensive and are installed piecemeal.

The benefits of smart wheelchair use for older adults with cognitive impairments are well documented in the specialized literature [16], [17], [18]. For instance, in [19], [20], the authors report an evaluation of joystick control input during the execution of different driving tasks. Immersive technologies involving head-mounted displays have also recently garrered attention as apt training simulators for off-line learning of wheelchair control [21], [22], [23]. Several research projects [24], [25], [26], [27], have addressed the problem of collision avoidance with robotic wheelchairs, by proposing solutions based on the combination of different active sensors (laser, ultrasonic or infrared sensors) for an increased data envelope. These sensors inform the user of the presence of obstacles near the wheelchair, and provide support to negotiate the environment. The main goal is to provide assistance for safe navigation and, at the same time, minimize user's intervention (thus reducing cognitive and physical workload). However, the majority of the systems developed in these projects are still early-stage technology, and though highly informative, visual feedback is often overlooked.

In recent years, cameras have been increasingly used for (semi-)autonomous localization and navigation of instrumented wheelchairs [28], [29]. In particular, multi-camera systems (three monocular cameras in [30], a Ladybug 2 camera in [31]), are gaining traction, since they are inexpensive and offer a full coverage of the surrounding environment. Nevertheless, to date, there is scant research pertaining to omnidirectional vision [32], and its potential for wheelchair driving assistance remains largely untapped.

In this article, we present a new vision-based decision support system, called SpheriCol (a portmanteau of “Spherical” and “Collision avoidance”), for navigation assistance of the users of electrically-powered wheelchairs (EPW). Occupational therapists and specialists in rehabilitation medicine have been consulted during the ADAPT project [26], to study the applicability of this visual aid in real-life practice. The proposed system is designed to be strap-on to any

<footnote n="1">¹MIS laboratory, University of Picardie Jules Verne, Amiens, France.</footnote>

<footnote n="2">²CNRS-AIST Joint Robotics Laboratory (JRL), IRL, Tsukuba, Japan.</footnote>

<footnote n="3">³Rainbow team, IRISA/Inria Rennes and INSA Rennes, France.</footnote>

<footnote>*Corresponding author. Email: fabio.morbidi@u-picardie.fr
Final manuscript submitted on February 28, 2023.</footnote>
commercial EPW, and its functional components have been
developed atop ROS (Robot Operating System). SpheriCol
relies on omnidirectional images captured by an overhead
pocket-size twin-fisheye camera and range measurements from
low-cost (ultrasonic or laser) sensors. The 360° images are
overlaid in real time with color-coded distance markers, to
generate an augmented view with obstacle clues, of the
surrounding environment. The user can select four different
viewpoints via a tactile display, from which clearance around
the wheelchair and thus collision risk, can be easily assessed.
The driver closes the loop via a standard joystick and has full
control authority over the wheelchair at all times. Extensive
clinical trials in cluttered indoor environments with a Quickie
Salsa M² EPW equipped with a ring of Time-of-Flight sensors
and a Ricoh Theta S twin-fisheye camera, have shown that the
proposed driving assistant offers tangible benefits in terms of
risk management and control effort.

It is worth mentioning here that a system akin to ours,
combining panoramic vision with laser range finders, has been
recently proposed in [33], for robot teleoperation. Differently
from our egocentric representation, the system in [33] gener-
ates third-person-view images from arbitrary viewpoints, but
requires four fisheye cameras with a large baseline. As a
consequence, it cannot be easily installed on a consumer-
grade EPW. Moreover, it only works in indoor corridor-like
environments. A similar setup (including an Insta360 Air
twin-fisheye camera), has been considered in [34] to improve
operator's situation awareness in hazardous environments, and
it has been successfully tested on an Asterix tracked robot.

An early prototype of our driving assistant was described in
our conference paper [35]. Compared to [35], in this article, we
introduce an enhanced software architecture and data logging
system, and we report the outcomes of two clinical trials which
involved nineteen able-bodied subjects and older adults with
mild-to-moderate motor impairments.

The rest of this paper is organized as follows. Sect. II presents the two main components of the driving assistant: the power wheelchair and the twin-fisheye camera. Sect. III describes the main functionalities of SpheriCol and the configuration/visualization interfaces. Sect. IV is devoted to the user study with a real EPW. Finally, in Sect. V, we highlight the significance of the system and its limitations, and we discuss potential directions for future of research.

## II. BASIC SYSTEM COMPONENTS

### A. Power wheelchair

The driving assistance system described in this article is
fully compatible with the existing consumer-grade EPWs (with
front-, mid-, or rear-wheel drive). The wheelchair should be
equipped with a set of exteroceptive sensors located around
its perimeter, 25 to 50 cm above the ground (e.g. a ring
of ultrasonic sensors [36]). The sensors (with a maximum
range indoors, of 4 to 5 m), provide proximity information
via range/bearing measurements to the obstacles around the
wheelchair. The driving assistant also takes advantage of
a tactile display (e.g. a tablet), for human-machine inter-
action (see Fig. 1). Besides these basic pieces of equip-
ment, SpheriCol is hardware-agnostic, i.e. it neither requires

<figure><img src="image_2.png" /><figcaption>Fig. 1: Overview of system components: (a) Power wheelchair equipped with a ring of range sensors; (b) Twin-fisheye camera; (c) Visualization interface.</figcaption></figure>

a specific kinematic/dynamic model for the EPW, nor a
particular range-sensing technology, and it can be installed
as a “kit” on any power wheelchair.

### B. Twin-fisheye camera

SpheriCol takes a stream of 360° images captured by a twin-fisheye camera, as input (see Fig. 2). These innovative cameras are small, lightweight, and they can record high-resolution videos with frame rates up to 30 fps. In the last decade, the compact optical design proposed by Ricoh, consisting of two fisheye lenses mounted back-to-back coupled with two light-sensitive surfaces, has gradually become mainstream and it has been adopted by other camera manufacturers (e.g. in Samsung Gear 360, LG 360 CAM, Insta360 X3, KanDao QooCam 8K and Kodak PIXPRO Orbit360).

Our driving assistant requires the twin-fisheye camera to be
mounted in an elevated position on the wheelchair, to have
an unobstructed view of the free space around it. To meet
this requirement, a common solution in instrumented EPW
[32], [37] or Personal Mobility Devices [38], is to install the
camera on top of a mast behind the backrest (even though
design studies have shown that added sensors above the
headrest might interfere with wheelchair transportation in a
vehicle [39], [40]).

It is well known that some categories of fisheye cameras are approximately equivalent to a central catadioptric system [41]. Therefore, the unified projection model [42], [43], can be used. We describe each fisheye lens by its own set of intrinsic parameters $\mathcal{P}_{c_j} = \{a_{u_j}, a_{v_j}, u_{0_j}, v_{0_j}, \xi_j\}, j \in \{1, 2\}$, where $a_{u_j}$ ($a_{v_j}$) is the size of unit length in horizontal (vertical) pixels, and $(u_{0_j}, v_{0_j})$ are the coordinates of the principal point in pixels. The fifth parameter, $\xi_j$, is the distance between the unit sphere's first projection center and the perspective second projection center of the j-th fisheye lens, as reported in [43, Fig. 2]. Following [44], we assume that the translation vector between the two fisheye frames $\mathcal{F}_{c_1}$ and $\mathcal{F}_{c_2}$ is zero,
<figure><img src="image_3.png" /><figcaption>Fig. 2: Twin-fisheye camera: (a) Front, side, and top view of the Ricoh Theta S camera (image courtesy of Ricoh): the optical system with the two fisheye lenses, prisms and CMOS sensors is shown in the lower right corner; (b) Schematic diagram of the camera (top view): $\mathcal{F}_c$ is the camera frame, and $\mathcal{F}_{c_1}$, $\mathcal{F}_{c_2}$ are the reference frames associated with the two fisheye lenses; (c) Example of dual-fisheye image captured by the Ricoh Theta S: the circular sub-images correspond to the two fisheye lenses.</figcaption></figure>

to ensure a unique projection viewpoint. Moreover, we assume that the camera frame $\mathcal{F}_c$ coincides with $\mathcal{F}_{c_1}$ (they are shown separately in Fig. 2b, for clarity). The extrinsic parameters of the twin-fisheye camera are then incorporated into the rotation matrix $e_2R_{c_1}$ from frame $\mathcal{F}_{c_1}$ to frame $\mathcal{F}_{c_2}$.

To calibrate the twin-fisheye camera, we used a custom-made calibration rig consisting of six circle patterns attached inside two half cubes, and followed the same procedure described in [44].

### III. MAIN FUNCTIONALITIES AND INTERFACES

In this section, the main functionalities of our driving assistance system are described in detail. SpheriCol leverages the middleware ROS (Robot Operating System), for data collection and communication between the different modules, and the OpenCV library for image processing.

#### A. Viewpoint selection

The environment around the wheelchair can be observed from four “virtual” viewpoints (equirectangular, panoramic, spherical and bird’s eye), as further elaborated below.

**a) Equirectangular view:** The input dual-fisheye images are mapped into equirectangular images by exploiting the estimated intrinsic parameters of the twin-fisheye camera (cf. Sect. II-B). The width of the equirectangular image corresponds to the 360° horizontal field of view (FoV) of the twin-fisheye camera. An example of dual-fisheye image (top) with

its equirectangular counterpart (bottom) is shown in Fig. 3a. Because of the wide availability in twin-fisheye cameras of different manufacturers, the equirectangular view is the default view in SpheriCol, and all the others are generated from it.

**b) Panoramic view:** A panoramic image corresponds to half of an equirectangular image (i.e. it only covers a 180° horizontal FoV). To synthesize a “front view”, we leveraged OpenCV’s Rect function to crop the left and right bands of the equirectangular image corresponding to the rear end of the wheelchair (white dashed areas in Fig. 3a, bottom). On the other hand, to generate a “rear view”, we stitched the two bands together using OpenCV’s hconcat function. The user is also allowed to select a specific angular sector of the panoramic image (in 5° steps): this facilitates early detection of obstacles in the blind spots of EPW (for more details, see Sect. III-C below).

**c) Spherical view:** The spherical view has the advantage of clearly showing the presence of potential obstacles at ground level (see Fig. 3b). It is generated from the equirectangular view via a simple Cartesian-to-polar coordinates transformation (cf. OpenCV’s linearPolar function).

**d) Bird’s-eye view:** As the spherical view, the bird’s-eye view is useful to detect obstacles on the floor around the EPW, but the images have a more uniform (perspective-like) spatial resolution. To generate the bird’s-eye view, we first map the $m \times n$ equirectangular image into a unit sphere, via the transformation (see Fig. 4),

$$ \theta = 360^{\circ} \frac{u_{\theta}}{m-1}, \quad \varphi = 90^{\circ} \frac{v_{\varphi}}{[(n-1)/2]} $$

where $(u_{\theta}, v_{\varphi})$ is the size of the sampling step of the equirectangular image in the horizontal and vertical direction, respectively, and $(\theta, \varphi) \in [0^{\circ}, 360^{\circ}) \times [-90^{\circ}, 90^{\circ}]$ are the coordinates of a point on the unit sphere. The bird’s-eye

<figure><img src="image_4.png" /><figcaption>Fig. 3: Viewpoint selection: [(a), top] Dual-fisheye image captured by a twin-fisheye camera, and [(a), bottom] corresponding equirectangular image; (b) Spherical view; (c) Bird’s-eye view. For ease of comparison, the images in (a), (b) and (c) have been captured at the same location.</figcaption></figure>
<figure><img src="image_5.png" /><figcaption>Fig. 4: Generation of bird's-eye image $I_{\text{bird}}$ (yellow), from the image on the unit sphere $I_{\text{sphere}}$.</figcaption></figure>

image $I_{\text{bird}}$ is obtained by projecting the point $(\theta, \varphi)$ on the unit sphere, onto a plane at point $(u_b, v_b)$ (see Fig. 4). The height $h$ (in centimeters) of the sphere above the plane, is a parameter of the system. It depends on several factors: footprint of EPW, user's body size, seating accommodation, and type of environment (indoor or outdoor).

## B. Configuration interface

The different functionalities of SpheriCol are accessible via a configuration interface. Note that differently from the visualization interface, to be presented in Sect. III-C below, the configuration interface is intended for the healthcare professionals and not for the wheelchair users. The “Options” tab in the top left corner of Fig. 5 allows to select the preferred visual/proximity information. With the remaining tabs in the same row, the user can choose one of the four viewpoints described in Sect. III-A. Infrared and ultrasonic sensors being the most popular proximity sensors in EPW, they are already available in the “Options” tab, for quick selection.

The “Points” buttons allow to display the distance measurements to the obstacles, at their corresponding locations in the images (see Fig. 7a). To project the proximity information onto the images, we leveraged the known calibration parameters of the twin-fisheye camera, and empirically estimated the rigid transformation between the camera frame $\mathcal{F}_c$ (cf. Fig. 2) and the local frames of the range sensors. To filter out noise and ensure smooth transitions on the user's screen, a moving average filter was applied to the raw range measurements (in our experiments, see Sect. IV, we computed the mean over the last 10 measurements, which corresponds to an update rate of about 2 Hz). The user can choose how many measurements to display at the same time: if the range sensors are organized in $p$ modules each containing $q$ sensors, then a minimum of $p$ and a maximum of $p \cdot q$ measurements, can be shown on the images.

By selecting “Lines”, the distance measurements from the range sensors are interpolated using Lagrange polynomials, and superimposed on the images (see Fig. 7b and Sect. IV-A

for more details). For the “Points” and “Lines” visualization modes, the following color convention was used: green, if the distance to the obstacle(s) $d \in [12, 15)$ cm, yellow, if $d \in [8, 12)$ cm, and red, if $d < 8$ cm.

The “Ellipses” button allows to display from 1 to 3 concentric ellipses (see Fig. 7c). The first ellipse (red) is placed at a distance $d = 50$ cm from the side of the wheelchair, the second (yellow) at 100 cm, and the third (green) at 150 cm. The ellipses are static, and they are not periodically refreshed using the new measurements from the range sensors.

The “Zones” button activates 12 colored bands (in four angular sectors of 90°) around the wheelchair: green bands for a distance $d \in [12, 15)$ cm, yellow bands if $d \in [8, 12)$ cm, and red bands if $d < 8$ cm (see Fig. 7d). To reduce visual overload, the static bands disappear when an obstacle is detected in the corresponding angular sector.

Similarly to “Ellipses”, the “Indics” (Indicators) button allows to display 1 to 3 red circle arcs on the four sides of the wheelchair (see Fig. 7e). The arcs are dynamic, and they only appear when an obstacle is detected: 1 arc if $d \in [12, 15)$ cm, 2 arcs if $d \in [8, 12)$ cm, and 3 arcs if $d < 8$ cm.

Note that the previous options are not mutually exclusive: in fact, multiple distance markers (e.g. “Points” and “Lines”) can be accommodated by the configuration interface at the same time.

Since other impairments (such as low peripheral vision or visual neglect [45], [46]), may coexist with motor disabilities, auditory feedback is also available in SpheriCol. By clicking on the “Beep” and “Voice” buttons under the “Sound Warnings” tab (top right in Fig. 5), acoustic signals are generated when an obstacle is detected. Normally sighted individuals could also benefit from the auditory feedback, when their visual and haptic channels are overloaded with information [47]. Finally, the “Colors”, “Max Distance”, and “Sound’s Threshold” buttons, give access to additional configuration parameters which allow to fully customize the interface.

<figure><img src="image_6.png" /><figcaption>Fig. 5: Configuration interface of SpheriCol.</figcaption></figure>
### C. Visualization interface

The visualization interface is active during normal usage of the wheelchair. A large portion of the window is used to display the video stream from the twin-fisheye camera (see Fig. 1c). The top bar and the right menu are similar to those in the configuration interface, and they can be used to switch on/off the selected viewpoint and distance markers, respectively. With the “+” and “−” buttons, the user can either change the angular sector in the panoramic images or zoom in/zoom out the bird’s-eye images, by virtually varying the height *h* (cf. Fig. 4).

If the user turns the auditory feedback on, either a series of beeps whose frequency is inversely proportional to the distance *d* to the obstacles (“Beep” option) or a voice message announcing the direction of the approaching obstacle(s) (“Voice” option), will be heard. The two options can be activated simultaneously.

## IV. USER STUDY

A user study was carried out to assess the performance and viability of our driving assistant versus a conventional EPW. The hypothesis under study is that using SpheriCol improves safety and minimizes user's intervention (and thus cognitive load) during complex maneuvers. The section begins with a description of the materials (Sect. IV-A), followed by a presentation of the participants and results of two experimental campaigns (Sect. IV-B and Sect. IV-C, respectively).

### A. Materials

SpheriCol has been tested with a Sunrise Medical Quickie Salsa M² EPW [48]. This mid-wheel drive platform has 6 wheels with independent suspensions: the 4 castor wheels have a diameter of 17.8 cm and the two drive wheels have a diameter of 33 cm. The wheelchair, which is endowed with a 7 cm curb-climbing ability, measures 61 cm at the widest point, and the 60 Ah batteries can propel it up to 10 km/h. A Ricoh Theta S camera [49] was installed overhead behind the passenger. The camera weighs 125 g and its external dimensions are 4.4 cm (W) × 13 cm (H) × 2.29 cm (D). Considering the reduced speed of an EPW in normal operation, we selected a medium video resolution of 1280 pixels × 720 pixels at 15 fps, which allowed for real-time image processing. The 48 range sensors around the wheelchair are grouped into 7 modules: 6 modules of 6 sensors are located on the left and right side, and under the footplates, and 1 module of 12 sensors is positioned behind the backrest (see Fig. 6a). The height above the floor of the modules, ranges between 13 cm (the two modules under the footplates) and 45.5 cm (the two modules installed on the mudguards). The housing of the 7 sensor modules was 3D printed at INSA Rennes. The ST VL53L1X Time-of-Flight (ToF) sensors have the following technical specifications [50]: distance measurement, up to 4 m; ranging frequency, up to 50 Hz; typical FoV, 27°; size, 4.9 mm × 2.5 mm × 1.56 mm.

Differently from [35], in our current implementation, SpheriCol runs under Linux Ubuntu 18.10 on an Intel NUC mini PC [51] installed on the wheelchair, and it relies on the

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 6: (a) Location of the 48 ToF sensors on the Quickie Salsa M² wheelchair (top view). The seven sensor modules are depicted in green; (b) Tablet mounted on the wheelchair: the visualization interface described in Sect. III-C is activated (photo courtesy of newspaper OUEST-FRANCE).</figcaption></figure>

ROS Melodic Morenia distribution and the OpenCV library (release 2.x). The camera is connected to the NUC via a USB port. To remote control the data-recording process (“rosbag” creation), we used a laptop computer connected to the NUC via SSH, a secure communication protocol. Finally, we leveraged VNC Connect [52], to establish a communication channel between the mini PC and a Samsung tablet (see Fig. 6b). This allows to display the images captured by the Ricoh Theta S and to transmit the data entered by the users.

Fig. 7 shows different ways of augmenting the omnidirectional images captured by the Ricoh camera, with the proximity information provided by the ToF sensors (cf. Sect. III-B). Note that the curves interpolating the range measurements, provide a useful approximate representation of obstacle surfaces, and they are easier to interpret for reactive control than a sparse set of distance points. However, different contiguous obstacles cannot always be easily discriminated by using this representation. To address this issue, the lines shown in Fig. 7b are computed from data points in 90° sectors (for example, the front line emanates from points lying within the [-45°, 45°]
<figure><img src="image_9.png" /><figcaption>Fig. 7: (a) Spherical view with the obstacles detected by the 48 ToF sensors: the green points indicate the free space around the wheelchair. Equirectangular view with, (b) lines obtained via Lagrange interpolation, and (c) ellipses; Spherical view with, (d) colored bands ("Zones"), and (e) circular arcs ("Indics"). In (d), the colored bands show obstacle clearance: the front wall is less than 1 m away, and the rear wall is less than 0.5 m away. The inset in (e), shows a blow-up of the six red circular arcs.</figcaption></figure>

interval, the forward direction of the wheelchair corresponding to 0°). Moreover, to improve readability, the lines are displayed at 10° intervals.

## B. First experimental campaign

a) Participants: A convenience sample of two users with various neurologic deficits took part in the first campaign (1 male and 1 female, 60+ years old, see Fig. 6b). They are patients of Pôle Saint-Hélier, a Physical and Rehabilitation Medicine Center, in Rennes, France, and have limited driving experience with an EPW (to minimize the learning effect). Subject 1 is with right hemispatial neglect [53], and subject 2 is with tetraplegia due to a neurodegenerative disease. The research ethics committee approved the experimental protocol. The participants signed a consent form and they were trained on wheelchair's features.

b) Experimental setup: The campaign took place in a test circuit specifically designed at Pôle Saint-Hélier, in October 2020. The circuit included two slaloms and two simulated elevator cabins (shaded areas in Fig. 8a), and several sharp turns in narrow passageways with a clearance on either side of the wheelchair of only few centimeters (similar standardized obstacle courses have been recently considered in [18]). The control inputs were provided by a standard z-axis joystick, and the participants were requested to use SpheriCol to complete the circuit as quickly and as safely as possible (i.e. with the fewest number of collisions). To compensate for

the reduced awareness of stimuli on the right side of the body, the first participant tested SpheriCol with a panoramic view covering the missing area. The joystick was installed on the left side for both subjects (the side of the wheelchair where subject 1 has full-arm mobility).

c) Experimental procedure and variables: The participants repeated the test circuit under the following five conditions:

* Without assistance, to familiarize themselves with the circuit (Trial I),

* SpheriCol with bird's-eye view (Trial II),

* SpheriCol with bird's-eye view and line markers (Trial III),

* SpheriCol with panoramic view and line markers (Trial IV),

* SpheriCol (with user's preferred view and markers) in conjunction with the assistance algorithm in [55] (Trial V).

d) Experimental design and evaluation criteria: To evaluate the effectiveness of SpheriCol, the following performance metrics were considered (cf. [19], [47]):

1) Total time required to complete the test circuit,

2) Number of collisions in the test circuit,

3) Number of sensors which detected an obstacle at a distance smaller than a given threshold ("danger", $d = 7$ cm; "near-collision", $d = 2$ cm),
<figure><img src="image_10.png" /><figcaption>Fig. 8: *Experimental setup in the first campaign.* (a) Test circuit (not to scale) with three photos of the two slaloms and the “nominal” path of the wheelchair (green); (b) Accurate plan of the test room obtained with OpenVSLAM [54]. The medial axis of the test room and the shortest path along it, are depicted in blue and red, respectively (see Sect. IV-C for more details).</figcaption></figure>

<table><caption>TABLE I: *Summary of the trials in the first experimental campaign.* In the 1st column, the trials in which no data were recorded (but for which the completion time and the number of collisions are available), are marked with an asterisk.</caption><thead><tr><th>Subject</th><th>Trial</th><th>Algorithm in [55]:<br>on ✓/off ✗</th><th>Visual feedback</th><th>Completion<br>time (s)</th><th>Number of<br>collisions</th></tr></thead><tbody><tr><td>1*</td><td>I</td><td>✗</td><td>None</td><td>144</td><td>15</td></tr><tr><td>1</td><td>II</td><td>✗</td><td>Bird's eye</td><td>118</td><td>11</td></tr><tr><td>1*</td><td>III</td><td>✗</td><td>Bird's eye + lines</td><td>135</td><td>15</td></tr><tr><td>1*</td><td>IV</td><td>✗</td><td>Panoramic + lines</td><td>127</td><td>7</td></tr><tr><td>1*</td><td>V</td><td>✓</td><td>Bird's eye + lines</td><td>125</td><td>0</td></tr><tr><td>2*</td><td>I</td><td>✗</td><td>None</td><td>194</td><td>4</td></tr><tr><td>2</td><td>II</td><td>✗</td><td>Bird's eye</td><td>175</td><td>3</td></tr><tr><td>2</td><td>III</td><td>✗</td><td>Bird's eye + lines</td><td>195</td><td>5</td></tr><tr><td>2</td><td>IV</td><td>✗</td><td>Panoramic + lines</td><td>194</td><td>1</td></tr><tr><td>2</td><td>V</td><td>✓</td><td>Bird's eye + lines</td><td>187</td><td>0</td></tr><tr><td>2</td><td>V'</td><td>✓</td><td>Bird's eye + lines</td><td>150</td><td>0</td></tr></tbody></table>

4) Variability of joystick input,

5) Information gathered from the users through a written survey. A questionnaire was handed out for completion to the participants, to assess the overall satisfaction and viewpoint preferences.

e) *Collected data, experimental results and discussion:* In the absence of an external motion capture system, the trajectories of EPW in each trial were estimated with OpenVSLAM [54], an accurate and flexible visual SLAM algorithm. Unfortunately, these trajectories are not aligned and they are not to scale: hence, they are not suitable for a rigorous quantitative evaluation of SpheriCol. However, we built upon the sparse 3D model of the test room generated by OpenVSLAM, to draw the accurate plan reported in Fig. 8b.

Table I summarizes the results of the experimental campaign. The last two columns of the table, report the duration of each trial and the number of collisions with the obstacles, counted by an operator visually monitoring the wheelchair. SpheriCol with the bird's-eye view and the line markers, combined with the navigation algorithm in [55], elicited the

best response in the two subjects, and it is the clear winner in terms of completion time and safety (125 s and 0 collisions for subject 1; 168.5 s on average and 0 collisions for subject 2). Giving equal importance to completion time and number of collisions, the second-best option is SpheriCol with the bird's-eye view only, and SpheriCol with the panoramic view and the line markers.

Fig. 9 and Fig. 10 report additional qualitative results from the same experimental campaign. Fig. 9 shows the number of ToF sensors which detected an obstacle at a distance of less than 7 cm (blue) or 2 cm (red) in six trials (“1-II” is a shorthand for trial II of subject 1). On the other hand, Fig. 10 reports the 2D histograms of the position of the joystick in the five recorded trials of subject 2, for the second simulated elevator (cf. Fig. 8a). The positions lie within a circle of radius 100, centered at the origin. To draw the histograms, 35 bins were considered. The heat map indicates how often the joystick traversed a bin (for better color rendering, the bin at the origin is set to zero, by default). By and large, the results in Fig. 9 and Fig. 10 show that SpheriCol with the bird's-eye view and the line markers, provides the best
<figure><img src="image_11.png" /><figcaption>Fig. 9: First experimental campaign. Number of sensors which detected an obstacle at a distance of less than 7 cm (blue) or 2 cm (red). The Arabic (Roman) numerals in the captions under each plot, refer to the subjects (trials).</figcaption></figure>

compromise between safety and variability of joystick input, during challenging maneuvers. Finally, in their questionnaires, the two subjects expressed a preference for the bird's-eye view with the line markers. In particular, subject 2 found the red line markers (which indicate that an obstacle is dangerously close to the wheelchair), very helpful during the navigation in confined spaces, like in the two simulated elevators.

In view of the encouraging results of the first experimental campaign, we decided to recruit a larger sample of able-bodied subjects and to focus on the maneuvers where SpheriCol was used the most (entering/exiting an elevator), by considering a simpler obstacle course.

### C. Second experimental campaign

a) Participants: In the second campaign, 17 (able-bodied experienced and non-experienced) subjects were recruited: 3 females and 14 males, 1 left-handed and 16 right-handed, average age of 30 years (range, 14-59 years). The research ethics committee approved the experimental protocol. The participants, who were trained on wheelchair's features, were asked to sign a consent form for the collection of data.

b) Experimental setup: The follow-up campaign took place in the Francis Querné hall, a gymnasium in the Beaulieu campus at INSA Rennes, in June 2021 (see Fig. 11). The test circuit is simpler than the one considered in the first campaign, and it only consists of a 4.1 m passageway and a 1 m × 1.85 m simulated elevator (top left in Fig. 11c). In fact, our previous

trials revealed that these are the locations where the users are expected to benefit the most from SpheriCol. Sector 3 is more challenging than sectors 1, 2 and 4, since backward motion is required. The Ricoh Theta S camera was installed 158 cm above the ground, and 60 cm above the push handles of the wheelchair.

c) Experimental procedure and variables: Each subject was requested to complete the circuit nine times in sequence, under the following conditions:

* Without assistance (three times),

* SpheriCol with bird's-eye view (three times),

* SpheriCol with bird's-eye view and line markers (three times).

We selected the three conditions above in random order, to minimize the learning effect. In this campaign, we restricted our attention to the bird's-eye view, since the wheelchair contours are clearly visible in the images (see Fig. 11b), and the free space around the wheelchair can be easily identified, to perform risk-aware maneuvers.

d) Experimental design and evaluation criteria: A five-pronged approach was adopted to evaluate SpheriCol:

1) Total time required to complete the test circuit,

2) Safety of trajectory. We computed the relative pose error of the wheelchair with respect to the "safest-possible" path (i.e. the ideal path that maximizes obstacle clearance at all times),
<figure><img src="image_12.png" /><figcaption>Fig. 10: First experimental campaign. Two-dimensional histograms of joystick position in the second simulated elevator. The Arabic (Roman) numerals in the captions under each histogram, refer to the subjects (trials).</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig. 11: Experimental setup in the second campaign. (a) One of the volunteers about to start the test circuit; (b) Bird's-eye view with overlaid line markers: the user is reversing the wheelchair, and the red marker warns him of the presence of a wall behind him; (c) The four sectors of the test circuit. In sector 3, the user must drive the wheelchair in reverse; (d) "Nominal" path of the wheelchair in the circuit (blue) and start position.</figcaption></figure>

3) Number of sensors which detected an obstacle at a distance smaller than a given threshold ("danger", $d = 7$ cm; "near-collision", $d = 2$ cm),

4) Variability of joystick input,

5) Information gathered from the volunteers through a written survey. A questionnaire was handed out for completion to the participants to assess the overall satisfaction, cognitive load, viewpoint preferences, and perceived utility of the system.

It is easy to verify that the "safest-possible" path mentioned in the 2nd item above, corresponds to the *medial axis* [56] of the test circuit. More formally, if we denote by $C \subset \mathbb{R}^2$ the room containing the test circuit and by $\partial C$ its boundary, the medial axis $M(C)$ of $C$ is defined as [57, Sect. 3.5.4]

$$M(C) = \{x \in C : \|x - x_i\| = \|x - x_j\| = \min_{y \in \partial C} \|x - y\|, x_i, x_j \in \partial C, x_i \neq x_j\},$$

where $\| \cdot \|$ denotes the Euclidean norm. Equivalently,
<table><caption>TABLE II: Summary of the trials in the second experimental campaign. The completion times in the 3rd, 5th and 7th column, are the averages over three runs per subject. The smallest (2nd smallest) values are boldface (underlined), respectively. As in Table I, the trials in which no data were recorded, are marked with an asterisk.</caption><thead><tr><th>Subject</th><th>Trials I to III</th><th>Aver. compl. time (s)</th><th>Trials IV to VI</th><th>Aver. compl. time (s)</th><th>Trials VII to IX</th><th>Aver. compl. time (s)</th></tr></thead><tbody><tr><td>2</td><td>Without assistance</td><td><b>82.33</b></td><td>Bird's eye</td><td><b>88.00</b></td><td>Bird's eye + lines</td><td>119.33</td></tr><tr><td>3</td><td>Bird's eye</td><td>50.66</td><td>Without assistance</td><td><b>32.33</b></td><td>Bird's eye + lines</td><td><u>40.00</u></td></tr><tr><td>4</td><td>Bird's eye + lines</td><td>56.00</td><td>Bird's eye*</td><td><u>50.33</u></td><td>Without assistance*</td><td><b>48.66</b></td></tr><tr><td>5</td><td>Without assistance</td><td><u>39.33</u></td><td>Bird's eye + lines</td><td>42.00</td><td>Bird's eye</td><td><b>34.00</b></td></tr><tr><td>6</td><td>Bird's eye</td><td>70.00</td><td>Bird's eye + lines</td><td>64.00</td><td>Without assistance</td><td><b>43.33</b></td></tr><tr><td>7</td><td>Bird's eye + lines</td><td>68.00</td><td>Without assistance</td><td><b>32.66</b></td><td>Bird's eye</td><td><u>46.66</u></td></tr><tr><td>8</td><td>Without assistance</td><td><b>33.66</b></td><td>Bird's eye</td><td>40.00</td><td>Bird's eye + lines</td><td><u>39.33</u></td></tr><tr><td>9</td><td>Bird's eye</td><td>49.33</td><td>Without assistance</td><td><b>31.33</b></td><td>Bird's eye + lines</td><td><u>46.33</u></td></tr><tr><td>10</td><td>Bird's eye + lines</td><td>42.33</td><td>Bird's eye</td><td><b>33.00</b></td><td>Without assistance</td><td><u>37.00</u></td></tr><tr><td>11</td><td>Without assistance</td><td><b>38.66</b></td><td>Bird's eye</td><td>48.33</td><td>Bird's eye + lines</td><td><b>43.33</b></td></tr><tr><td>12</td><td>Bird's eye</td><td>52.66</td><td>Without assistance</td><td><b>31.66</b></td><td>Bird's eye + lines</td><td><u>52.00</u></td></tr><tr><td>13</td><td>Bird's eye + lines</td><td>58.33</td><td>Bird's eye</td><td><u>48.33</u></td><td>Without assistance</td><td><b>43.00</b></td></tr><tr><td>14</td><td>Without assistance</td><td><b>33.66</b></td><td>Bird's eye + lines</td><td>52.66</td><td>Bird's eye</td><td><u>40.33</u></td></tr><tr><td>15</td><td>Bird's eye</td><td>39.00</td><td>Bird's eye + lines</td><td><u>37.00</u></td><td>Without assistance</td><td><b>33.33</b></td></tr><tr><td>16</td><td>Bird's eye + lines</td><td><u>55.66</u></td><td>Without assistance</td><td><b>43.66</b></td><td>Bird's eye</td><td>56.66</td></tr><tr><td>17</td><td>Without assistance</td><td><u>45.00</u></td><td>Bird's eye</td><td>50.00</td><td>Bird's eye + lines</td><td><b>43.33</b></td></tr><tr><td>18</td><td>Bird's eye</td><td>29.33</td><td>Without assistance</td><td><b>27.66</b></td><td>Bird's eye + lines</td><td><u>29.00</u></td></tr></tbody></table>

the medial axis is the set of centers of maximal balls, i.e., of balls in $C$ that are themselves not enclosed in another ball in $C$ [58, Sect. 54.4]. To determine the shortest path $S(C)$ along $M(C)$ connecting the start and end pose of the wheelchair, we used Dijkstra's algorithm [59, Ch. 22]. Note that $S(C)$ is a planar graph whose vertices are connected by generalized edges, which are either straight-line segments or arcs (of parabolas). The path is continuous, but it contains several cusps where a real wheelchair is forced to turn in place (which is undesirable). In spite of this limitation, we chose $S(C)$ as our ground truth in the comparative study of wheelchair's trajectories, since it is easy to compute and well-defined in environments of arbitrary shape. For the sake of illustration, the medial axis $M(C)$ of the test room $C$ considered in the first campaign, and the shortest path $S(C) \subset M(C)$ found by Dijkstra's algorithm, are depicted in blue and red, respectively, in Fig. 8b.

We computed the *Root Mean Square Error* (RMSE) of the translational component of the *Relative Pose Error* (RPE), to measure the deviation from $S(C)$ of wheelchair's trajectories in the different runs. Following [60], the RPE at time step $i$ is defined as

$$ \mathbf{E}_i \triangleq (\mathbf{Q}_i^{-1} \mathbf{Q}_{i+\Delta})^{-1} (\mathbf{P}_i^{-1} \mathbf{P}_{i+\Delta}), \quad (1) $$

where $\mathbf{P}_1, \dots, \mathbf{P}_n \in SE(3)$ is the sequence of 3D poses from the estimated trajectory, $\mathbf{Q}_1, \dots, \mathbf{Q}_n \in SE(3)$ is the sequence of poses from the ground-truth trajectory $S(C)$, and $\Delta$ is a fixed interval between two poses. From equation (1), the RMSE over all time indices of the translational component is computed as

$$ \text{RMSE}(\mathbf{E}_1, \dots, \mathbf{E}_n, \Delta) \triangleq \sqrt{\frac{1}{m} \sum_{k=1}^{m} \|\text{trans}(\mathbf{E}_k)\|^2}, \quad (2) $$

where $m = n - \Delta$ is the number of individual pose errors along the sequence, and $\text{trans}(\mathbf{E}_k)$ is the translational component of RPE, $\mathbf{E}_k$. Note that since $S(C)$ is orientation-free, we associated each sample of $S(C)$ with a reference

frame whose $z$-axis is aligned with the forward direction of the wheelchair.

**Remark 1:** Note that while the RPE is widely used in the SLAM literature, it is not the only option to measure the deviation of executed wheelchair's trajectory from the reference one. For example, in [61], the authors used a discrete version of Fréchet distance [62], to compare (the projection of) human commands with the commands generated by a planner, for autonomous navigation. The Fréchet distance accounts for the velocity and ordering of points along a trajectory, a property not shared by other classical metrics, such as the Hausdorff distance (which computes the distance between two curves without explicitly considering the paths as time-indexed trajectories). However, while the discrete variant of Fréchet distance can be computed in real time (which is of practical interest for interactive assistance systems), unlike the RPE, the orientation of a rigid body along a trajectory is not taken into account. Therefore, the RPE ultimately remains the best option for the problem under investigation. ♦

e) *Collected data, experimental results and discussion:* In the second campaign, we took advantage of a marker-based motion capture system (8 Qualysis Miqus cameras covering the test circuit) to precisely estimate the pose of the wheelchair. We expressed the poses of the Ricoh Theta S in the body frame of the wheelchair (whose origin is located at the mid-point of the two driving wheels), by applying a pure translation of 53 cm along the $z$-axis of the reference frame of the camera. Table II reports the completion time of the circuit in seconds, averaged over 3 runs per subject. We observe a fairly large inter-subject variability (which is imputable to the personal driving skills), and shorter completion times with no assistance (in fact, most of the users tried to push the limits of SpheriCol, and took more risk during the trials).

Table III reports the RMSE in centimeters, for a fixed interval $\Delta = 25$ (this value of $\Delta$ has been determined empirically, and it provides the best trade-off between sensitivity to measurement noise and degree of informativeness of RMSE). To reduce the variability of results, we com-
puted the averages over 3 runs per subject, under the same conditions (i.e., “Without assistance”, “Bird’s eye”, and “Bird’s eye + lines”, see Fig. 11b). It is apparent from Table III that the smallest (2nd smallest) RMSE is obtained with SpheriCol 88% (70%) of the time: the smallest (2nd smallest) values are boldface (underlined). The driving assistant is thus effective in centering the EPW along the circuit and it improves user’s safety, thus confirming our initial hypothesis.

As for the range measurements, in each trial, we determined how many ToF sensors detected obstacles at a distance of less than 7 cm or 2 cm. As above, we computed the average over 3 runs, under the same conditions. For a fair comparison, the results are reported in percentage: in fact, the duration of each run is different and it depends on the user’s driving skills. With the 7-cm threshold, the lowest (2nd lowest) percentage of collisions is obtained 82% (59%) of the time with SpheriCol, whereas with the 2-cm threshold, the percentages are 71% (76%). This further supports the case for improved safety of our driving assistant.

To evaluate the effectiveness of SpheriCol (and user’s confidence in a candidate collision-free trajectory), we also measured the variability of joystick input [19]. To this end, we computed the standard deviation of joystick’s azimuthal angles (degrees), in sector 3 of the test circuit (see Fig. 11c). We focused on this sector rather than on sectors 1 and 4, since the user must put the wheelchair into reverse and needs more assistance from SpheriCol. Table IV reports the averages over 3 runs per subject, of the standard deviation of joystick’s azimuthal angles, under the same conditions (“Without assistance”, “Bird’s eye”, and “Bird’s eye + lines”). The smallest (2nd smallest) standard deviations are obtained with SpheriCol 70% (76%) of the time, with deviations ranging from 2°

<table><caption>TABLE III: Second experimental campaign. RMSE (cm) with $\Delta = 25$, see equation (2). The reported values are the averages over 3 runs per subject (under the same conditions). The ground truth is the trajectory determined by Dijkstra’s algorithm on the medial axis $M(C)$ of the test room. The smallest (2nd smallest) values are boldface (underlined).</caption><thead><tr><th>Subject</th><th>Without assistance</th><th>Bird's eye</th><th>Bird's-eye + lines</th></tr></thead><tbody><tr><td>2</td><td><strong>8.74</strong></td><td>9.21</td><td><u>9.11</u></td></tr><tr><td>3</td><td><u>15.99</u></td><td><strong>8.33</strong></td><td>17.14</td></tr><tr><td>4</td><td><u>12.59</u></td><td>15.66</td><td><strong>10.74</strong></td></tr><tr><td>5</td><td>19.03</td><td><u>17.38</u></td><td><strong>14.11</strong></td></tr><tr><td>6</td><td>15.50</td><td><strong>8.10</strong></td><td><u>10.07</u></td></tr><tr><td>7</td><td>20.15</td><td><strong>12.29</strong></td><td><u>14.68</u></td></tr><tr><td>8</td><td>24.42</td><td><u>18.11</u></td><td><strong>16.98</strong></td></tr><tr><td>9</td><td>22.70</td><td><u>12.79</u></td><td><strong>11.22</strong></td></tr><tr><td>10</td><td>18.25</td><td><strong>13.12</strong></td><td><u>15.38</u></td></tr><tr><td>11</td><td>18.76</td><td><strong>13.67</strong></td><td><u>14.88</u></td></tr><tr><td>12</td><td>21.87</td><td><strong>13.57</strong></td><td><u>14.05</u></td></tr><tr><td>13</td><td>12.74</td><td><u>11.20</u></td><td><strong>9.29</strong></td></tr><tr><td>14</td><td><u>15.23</u></td><td>17.15</td><td><strong>12.43</strong></td></tr><tr><td>15</td><td><u>19.04</u></td><td>21.79</td><td><strong>18.08</strong></td></tr><tr><td>16</td><td>15.17</td><td><strong>10.70</strong></td><td><u>12.20</u></td></tr><tr><td>17</td><td><u>15.01</u></td><td><strong>13.47</strong></td><td>16.58</td></tr><tr><td>18</td><td><strong>16.30</strong></td><td><u>16.41</u></td><td>17.32</td></tr></tbody></table>

<table><caption>TABLE IV: Second experimental campaign. Standard deviation of joystick’s azimuthal angles (degrees). The reported values are the averages over 3 runs per subject (under the same conditions): they have been computed in sector 3 of the test circuit (see Fig. 11c). The smallest (2nd smallest) values are boldface (underlined), and NA stands for “Not Available”.</caption><thead><tr><th>Subject</th><th>Without assistance</th><th>Bird's eye</th><th>Bird's-eye + lines</th></tr></thead><tbody><tr><td>2</td><td><strong>132.87</strong></td><td><u>146.73</u></td><td>148.91</td></tr><tr><td>3</td><td><strong>125.88</strong></td><td><u>137.80</u></td><td>148.22</td></tr><tr><td>4</td><td>NA</td><td>NA</td><td>127.60</td></tr><tr><td>5</td><td>153.04</td><td><u>150.00</u></td><td><strong>140.32</strong></td></tr><tr><td>6</td><td><u>136.94</u></td><td><strong>129.60</strong></td><td>138.66</td></tr><tr><td>7</td><td><u>131.89</u></td><td>136.77</td><td><strong>114.08</strong></td></tr><tr><td>8</td><td><u>137.91</u></td><td><strong>134.82</strong></td><td>145.82</td></tr><tr><td>9</td><td>143.81</td><td><strong>136.59</strong></td><td><u>137.17</u></td></tr><tr><td>10</td><td><u>139.00</u></td><td><strong>133.90</strong></td><td>146.96</td></tr><tr><td>11</td><td>147.82</td><td><strong>145.65</strong></td><td><u>146.91</u></td></tr><tr><td>12</td><td>151.15</td><td><u>143.93</u></td><td><strong>137.22</strong></td></tr><tr><td>13</td><td>137.62</td><td><strong>133.50</strong></td><td><u>136.59</u></td></tr><tr><td>14</td><td><strong>145.30</strong></td><td>150.75</td><td><u>146.96</u></td></tr><tr><td>15</td><td>147.19</td><td><strong>130.41</strong></td><td><u>141.18</u></td></tr><tr><td>16</td><td><strong>140.43</strong></td><td>150.00</td><td><u>142.44</u></td></tr><tr><td>17</td><td>141.64</td><td><strong>135.79</strong></td><td><u>141.58</u></td></tr><tr><td>18</td><td><strong>141.35</strong></td><td>154.18</td><td>154.24</td></tr></tbody></table>

to 22.6° (mean deviation, 11.5°). These results mirror the findings of the first campaign, and suggest that SpheriCol contributes to smoother wheelchair trajectories.

From a statistical analysis of the answers to the questionnaire, SpheriCol received neutral to positive satisfaction and usability results from the large majority of participants (see Figs. 12a-12c). Fig. 12d reports the percentage of time, SpheriCol was used by the 17 subjects in the test circuit. In addition, see Fig. 12e, 44% of the users stated that the video stream is one of the major strengths of SpheriCol, and 16% of the participants appreciated the additional piece of information conveyed by the colored line markers. One out of five users deemed the system helpful for reversing the wheelchair and for risk management (collision avoidance), while 47% of the participants pointed out a time lag due to the computation of the bird’s-eye view from the equirectangular image: this is potentially critical for safe navigation, especially for high-speed motions (see Fig. 12f). In our current implementation, image warping is not optimized and video latency in live streaming (on the order of tens of milliseconds) adds up to the time delay introduced by VNC Connect: hence, the average frame rate is about 5 fps. The users also reported some visualization issues: the graphical information is too rich (24% of the respondents), the elevated point of view (PoV) of the camera is hard to understand on the first try (18%), and the resolution of the image is not uniform (12%). Finally, Fig. 12g indicates that 73% of the participants opted for the driving assistant (with or without the line markers). The yellow slice (28%) in Fig. 12g, includes the subjects who expressed no preferences (“None”) and those who did not answer the question (“NoA”). Note that in the pie charts in Fig. 12f and Fig. 12g, the percentages do not sum up to 100%,
<figure><img src="image_14.png" /><figcaption>Fig. 12: Summary of the answers to the questionnaire in the second experimental campaign. (a)-(c) Usability of SpheriCol and overall level of satisfaction; (d) Percentage of time, the 17 participants used SpheriCol in the test circuit (NA stands for "Not Available"); (e) Strengths, and (f) weaknesses of SpheriCol; (g) Preferred type of assistance (NoA stands for "No Answer").</figcaption></figure>

since some of the participants provided multiple answers and the digits after the decimal point have been omitted.

## V. CONCLUSION AND FUTURE WORK

In this paper, we have presented SpheriCol, a new simple yet versatile driving assistant for the users of power wheelchairs. The system, developed atop ROS, integrates spherical vision from an overhead camera with range measurements, to support navigation in confined spaces. SpheriCol has been successfully validated on a consumer-grade wheelchair equipped with our plug-and-play system, during extensive experiments with a target audience of able-bodied volunteers and people with motor impairments.

The proposed navigation system elicited the intended user's response and holds promise for achieving the next-generation robotic assistive devices. However, our clinical trials also exposed a number of limitations:

* For people who are naïve to power wheelchairs, there is not much evidence of benefit from SpheriCol in terms of task completion time, but safety and comfort are clearly improved,

* With the existing visualization interface, most of the users appreciated the live video feed from the twin-fisheye camera, and, to a less extent, the overlaid distance information,

* The present study seems hard to generalize to any clinical population of expert wheelchair users. In fact, these individuals are highly skilled at judging distances, and

they might not be willing to undertake long training sessions to embrace a new technology whose benefits become manifest only in a few complex maneuvers. In addition, the screen in front of them might obstruct a direct view of the surrounding environment, and it could be perceived as a barrier to mobility.

In future works, we will adapt the shape of distance markers according to the speed of the wheelchair (as reconstructed from joystick angle), to provide more accurate visual feedback to the user. We would also like to add a social component to SpheriCol, to improve navigation among pedestrians in crowded environments [63]. Finally, plans are afoot to develop a purely vision-based version of SpheriCol which requires no range measurements from other sensors on the wheelchair, and which provides high dynamic range omnidirectional images for both indoor and outdoor use [64]. To this end, we work towards developing a system which includes two or more twin-fisheye cameras rigidly attached to the same support, as in [65], [66]. As an alternative, one could adopt a machine-learning approach inspired by [67], [68], to estimate depth from a single spherical panorama.

## Supplementary material

For further details, a video featuring an EPW equipped with SpheriCol in a selection of our clinical trials, is available at the address reported at the bottom of the page<fnref n="1" />. In particular, the video shows the visualization interface described in Sect. III-C.

<footnote n="1">¹https://home.mis.u-picardie.fr/~fabio/Eng/Video/DelmasTMRB23.mp4</footnote>
## ACKNOWLEDGMENTS

This work was carried out as part of the Interreg VA France (Channel) England ADAPT project "Assistive Devices for empowering disAbled People through robotic Technologies" (http://adapt-project.com). The Interreg FCE Programme is a European Territorial Cooperation programme that aims to fund high quality cooperation projects in the Channel border region between France and England. The Programme is funded by the European Regional Development Fund (ERDF).

The authors would like to thank Bastien Fraudet and Émilie Leblong (Pôle Saint-Hélier, Rennes), and Louise Devigne (IRISA/Inria Rennes and INSA Rennes) for their assistance during the clinical trials and for helpful discussion.

## REFERENCES

[1] D. Taylor, "Americans With Disabilities: 2014," US Department of Commerce, Economics and Statistics Administration, US Census Bureau, 2018, www.census.gov.

[2] World Health Organization (WHO), "Assistive Technology," 2018, www.who.int/news-room/fact-sheets/detail/assistive-technology.

[3] B. Rebsamen, C. Guan, H. Zhang, C. Wang, C. Teo, M. Ang, and E. Burdet, "A Brain Controlled Wheelchair to Navigate in Familiar Environments," *IEEE Trans. Neur. Sys. Reh.*, vol. 18, no. 6, pp. 590-598, 2010.

[4] T. Carlson and Y. Demiris, "Collaborative Control for a Robotic Wheelchair: Evaluation of Performance, Attention, and Workload," *IEEE Trans. Syst. Man Cy. B*, vol. 42, no. 3, pp. 876-888, 2012.

[5] A. Erdogan and B. D. Argall, "The effect of robotic wheelchair control paradigm and interface on user performance, effort and preference: an experimental assessment," *Robot. Autonom. Syst.*, vol. 94, pp. 282-297, 2017.

[6] Y. Morales, A. Watanabe, F. Ferreri, J. Even, K. Shinozawa, and N. Hagita, "Passenger discomfort map for autonomous navigation in a robotic wheelchair," *Robot. Autonom. Syst.*, vol. 103, pp. 13-26, 2018.

[7] Z. Lei, B. Tan, N. Garg, L. Li, A. Sidarta, and W. Ang, "An Intention Prediction Based Shared Control System for Point-to-Point Navigation of a Robotic Wheelchair," *IEEE Robot. Autonom. Lett.*, vol. 7, no. 4, pp. 8893-8900, 2022.

[8] C. Torkia, D. Reid, N. Korner-Bitensky, D. Kairy, P. Rushton, L. Demers, and P. Archambault, "Power wheelchair driving challenges in the community: a users' perspective," *Disabil. Rehabil. Assist. Technol.*, vol. 10, no. 3, pp. 211-215, 2015.

[9] S. Sundaram, H. Wang, D. Ding, and R. A. Cooper, "Step-Climbing Power Wheelchairs: A Literature Review," *Top. Spinal Cord Inj. Rehabil.*, vol. 23, pp. 98-109, 2017.

[10] J. Podobnik, J. Rejc, S. Šlajpah, M. Munih, and M. Mihelj, "All-Terrain Wheelchair: Increasing Personal Mobility with a Powered Wheel-Track Hybrid Wheelchair," *IEEE Robot. Autonom. Lett.*, vol. 24, no. 4, pp. 26-36, 2017.

[11] H. Ikeda, T. Toyama, D. Maki, K. Sato, and E. Nakano, "Cooperative step-climbing strategy using an autonomous wheelchair and a robot," *Robot. Autonom. Syst.*, vol. 135, p. 103670, 2021.

[12] J. L. Candiotti, B. J. Daveler, S. Sivakanthan, G. G. Grindle, R. Cooper, and R. A. Cooper, "Curb Negotiation With Dynamic Human-Robotic Wheelchair Collaboration," *IEEE Trans. Hum.-Mach. Syst.*, vol. 52, pp. 149-155, 2022.

[13] A. Botta, R. Bellincioni, and G. Quaglia, "Autonomous detection and ascent of a step for an electric wheelchair," *Mechatronics*, vol. 86, 2022, Article no. 102838.

[14] N. Welage and K. Liu, "Wheelchair accessibility of public buildings: a review of the literature," *Disabil. Rehabil. Assist. Technol.*, vol. 6, no. 1, pp. 1-9, 2011.

[15] S. Sivakanthan, J. L. Candiotti, A. S. Sundaram, J. A. Duvall, J. J. G. Sergeant, R. Cooper, S. Satpute, R. L. Turner, and R. A. Cooper, "Mini-review: Robotic wheelchair taxonomy and readiness," *Neurosci. Lett.*, vol. 772, 2022, Article no. 136482.

[16] T.-V. How, R. Wang, and A. Mihailidis, "Evaluation of an intelligent wheelchair system for older adults with cognitive impairments," *J. NeuroEng. Rehabil.*, vol. 10, no. 1, 2013, Article no. 90.

[17] P. Viswanathan, E. Zambalde, G. Foley, J. Graham, R. Wang, B. Adhikari, A. Mackworth, A. Mihailidis, W. Miller, and I. Mitchell, "Intelligent wheelchair control strategies for older adults with cognitive impairment: User attitudes, needs, and preferences," *Auton. Robot.*, vol. 41, no. 3, pp. 539-554, 2017.

[18] É. Leblong, B. Fraudet, L. Devigne, M. Babel, F. Pasteau, B. Nicolas, and P. Gallien, "SWADAPT1: assessment of an electric wheelchair-driving robotic module in standardized circuits: a prospective, controlled repeated measure design pilot study," *J. NeuroEng. Rehabil.*, vol. 18, no. 1, 2021, Article no. 140.

[19] G. Sorrento, P. Archambault, F. Routhier, D. Dessureault, and P. Boissy, "Assessment of Joystick control during the performance of powered wheelchair driving tasks," *J. NeuroEng. Rehabil.*, vol. 8, no. 1, 2011.

[20] M. Gillham, M. Pepper, S. Kelly, and G. Howells, "Feature determination from powered wheelchair user joystick input characteristics for adapting driving assistance," *Wellcome Open Res.*, vol. 2, no. 93, 2018.

[21] M. Zolotas, J. Elsdon, and Y. Demiris, "Head-Mounted Augmented Reality for Explainable Robotic Wheelchair Assistance," in *Proc. IEEE/RSJ Int. Conf. Intel. Robots Syst.*, 2018, pp. 1823-1829.

[22] G. Vailland, Y. Gaffary, L. Devigne, V. Gouranton, B. Arnaldi, and M. Babel, "Vestibular Feedback on a Virtual Reality Wheelchair Driving Simulator: A Pilot Study," in *Proc. ACM/IEEE Int. Conf. Human-Robot Inter.*, 2020, pp. 171-179.

[23] G. Vailland, L. Devigne, F. Pasteau, F. Nouviale, B. Fraudet, É. Leblong, M. Babel, and V. Gouranton, "VR based Power Wheelchair Simulator: Usability Evaluation through a Clinically Validated Task with Regular Users," in *Proc. IEEE Virt. Real. 3D User Interf.*, 2021, pp. 420-427.

[24] S. Levine, D. Bell, L. Jaros, R. Simpson, Y. Koren, and J. Borenstein, "The NavChair Assistive Wheelchair Navigation System," *IEEE Trans. Rehabil. Eng.*, vol. 7, no. 4, pp. 443-451, 1999.

[25] E. Demeester, E. Vander Poorten, A. Hüntemann, and J. De Schutter, "Wheelchair Navigation Assistance in the FP7 Project RADHAR: Objectives and Current State," in *IEEE/RSJ Int. Conf. Intel. Robots Syst. Workshop on "Navigation and Manipulation Assistance for Robotic Wheelchairs"*, 2012.

[26] F. Morbidi, L. Devigne, C. Teodorescu, B. Fraudet, E. Leblong, T. Carlson, M. Babel, G. Caron, S. Delmas, F. Pasteau, G. Vailland, V. Gouranton, S. Guégan, R. Le Breton, and N. Ragot, "Assistive Robotic Technologies for Next-Generation Smart Wheelchairs: Codesign and Modularity to Improve Users' Quality of Life," *IEEE Rob. Autom. Mag.*, 2023, in press, DOI: 10.1109/MRA.2022.3178965.

[27] J. Vogel, D. Leidner, A. Hagengruber, M. Panzirsch, B. Bäuml, M. Denninger, U. Hillenbrand, L. Suchenwirth, P. Schmaus, M. Sewtz, A. Bauer, T. Hulin, M. Iskandar, G. Quere, A. Albu-Schäffer, and A. Dietrich, "An Ecosystem for Heterogeneous Robotic Assistants in Caregiving: Core Functionalities and Use Cases," *IEEE Rob. Autom. Mag.*, vol. 28, no. 3, pp. 12-28, 2021.

[28] J. Leaman and H. La, "A Comprehensive Review of Smart Wheelchairs: Past, Present, and Future," *IEEE Trans. Hum.-Mach. System*, vol. 47, no. 4, pp. 486-499, 2017.

[29] M. Kutbi, X. Du, Y. Chang, B. Sun, N. Agadakos, H. Li, G. Hua, and P. Mordohai, "Usability Studies of an Egocentric Vision-Based Robotic Wheelchair," *ACM Trans. Hum.-Rob. Interact.*, vol. 10, no. 1, 2020, Article no. 4.

[30] F. Pasteau, V. Narayanan, M. Babel, and F. Chaumette, "A visual servoing approach for autonomous corridor following and doorway passing in a wheelchair," *Robot. Autonom. Syst.*, vol. 75, Part A, pp. 28-40, 2016.

[31] J. Nguyen, S. Su, and H. Nguyen, "Experimental Study on a Smart Wheelchair System Using a Combination of Stereoscopic and Spherical Vision," in *Proc. IEEE Int. Conf. Eng. Med. Biol.*, 2013, pp. 4597-4600.

[32] S. Li, T. Fujiura, and I. Nakanishi, "Recording Gaze Trajectory of Wheelchair Users by a Spherical Camera," in *Proc. Int. Conf. Rehab. Rob.*, 2017, pp. 929-934.

[33] R. Komatsu, H. Fujii, Y. Tamura, A. Yamashita, and H. Asama, "Free viewpoint image generation system using fisheye cameras and a laser rangefinder for indoor robot teleoperation," *Robomech J.*, vol. 7, 2020, Article no. 15.

[34] M. Oehler and O. von Stryk, "A Flexible Framework for Virtual Omnidirectional Vision to Improve Operator Situation Awareness," in *Proc. 10th Eur. Conf. Mob. Rob.*, 2021.

[35] S. Delmas, F. Morbidi, G. Caron, J. Albrand, M. Jeanne-Rose, L. Devigne, and M. Babel, "SpheriCol: A Driving Assistance System for Power Wheelchairs Based on Spherical Vision and Range Measurements," in *Proc. IEEE/SICE Int. Symp. System Integ.*, 2021, pp. 505-510.

[36] "Braze Mobility, Blind Spot Sensors for Wheel Chairs & Mobility Scooters," https://brazemobility.com.
[37] A. Erdogan and B. Argall, “The effect of robotic wheelchair control paradigm and interface on user performance, effort and preference: An experimental assessment,” *Robot. Autonom. Syst.*, vol. 94, pp. 282–297, 2017.

[38] M. Jayasuriya, R. Ranasinghe, and G. Dissanayake, “Active Perception for Outdoor Localisation with an Omnidirectional Camera,” in *Proc. IEEE/RSJ Int. Conf. Intel. Robots Syst.*, 2020, pp. 4567–4574.

[39] A. Hartman and V. Nandikolla, “Human-Machine Interface for a Smart Wheelchair,” *J. Robot.*, 2019, Article no. 4837058.

[40] K. Klinich, M. Manary, N. Orton, K. Boyle, and J. Hu, “A Literature Review of Wheelchair Transportation Safety Relevant to Automated Vehicles,” *Int. J. Environ. Res. Public Health*, vol. 19, no. 3, 2022, Article no. 1633.

[41] X. Ying and Z. Hu, “Can We Consider Central Catadioptric Cameras and Fisheye Cameras within a Unified Imaging Model?” in *Proc. Eur. Conf. Comp. Vis.*, 2004, pp. 442–455.

[42] C. Geyer and K. Daniilidis, “A Unifying Theory for Central Panoramic Systems and Practical Implications,” in *Proc. Eur. Conf. Comp. Vis.*, 2000, pp. 445–461.

[43] J. P. Barreto, “A unifying geometric representation for central projection systems,” *Comput. Vis. Image Und.*, vol. 103, no. 3, pp. 208–217, 2006.

[44] G. Caron and F. Morbidi, “Spherical Visual Gyroscope for Autonomous Robots using the Mixture of Photometric Potentials,” in *Proc. IEEE Int. Conf. Robot. Automat.*, 2018, pp. 820–827.

[45] L. I. Iezzoni, *When Walking Fails: Mobility Problems of Adults with Chronic Conditions*. University of California Press, 2003.

[46] A. Riddering, “Keeping Older Adults with Vision Loss Safe: Chronic Conditions and Comorbidities that Influence Functional Mobility,” *J. Vis-ual Impair. Blin*, vol. 102, no. 10, pp. 616–620, 2008.

[47] L. Devigne, M. Aggravi, M. Bivaud, N. Balix, C. Teodorescu, T. Carlson, T. Spreters, C. Pacchierotti, and M. Babel, “Power Wheelchair Navigation Assistance Using Wearable Vibrotactile Haptics,” *IEEE Trans. Haptics*, vol. 13, no. 1, pp. 52–58, 2020.

[48] “Sunrise Medical, Powered Wheelchairs by QUICKIE,” www.sunrisemedical.co.uk.

[49] “Ricoh Theta S camera,” www theta360 com.

[50] “STMicroelectronics: Long distance ranging Time-of-Flight sensor,” www.st.com/en/imaging-and-photonics-solutions/v15311x.html.

[51] “Intel NUC mini PC,” www.intel.com/content/www/us/en/products/boards-kits/nuc.html.

[52] “RealVNC: Remote access software for desktop and mobile devices,” www.realvnc.com.

[53] A. Parton, P. Malhotra, and M. Husain, “Hemispatial neglect,” *J. Neurol. Neurosur Ps.*, vol. 75, no. 1, pp. 13–21, 2004.

[54] S. Sumikura, M. Shibuya, and K. Sakurada, “OpenVSLAM: A versatile visual SLAM framework,” in *Proc. 27th ACM Int. Conf. Multimedia*, 2019, pp. 2292–2295.

[55] L. Devigne, F. Pasteau, T. Carlson, and M. Babel, “A shared control solution for safe assisted power wheelchair navigation in an environment consisting of negative obstacles: a proof of concept,” in *Proc. IEEE Int. Conf. Systems, Man and Cyber*, 2019, pp. 1043–1048.

[56] K. Siddiqi and S. Pizer, Eds., *Medial Representations: Mathematics, Algorithms and Applications*. Springer, 2008, vol. 37 of Computational Imaging and Vision.

[57] A. Okabe, B. Boots, K. Sugihara, and S. Chiu, *Spatial Tessellations: Concepts and Applications of Voronoi Diagrams*, 2nd ed. John Wiley & Sons, 2000.

[58] J. E. Goodman, J. O'Rourke, and C. D. Tóth, Eds., *Handbook of Discrete and Computational Geometry*, 3rd ed., ser. Discrete Mathematics and its Applications. CRC press, 2018.

[59] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, *Introduction to Algorithms*, 4th ed. MIT press, 2022.

[60] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of RGB-D SLAM systems,” in *Proc. IEEE/RSJ Int. Conf. Intel. Robots Syst.*, 2012, pp. 573–580.

[61] C. X. Miller, T. Gebrekristos, M. Young, E. Montague, and B. D. Argall, “An Analysis of Human-Robot Information Streams to Inform Dynamic Autonomy Allocation,” in *Proc. IEEE/RSJ Int. Conf. Intel. Robots Syst.*, 2021, pp. 1872–1878.

[62] H. Alt and M. Godau, “Computing the Fréchet distance between two polygonal curves,” *Int. J. Comput. Geom. Ap.*, vol. 5, no. 1 and 2, pp. 75–91, 1995.

[63] F. Grzeskowiak, D. Gonon, D. Dugas, D. Paez-Granados, J. Chung, J. Nieto, R. Siegwart, A. Billard, M. Babel, and J. Pettré, “Crowd against the machine: A simulation-based benchmark tool to evaluate and compare robot capabilities to navigate a human crowd,” in *Proc. IEEE Int. Conf. Robot. Automat.*, 2021, pp. 3879–3885.

[64] G. Caron, F. Morbidi, and M. Babel, “HaDROs: Omni-HDR Vision for Localization and Cartography in Environments with a Wide Range of Radiance,” 2022-2024, Exploratory research project of GdR 720 ISIS, www.gdr-isis.fr.

[65] N.-H. Wang, B. Solarte, and Y.-H. Tsai, “360SD-Net: 360° Stereo Depth Estimation with Learnable Cost Volume,” in *Proc. IEEE Int. Conf. Robot. Automat.*, 2020, pp. 582–588.

[66] A. Meuleman, H. Jang, D. Jeon, and M. Kim, “Real-Time Sphere Sweeping Stereo from Multiview Fisheye Images,” in *Proc. IEEE Conf. Comp. Vis. Pattern Recogn.*, 2021, pp. 11423–11432.

[67] N. Zioulis, A. Karakottas, D. Zarpalas, and P. Daras, “OmniDepth: Dense Depth Estimation for Indoors Spherical Panoramas,” in *Proc. Eur. Conf. Comp. Vis.*, 2018, pp. 448–465.

[68] H. Jiang, Z. Sheng, S. Zhu, Z. Dong, and R. Huang, “UniFuse: Unidirectional Fusion for 360° Panorama Depth Estimation,” *IEEE Robot. Autonom. Lett.*, vol. 6, no. 2, pp. 1519–1526, 2021.

<figure><img src="image_15.png" /></figure>

**Sarah Delmas** received the engineer's degree in mechatronics, robotics and control engineering from ENSTA Bretagne, France, in 2019. She was an intern at Polytechnic University of Catalonia, Spain, and at the University of Picardie Jules Verne, France, in 2018 and 2019, respectively. She was a research engineer at the MIS laboratory, University of Picardie Jules Verne, from 2019 to 2021. She is currently an electronics engineer with Astek, Nantes, France.

<figure><img src="image_16.png" /></figure>

**Fabio Morbidi** (Senior Member, IEEE) received the Ph.D. degree in control engineering and robotics from the University of Siena, Italy, in 2009. He was a visiting researcher at the University of California, Santa Barbara, USA, in 2008, and he held post-doctoral positions at Northwestern University, University of Texas at Arlington, USA, and at Inria Grenoble Alpes, France. Since 2014, he has been an Associate Professor of robotics with the University of Picardie Jules Verne, France. He currently serves as Associate Editor for the IEEE TRANSACTIONS ON ROBOTICS and for the IEEE ROBOTICS AND AUTOMATION LETTERS.

His main research interests include multi-agent systems and robotic vision.

<figure><img src="image_17.png" /></figure>

**Guillaume Caron** (Senior Member, IEEE) received the Ph.D. degree in robotics and the HDR degree from the University of Picardie Jules Verne (UPJV), France, in 2010 and 2019, respectively. He has been an Associate Professor with the UPJV since 2011, and a CNRS delegate with CNRS-AIST Joint Robotics Laboratory (JRL), IRL, Japan, since 2019. He has been the co-director of JRL since 2022. He has been serving as Associate Editor for the IEEE ROBOTICS AND AUTOMATION LETTERS since 2023. His research interests include artificial vision for robotics, real-time visual tracking and servoing, and digital heritage.

<figure><img src="image_18.png" /></figure>

**Marie Babel** received the Ph.D. degree and the HDR degree from INSA Rennes, France, in 2005 and 2012, respectively. She has been a Full Professor with INSA Rennes since November 2022, and she is a member of the Rainbow team at IRISA/Inria Rennes. She was the local coordinator of the Interreg FCE ADAPT project, she is the co-leader of ISI4NAVE team with University College London, and the coordinator of DORNELL Inria Challenge. She works in the field of assistive robotics and her research focuses on the design of mobility aids for people with disabilities, including rehabilitation devices and multi-sensory interfaces for human-robot interaction and social navigation.

<figure><img src="image_19.png" /></figure>

**François Pasteau** received the Ph.D. degree in signal and image processing from INSA Rennes, France, in 2011. From 2015 to 2018, he was the R&D manager at Ergovie, a company specialized in medical equipments, including power wheelchairs. He is currently a research engineer at INSA Rennes and a member of the Rainbow team at IRISA/Inria Rennes. His main research interests include assistive robotics and embedded computing systems.