<figure><img src="image_1.png" /></figure>

HAL
open science

# F2T: A Novel Force-Feedback Haptic Architecture
## Delivering 2D Data to Visually Impaired People

Simon Gay, Edwige Pissaloux, Katerine Romeo, Ngoc-Tan Truong

► To cite this version:

Simon Gay, Edwige Pissaloux, Katerine Romeo, Ngoc-Tan Truong. F2T: A Novel Force-Feedback Haptic Architecture Delivering 2D Data to Visually Impaired People. IEEE Access, 2021, 9, pp.94901 - 94911. 10.1109/access.2021.3091441. hal-03665578

HAL Id: hal-03665578

https://hal.science/hal-03665578

Submitted on 11 May 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Received May 28, 2021, accepted June 11, 2021. Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.

Digital Object Identifier 10.1109/ACCESS.2021.3091441

# F2T: A Novel Force-Feedback Haptic Architecture
Delivering 2D Data to Visually Impaired People

SIMON L. GAY<sup>1</sup>, EDWIGE PISSALOUX<sup>2</sup>, KATERINE ROMEO<sup>2</sup>,
AND NGOC-TAN TRUONG<sup>2</sup>

Corresponding author: Simon L. Gay (simon.gay@univ-grenoble-alpes.fr)

This work was supported in part by the Fondation Internationale de la Recherche Appliquée sur le Handicap (FIRAH), in part by the Comité National Coordination Action Handicap (CCAH), in part by the Normandy University, in part by the Région Normandie, in part by The European Commission, in part by the Centre National de la Recherche Scientifique (CNRS) and the Ministère de l'Europe et des Affaires Etrangères (MEAE) (PHC Polonium Program, France Embassy in London), and in part by the Espace Handicap of Rouen University.

**ABSTRACT** Today's technology still limits the accessing to 2D information (images, graphics, charts, maps, paintings, etc.) for visually impaired people (VIP). Existing solutions for human-machine interfaces are expensive and have yet to be available to the many. Therefore, we propose a novel and inclusive haptic architecture to access and interact with 2D data, relying on the force-feedback principle, and named Force-Feedback Tablet (F2T). The F2T's architecture is based on a flat thumbstick mounted on a 2D actuated support, enabling force feedback effects on user's finger. The flat thumbstick measures the user's intended movements independently from mobile support's actual movements, enabling highly interactive effects, static and dynamic effects, or guidances. Moreover, the mechanical structure does not need to be backdrivable, simplifying the motion control, and reducing the costs of the device. Finally, the F2T architecture allows designing lightweight, compact and scalable devices. To validate and demonstrate the relevance of the F2T architecture, we present two lightweight prototypes. Two series of preliminary tests with VIP and blindfolded participants confirmed the system's effectiveness at providing shape and spatial layout understanding. Results obtained during the evaluation are encouraging with high recognition rates.

**INDEX TERMS** Haptic interfaces, force feedback, human-computer interaction, visually impaired people, accessibility.

## I. INTRODUCTION

Nowadays, graphical data is increasingly ubiquitous. Most information sources have been digitized, and the advent of smartphones, social media platforms, and wearable smart devices have powered an ever-growing exchange of video and image content over the web. Thus, culture and education are getting increasingly reliant on pictures, schematics, videos, or maps. Accessing this information is difficult for Visually Impaired People (VIP). There are 285 million VIP worldwide (including 39 million blind people) [1]. Although many solutions have been proposed to help tackle this issue over the years [2], they provide limited access to graphics. Their main drawbacks are the reliance on pre-made (text-audio) descriptions, their linear nature, and the lack of spatial

information [3]. This prevents their users to get general content, or at-a-glance information.

Another solution is also widely used, relief documents, such as maps with magnets, thermoformed surfaces or 3D representations of objects. Especially, they are used for educational purposes in classic or orientation-mobility classes of dedicated schools. However, such documents are usually expensive to print, static, and require well-thought simplifications to be intuitively understood. They often present many details, and the sighted person assistance is necessary to understand the presented information.

More recently, electronic touch simulating devices have been developed to convey refreshable haptic information (cf. infra). Compared to audio-based solutions, haptic devices allow the active exploration of the represented graphical content. Therefore, they provide more information about the explored element's spatial structure. Many theoretical and experimental studies have highlighted the importance of

The associate editor coordinating the review of this manuscript and approving it for publication was Zheng Chen<sup>1</sup>.

<footnote n="1"><sup>1</sup>LCIS, University of Grenoble-Alpes, 26 902 Valence, France</footnote>

<footnote n="2"><sup>2</sup>LITIS, University of Rouen Normandy, 76 130 Rouen, France</footnote>
active exploration to learn new sensori-motor control laws. It is a necessary prerequisite to the emergence of spatial awareness ( [4], [5]).

Despite those advantages, few haptic devices are developed for VIP [6]. Such devices usually require an external interface which often relies on specific hardware. As a result, their cost is often prohibitive. Furthermore, they do not allow for intuitive “non-visual” scene recognition, limited by the feedback possibilities of existing touch-stimulation technologies. To allow an intuitive exploration, a device should provide information to both localize and recognize the various elements (i.e. objects or areas) of the scene through haptic exploration. Localizing and recognizing an object requires the integration of its shape through movement, and relies on edge perception. Tactile edges may be induced with tactile gradient perception through our fingertips [7], the object’s textures provide additional information and reduce the cognitive load during the recognition.

Many efforts have been devoted to designing efficient actuated systems for tactile displays ( [8], [9]). However, most haptic technologies provide either edge or texture-based perception. Consequently, they present incomplete representations of data which prevents the user to understand graphical information [3]. This lack of appropriate haptic feedback technology is a real obstacle to VIP inclusion in today's digital society. Therefore, we propose a novel architecture to design the graphical displays for VIP.

Our architecture relies on sensori-motor theories of perception [5] and interactive force-feedback to allow the intuitive perception of both the shape and texture of objects. Our proposed architecture, the Force Feedback Tablet (F2T), is based on an original 2D force feedback system, and employs a combination of interactive effects (controlled by both the user's movements and image content) and guided movements to help understanding complex dynamic graphical content. The F2T architecture will allow the development of affordable haptic interface using force-feedback effects to facilitate access to graphical information (such as art, schemas or maps).

The paper is organized as follows: Section II describes a representative overview of current devices based on the force feedback principle. Section III introduces the theoretical basis, principles and possibilities of the F2T. Sections IV and V present the two prototypes of the F2T and the results of their evaluations. Section VI recalls this research's main contributions and offers some future development perspectives.

## II. RELATED WORKS ON FORCE FEEDBACK HAPTIC DEVICES

Current technologies for 2D data display consider haptic surfaces and force feedback devices.

Haptic surfaces can be classified into three categories: taxel-based matrices, stimulation surfaces, and wearable actuators. Taxel surfaces (e.g. [10]–[12]) allow free exploration of the whole surface and feeling object edges;

however, they display images with a limited (fixed by their construction) resolution, and cannot display textures. Stimulation surfaces (e.g. [13], [14]) display textures and are less expensive than taxel surfaces, but they cannot display edges and are limited to single-touch exploration. In general, wearable actuators (e.g. [15], [16] can display textures, some of them allow even multi-touch exploration; nevertheless, to work properly, they usually require additional devices.

The force-feedback devices try to overcome these limits in textures and landforms display. Indeed, they aim to simulate physical properties of virtual objects by applying mechanical feedback to the user's body (mainly hand and forearm). Recent solutions propose to (re-)present 2D or 2.5D objects. Kim et al. [17], Hausberger et al. [18], and Maiero et al. [19] address solutions based on a touchscreen mounted on actuated structures. These structures generate a force feedback by moving the touchscreen to simulate 2.5D elements with different kinds of frictions. Despite their effectiveness, these devices use a cumbersome structure and require the user to see the screen. Consequently, these solutions are inappropriate for VIP.

Other approaches use a small surface or a stylus attached to a force feedback robotic arm ( [21], [26]), or a 2 degrees of freedom arm integrated into a desktop computer; that makes the user interact with the screen's content [20]. However, like the previously mentioned devices, these systems are cumbersome and expensive.

Many 2D haptic systems use a pantograph architecture ([22], [23]). As the arm's position is acquired by measuring motor angles, these systems require a high backdrivability to allow the user to manipulate easily the devices' arms. This backdrivability implies no reduction gears and high-torque motors. The position is estimated using encoders on motor axes, requiring the end effector to move to get the user's intentions. This approach limits the possibilities of interactions with the user. Moreover, this device cannot provide a consistent maximum force and velocity on the usable surface, and the complex control model is necessary to move arms properly. The interaction issue of pantographs has been partially solved by acquiring the intended orientation with a rotary end-effector; that makes it possible to move an avatar in an interactive environment [24]. Generally, the specific pantograph arms architecture makes these devices bulky for a limited available surface. It cannot be considered as an alternative for a portable system used by the VIP.

Saga and Raskar [25] propose a 2D device based on an actuated sliding pad using a 2D SPIDAR architecture. The sliding pad is moved over a touchscreen with four actuated pulling wires. This system generates the haptic effect simulating frictions and the height variations with a compact design. However, it requires some movement of the end effector which limits the interactive possibilities with the user.

Consequently, we present a novel force feedback architecture, named F2T, which provides edge and texture effects, high interactive capabilities, and a high resolution dynamic
<table><caption>TABLE 1. Comparison of some solutions for force feedback devices with the proposed architecture.</caption><thead><tr><th></th><th>Ratio Surface/Size</th><th>Actuation Requirement</th><th>User Input</th><th>User Interactively</th><th>Non-backdrivable</th><th>For VIP</th></tr></thead><tbody><tr><td>ActuatedTouchscreen [17]–[19]</td><td>Low</td><td>&gt;= 2 low torque motors</td><td>Touchscreen</td><td>Movement - independent</td><td>Yes</td><td>No</td></tr><tr><td>Robotic Arm [20], [21]</td><td>Highly low</td><td>&gt;=2 high torque motors</td><td>Motor encoders</td><td>Movement - dependant</td><td>No</td><td>Yes</td></tr><tr><td>Pantograph Architecture [20], [22], [23]</td><td>Low</td><td>2 high torque motors</td><td>Motor encoders</td><td>Movement - dependant</td><td>No</td><td>Yes</td></tr><tr><td>Dualpanto [24]</td><td>Low</td><td>2 high torque motors</td><td>Motor encoders</td><td>Rotation - dependant<br>Translation - independant</td><td>No</td><td>Yes</td></tr><tr><td>2D spider [25]</td><td>High</td><td>4 Low power gear motors</td><td>Touchscreen</td><td>Encoders<br>partially movement - dependant</td><td>No</td><td>Yes</td></tr><tr><td>Our architecture</td><td>High</td><td>2 low power gear motors</td><td>Thumbstick</td><td>Movement - fully independant</td><td>Yes</td><td>Yes</td></tr></tbody></table>

<figure><img src="image_3.png" /><figcaption>FIGURE 1. Principle of the Force Feedback Tablet (F2T): the F2T (left) consists of a mobile actuated support with a flat thumbstick on top of it. The control system (right) controls the mobile support in position to follow the user's finger using thumbstick' values. The control system gets the haptic information about the touched point (x, y) of the displayed image (here, a map where colors define height and friction properties, c.f. Section IV-A) to define an offset to induce in the position control of the mobile support, and producing a force feedback on the user's finger through the centering spring of the thumbstick (c.f. Figure 2).</figcaption></figure>

display of 2D data. In Table 1, we compare the technique of our architecture with the recent solutions.

The originality of this device is twofold: a) the use of a mobile flat thumbstick as the end effector to estimate the user's intended movement independently from actual movement; this allows a large variety of complex haptic and kinesthetic effects, and simplifying motion control and mechanical actuation; b) the use of a simple yet efficient structure allowing the design of lightweight, compact, scalable, and affordable assistive devices suitable for VIP's daily tasks. The principle of F2T is outlined in Section III.

### III. A NOVEL FORCE FEEDBACK HAPTIC ARCHITECTURE

The Force Feedback Tablet (F2T, Figure 1) is a new architecture for devices using force-feedback to generate haptic effects such as height/depth variations, borders (edges) and textures. This device acquires the user's intentions independently from the current end-effector movement via the thumbstick (Figure 2). This specificity allows applying a wide variety of dynamic and interactive haptic effects such as flow or movement deviation. We consider these effects as *active* as they require external movements to be represented.

<figure><img src="image_4.png" /><figcaption>FIGURE 2. A flat thumbstick is placed on top of the mobile support (in blue) to acquire user's intended movement. As the user manipulates the thumbstick cap, the difference *J* between the finger's position and the support's position is measured. The motion control moves then the mobile support to follow user's finger. By inducing offsets in this motion control, it is possible to set a predefined offset in the position difference *J*. The centering springs of the thumbstick apply a force that is nearly proportional to the offset, allowing to control the force experienced by the user.</figcaption></figure>

#### A. PRINCIPLE OF THE FORCE FEEDBACK TABLET

The F2T is an architecture based on a 2D frame. The mobile support is actuated through two orthogonal motorized axes to allow the thumbstick to be moved on the X and
Y axes. Therefore, the frame is similar to a tracing table. However, it uses a speed control instead of position control. A thumbstick is used as input for the user's movement intentions. The frame is servo-controlled in speed based on this input to follow the user's finger.

A flat thumbstick consists of two orthogonal linear potentiometers which give the position difference $\vec{J} = (x_j, y_j)$ in $[-1, 1]^2$ of the thumbstick cap ($P_{finger}$) from thumbstick base ($P_{support}$) within a range of several millimeters. Centering springs maintain the thumbstick cap to its center position, and apply an opposite force when the cap is moved, that can be considered as proportional to the distance from the center position.

The user's intended movement can be measured (through $\vec{J}$) independently from the support's movement making possible to use non-reversible transmission gears (e.g. high reduction gear or screw-nut gears). Therefore, the mobile support can stop the user's movement without powering the motors. Moreover, the mobile support movements are not influenced by the lateral forces applied by the user, allowing haptic effects and movements of higher precision. Instead of letting the user to move the mobile support, the system uses the thumbstick vector $\vec{J}$ (Figure 1) to detect the user's finger intentions and control the mobile support speed through proportional control in (1) or a PID controller to define the motors' speed command $\vec{V} = (V_x, V_y)$ and make the mobile support following user's finger:

$$ \vec{V} = \alpha \cdot \vec{J}, \quad \vec{J} = P_{finger} - P_{support} \quad (1) $$

where $\alpha$ is the maximum speed value. Therefore, the support follows the user's finger by minimizing the value of $|\vec{J}|$. Note that in this configuration, the user only manipulates the thumbstick cap, which significantly reduces the problem of perceived inertia of the manipulated device. Then, the force feedback is obtained by applying offsets on the support movements, simulating perturbations produced by the presence of virtual objects or textures (Figure 1). The user experiences force feedback through the thumbstick's centering spring that generates a force proportional to $-\vec{J}$.

In Section II.B, we note $\vec{V}$ the initial intended speed, that is the movement generated by the device to follow the user's finger. The output speed $\vec{V}'$ is modified by haptic effects that is actually applied to the mobile support, and $P = (x_p, y_p)$ the current position of the mobile support on the available surface.

## B. MOTOR CONTROL

The F2T's principles allow producing several haptic effects, which add offsets to the control speed $\vec{V}$. These effects can be cumulated by applying them successively. In this case, the output speed $\vec{V}'$ of an effect is used as the input speed vector $\vec{V}$ of the next effect. The number and order of applied effects can be changed according to the desired haptic perceptions to induce. In its first academic prototypes, the following effects were proposed and implemented:

frictions, edges and reliefs, attractors, flows and railings. The guidance modes of the device are also presented.

### 1) FRICTION

Friction is the simplest effect. It reduces the speed of the mobile support to oppose the intended movement. Two friction types can be simulated: fluid and solid. With fluid friction, the initial speed is reduced proportionally (2):

$$ \vec{V}' = \vec{V} - \vec{V} \cdot f_f(x_p, y_p) \quad (2) $$

where $f_f \in [0; 1[$ is the fluid friction coefficient. When $f_f$ is close to 1, the effect produces the feeling of moving in a viscous liquid.

The solid friction effect is generated by reducing the initial speed with a constant value when the speed is higher than this value, and stopping the movement otherwise (3):

$$ \vec{V}' = \begin{cases} \vec{V} - f_s(x_p, y_p) \cdot \frac{\vec{V}}{|\vec{V}|} & \text{when } |\vec{V}| > f_s(x_p, y_p) \\ 0 & \text{else} \end{cases} \quad (3) $$

where $f_s \in [0; 1[$ is the solid friction coefficient. When $f_s$ is close to 1, the effect produces the feeling of moving on a rubber surface. Different levels of solid and fluid frictions help users discriminate different objects through their simulated textures, and thus recognize them faster.

### 2) EDGE AND RELIEF

Relief simulation is done by applying an offset proportional to the gradient of the virtual object's $S$ height. Then, the resulting speed is defined as (4):

$$ \vec{V}' = \vec{V} - \overrightarrow{grad} S(x_p, y_p) \quad (4) $$

Various effects can be experienced by applying this principle (Figure 3). When the slope of the object's edge is small, the user's finger will be either slowed down or accelerated depending on the direction of the gradient. This effect can be used to explore a bas-relief or to guide the users along a trench, maintaining their finger on a path.

When the gradient is strong enough, the movement will stop. In that case, the finger's user can slide along the object's edge which makes the user easily recognizes object's shape and object's outline.

### 3) ATTRACTORS

The F2T allows creating spatial attractor effects. These attractors are 2D areas towards which the mobile support will be pulled or repulsed with varying intensity. The attractor's strength of pull depends on the distance between the finger's user and the closest point of the attractor. This attraction force is applied to guide the thumbstick until reaching the attractor when the thumbstick is moving outside the attractor area. This force is defined by (5):

$$ \vec{V}' = \vec{V} + \frac{B}{d(P, M)^2} \times \frac{\overrightarrow{PM}}{|\overrightarrow{PM}|} \quad (5) $$
<figure><img src="image_5.png" /><figcaption>**FIGURE 3.** a) (Side view) the gradient of the surface height is subtracted from the initial speed, decreasing it when climbing or increasing it when going down, simulating height variations. b) (Top view) when the height gradient is high, such as in the case of a border, the mobile support is stopped and slide along the border, allowing the feeling of the edge of an object.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>**FIGURE 4.** a) The surface defines a vector field that adds a vector $\vec{F}(x_p, y_p)$ to the initial speed, simulating a dynamic surface such as a waterflow. b) The railing system constraints the speed toward a predefined direction, given by direction of $\vec{R}(x_p, y_p)$ by reducing the perpendicular component of $\vec{V}$. This effect can be used to guide the user along a given direction.</figcaption></figure>

where B is the character force of the attractor, M is the closest of attractor point, and P is the position of thumbstick. The attractor effect can be used to guide the user in a better way than a trench, as the user's finger is attracted when it goes out of a path.

## 4) FLOW

As the tablet is actuated, it is possible to simulate a dynamic area by defining an offset vector at each position (x, y) of the image (Figure 4a). The flow effect adds the associated offset to the position of the thumbstick (6):

$$ \vec{V}' = \vec{V} + \vec{F}(x_p, y_p) \quad (6) $$

The flow effect allows simulating mobile (dynamic) elements, such as water or wind, that carry the user's finger along with a specific flow.

## 5) RAILING

The railing effect intends to partially constrain the user to move in a given direction (Figure 4b). Unlike trenches and attractors, the railing effect (7) does not guide the user on a predefined path, but deviates the movement make it closer to the predefined direction. The rail direction vector $\vec{R}$ can be defined at each position (x, y) of the image. The railing effect

is generated by reducing the component of speed vector $\vec{V}$ that is orthogonal to the direction vector $\vec{R}(x, y)$:

$$ \vec{V}' = \vec{V}_{\vec{u}} + \vec{V}_{\vec{u}T} \times (1 - \frac{\|\vec{R}(x, y)\|}{\|\vec{R}(x, y)\|}, \quad \vec{u} = \frac{\vec{R}(x, y)}{\|\vec{R}(x, y)\|}) \quad (7) $$

where $\vec{R}(x, y)$ is the rail vector at current position $P = (x, y)$, and $\|\vec{R}(x, y)\| \in [0, 1]$, $\vec{V}_{\vec{u}}$ is the component of $\vec{V}$ on direction of $\vec{R}(x, y)$, and $\vec{V}_{\vec{u}T}$ is the orthogonal component that is reduced according to the norm of $\vec{R}$. This effect can be used to lead the user in a given direction, or increase the perception movements.

## C. F2T GUIDANCE MODES

The tablet is actuated and can move without being influenced by user's movements, therefore, it offers an original feature to explore the haptic image. Indeed, it is possible to "forcefully" guide the user along a path. The proposed guiding system uses a sequence of waypoints defining the path to follow. The thumbstick moves towards the first point $T_i$ of the sequence, with a predefined speed $v_i$. When the thumbstick's position is close enough to this point ($d(P, T_i) < d_{min}$, where $d_{min}$ must be defined according to the precision of the tablet), point $T_i$ is removed from the sequence and, if the sequence is not empty, the thumbstick moves towards the next point $T_{i+1}$. Pauses can be added among points to provide additional feedback, such as an audio description of the currently explored area.

Two guiding principles are proposed: full guidance, and guidance through a series of attractors. In the fully guided mode, the user's inputs are no longer taken into account. Therefore, the thumbstick is only driven by the control system as in (8):

$$ \vec{V}' = v_i \times \frac{\vec{PT}_i}{\|\vec{PT}_i\|} \quad (8) $$

where $T_i$ is the following point of the sequence and $v_i$ is the speed associated with the point $T_i$.

In attractor sequence mode, points successively attract the thumbstick with a constant force (9). However, the user's input is still taken into account, allowing resist the movement and experience other effects.

$$ \vec{V}' = \vec{V} + f_i \times \frac{\vec{PT}_i}{\|\vec{PT}_i\|} \quad (9) $$

where $f_i$ is the attraction force associated with the attracting point $T_i$. These guidance principles allow showing important places and paths on a map, or even to tell the story behind a piece of art.

This Section presented the principles of F2T and a set of haptic effects. These principles have been integrated in two academic prototypes, presented in Section IV and Section V.

# IV. EVALUATION OF THE FIRST PROTOTYPE

## A. F2T FIRST DESIGN

We applied the F2T architecture to build a first prototype, shown in Figure 5. Its concept is validated by blindfolded
<figure><img src="image_7.png" /><figcaption>**FIGURE 5.** First version of the F2T built using a Lego frame. The mobile support is actuated with two continuous rotation servomotors, and the thumbstick's position is obtained using a top-view camera detecting the servomotor's position with blue markers placed on them.</figcaption></figure>

and visually impaired people through several experiments. The prototype's mainframe is made of Lego pieces. The mobile support is actuated by two continuous rotation servomotors, through a pinion and rack transmission. The user's movements are detected using a flat thumbstick which is made from portable game consoles. The thumbstick's position is obtained using a top-view camera detecting the position of the servomotors on their respective racks. The device is controlled by a standard PC through an Arduino Nano board. This prototype weighs around 400g. The F2T's software is implemented in Java, and the tactile properties are encoded using up to four classes of RGB images.

The first class of images encodes fluid friction, solid friction and height: for each pixel of the image, red channel encodes the fluid friction, blue channel encodes solid friction and green channel indicates the elevation of the pixel. The luminosity of color corresponds to different levels of friction (friction coefficients in (2) and (3)) or levels of height (gradient in (4)). Figure 6 gives some examples of tactile images.

The second class for flows (6) and the third class for rails (7) are both encoded using specific images which define a vector ($\vec{F}$ or $\vec{R}$) in each pixel through the red and green channels, corresponding to the X and Y components respectively, as illustrated in Figure 7.

The fourth class for attractor effects is encoded through binary images where white pixels define the attractor's area in Figure 8.

The guidance is defined using a text file, where each line defines a point in the path sequence with its coordinates, the speed or attraction force, and guidance mode. This file is loaded by the F2T software to generate a path (as illustrated in Figure 9).

Using this list of files, the control software generates a virtual environment that the user can explore. We developed

<figure><img src="image_8.png" /><figcaption>**FIGURE 6.** Examples of color-coded representations of the haptic effects used to simulate image properties. From left to right: fluid friction, solid friction and elevation. The last image combines these three properties (yellow is a combination of red and green).</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>**FIGURE 7.** Illustrations of color-coded representation of the flow and rail effects, where each pixel's color encodes a vector. Left: the red and green components of a pixel color encode the X and Y components of the vector. Middle: colour chart. Right: example of a flow image descriptor.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>**FIGURE 8.** Example of attractor image. White areas attract user's finger like magnets. We intend to use such a feature in reading assistance applications to help spatializing texts and control text-to-speech software speed.</figcaption></figure>

a simple script language to load or change images and paths, and to play sound files when predefined conditions are fulfilled. This scripted system makes it possible to define scenarios and synchronize the guidance with audio-descriptions, or to change the virtual environment configuration.

## B. FIRST PROTOTYPE'S EVALUATION

We evaluated the suitability of the device to convey shape and movements information through three evaluation tests without time limits: E1, E2, and E3:

- E1 evaluates the perception of movement direction (cardinal directions and their combinations), cf Figure 10.

- E2 evaluates the perception of a sequence of curve directions (Figure 11) and basic geometric shapes (Figure 12). Participants must answer by reproducing the shape or angle manually.

- E3 perception of complex shapes, using curved lines representing rivers, a snake and interconnected circles, inspired by Australian aboriginal paintings.<fnref n="1" /> These shapes

<footnote n="1">¹*Dream of the Snake* by Warlimpirrnga Tjapaltjarri, and *Three Rivers* by Rover Thomas Joolama, *Musée du Quai Branly*, France.</footnote>
<figure><img src="image_11.png" /><figcaption>**FIGURE 9.** Guidance points (green points) on the interface of F2T's software. The generated path is used to guide the user through the image, facilitating a subsequent free exploration.</figcaption></figure>

are illustrated in Figure 13. Participants had to answer by describing the perceived shapes through oral feedback and gestures.

Fourteen participants from both France and Poland performed these tests; 7 were congenitally blind, 3 late blind and 4 blindfolded. There are eight women and six men; six of them were in the age range 40 to 60 and the others were in the age range 20 to 40. Half of the participants were from academia, the other half from various VIP charities. The candidates do not have other impairments

<figure><img src="image_12.png" /><figcaption>**FIGURE 10.** Line directions. Starting point of thumbstick is a black circle. From left to right the figure present north to south, west to east, south-east to north-west, north-east to south-west and south-west to north to south-east directions.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>**FIGURE 11.** Curve directions. Starting point of thumbstick is a black circle. From left to right the figure present south-west to north-east, north-east to south-west, south-west to north to south-east directions and S-shape.</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>**FIGURE 12.** Basic forms. Starting point of thumbstick is a black circle. From left to right the figure present square, circle and triangle form.</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>**FIGURE 13.** Examples of complex figures. Left: *Three Rivers* painting. The F2T guides the user through the three rivers and their confluence. Middle: part of the *Dream of the Snake* painting. The shape of the snake's body was accentuated to cope with the first prototype's smoothness limitations. Right: simplified representation of the circles and path patterns of the *Dream of the Snake* painting.</figcaption></figure>

(no comorbidities). Each participant declared being unfamiliar with force-feedback devices.

Each participant signed an informed consent form. The participants were placed in front of a table with the F2T. A brief description of the F2T was given, followed by a two minutes training with the device's thumbstick manipulation.

Then, the evaluation consisted of the recognition of a series of simple movements and geometrical shapes. Firstly, 8 cardinal directions were tested in E1. Secondly, simple shapes were tested in E2. Finally, complex shapes in Figure 13 were tested in E3.

After three evaluation tests, we show the recognition rate in Figure 14 for simple directions and simple geometric shapes. The average overall recognition rate for simple shapes is 89.3%. The participants easily recognized orthogonal directions, curves and geometric figures, while the diagonal recognition rate was lower ($\mu$ = 75%).

For complex shapes, every participant reproduced them correctly. The length of the rivers and their relative location were adequately estimated. However, the connected circles were correctly recognized by less than half of the participants.

### C. DISCUSSION

The perceived stimulations were qualified as suitable and non-invasive, which made their cognitive interpretation possible. Overall, simple shapes were recognized easily by the participants. The observed confusion between curves and stepped lines, and the lower recognition rates on diagonals may be explained by the thumbstick's jerky movements. Those jerky movements are the weakness of this

<figure><img src="image_16.png" /><figcaption>**FIGURE 14.** Recognition rates for simple directions and simple geometric figures.</figcaption></figure>
<figure><img src="image_17.png" /><figcaption>**FIGURE 15.** Second prototype of the F2T. The mobile support (thumbstick) is actuated with two high-torque electric gearmotors, and its position is obtained using magnetic encoders mounted on the motors' axes, allowing to measure the support position with low latency.</figcaption></figure>

prototype's motion control system: the motors, at low speed, had insufficient power to drive the thumbstick in straight diagonals and the long control loop (40ms) prevented smooth motion control.

However, the interpretation is influenced by the end-user's education; therefore, an additional sensory channel would help provide complementary data to haptic and tactile representations.

Following this first series of tests, the participants suggested several possible improvements for the device:

- increase the fluidity of the movements, which is critical for curve recognition;

- use variations in guidance speed to simulate different perceptual effects;

- implement multiscale exploration, through distinct levels of details, in order to better perceive complex figures;

- let the participants freely explore the usable area of the device before the tests;

- let the participants control the interaction type (guidance vs free exploration).

The above reasons motivate us to build the second prototype (cf. Section V).

# V. EVALUATION OF THE SECOND PROTOTYPE

Most of the hardware and software related feedbacks collected during the evaluation of the first prototype were integrated into the design of the second one. This section introduces the F2T second prototype and its experimental evaluation.

## A. F2T SECOND DESIGN

The F2T's second prototype (Figure 15) was built with 3D-printed pieces and acrylic plates. This prototype is actuated with DC gearmotors (6V, reduction of 250:1, 120RPM, 40N.cm⁻¹ under 9V with a maximum current of 500mA), which allow more precise and fluid movements

<figure><img src="image_18.png" /><figcaption>**FIGURE 16.** Six different angles presented in a randomized order. The green area represents *walls* forming an acute, right or obtuse angle. The participants could freely move in the black area, but were blocked by the green one.</figcaption></figure>

with a maximum speed of 8cm.s⁻¹. The thumbstick's position is obtained through encoders instead of a camera (first prototype) to reduce the latency to 1ms (compared 40 ms of the camera); it can increase the reliability and responsiveness of the motion control system, and also allow a more compact design. The usable surface is increased from 15 × 15 cm (in first prototype) to 20 × 20 cm; therefore, more detailed image's displays (and zoom in/out operations) can be offered. The prototype weighs light around 900g and the cost is low (does not exceed 100 €).

An improved control system based on two control loops was designed. The first loop controls the position of a virtual thumbstick (the theoretical position). The second loop constrains the real mobile support to follow the virtual thumbstick position using a PID controller. This solution increases the fluidity and precision of the movements by reducing the delay of the main control loop and by removing the accumulation of errors on the estimation of the thumbstick's position (due to inertia and the motor's imprecisions).

To evaluate its performance, two experiments were designed: angle perception in Section V.B and room layout perception in Section V.C.

Twelve blindfolded participants were involved in this study (9 of them were in the age range of 20 to 30, and the others were in the age range of 30 and 50). We involved only blindfolded participants as some experiments require drawing again the haptic image (displayed on the F2T). These participants did not participate in the evaluation of the first prototype, and they declared being unfamiliar with force-feedback devices.

## B. ANGLE PERCEPTION

### 1) PROTOCOL AND RESULTS

In the first step, six angles were presented in a random order to every participant: 45, 90, 135, 225, 270 and 315 degrees (Figure 16). They were asked to freely explore the image and to slide along the two borders forming of the angle when they found them. They were tasked to give the perceived approximate angle value orally when ready. The exploration time was recorded.

The collected results are summarized in Figure 17. The average exploration time for all 6 angles was 58.5 seconds (σ = 12.8 seconds). Right angles were easily identified, with a Mean Average Error (MAE) of 1.7 degrees. The highest MAE was observed for 45° angle (MAE = 9.5°) and for the 135° angle (MAE = 8.2°).
<figure><img src="image_19.png" /><figcaption>**FIGURE 17.** Mean Average Error (red - in degrees) and response times (blue - in seconds) for the six tested angles. The perceived angles were expressed in the range 0-180 degrees instead of 0-360 degrees (e.g. the angle of 315 degrees was expressed as this of 45 degrees).</figcaption></figure>

## 2) DISCUSSION

Overall, this first experiment shows that the F2T can be used to follow and recognize angles of an object's edge with an average angle estimation error of less than 10° across all tested angles. Right angles (90° and 270°) seem to be easier to recognize, because horizontal and vertical movements are smoother than diagonal ones. Besides, this could also be that right angles occur more frequently in our daily environment; as a results we are more familiar with them.

The participants needed longer exploration times for re-entrant angles (> 180°) as the explorable surface (black areas in Figure 16) is larger. The reason is moving fast around the junction point of re-entrant angles; this may cause the loss of contact with the followed edge due to inertia. While salient angles (< 180°) are easy to follow by slightly pushing against their borders. Indeed, we often observed that users lost time trying to retrieve the lost re-entrant angles.

## C. LAYOUT PERCEPTION

### 1) PROTOCOL AND RESULTS

The second experiment consisted of the exploration of 4 virtual rooms, which is symbolizing the layout of an apartment. The participants were informed that each room has an entrance and an exit door, with exit doors identified by a high friction area (shown in red color in Figure 18). The layouts of the rooms were presented from simple to complex. Each room was explored successively by the 12 blindfolded participants, starting at the entrance door. Before moving to the following room, the participants were tasked to draw the outline of the explored room (without the blindfold). This outline must include the position of the entrance and exit doors. The exploration time was recorded in each trial.

To understand deeper the layout perception, the participants were divided in two groups with respect to of two experimental conditions. The first group started by freely exploring the virtual room until they had a clear idea of its shape, and were then asked to show the location of the exit door, and to draw the perceived layout of the room on paper.

<figure><img src="image_20.png" /><figcaption>**FIGURE 18.** The four virtual rooms generated by the F2T (left). The walls of the rooms are shown in green, and cannot be crossed with the thumbstick. The entrance door was always at the bottom of the image (in front of the user), and the exit door (in red) was encoded by a specific force feedback *texture*. An example of the drawings from one of the participants is given on the right.</figcaption></figure>

<figure><img src="image_21.png" /><figcaption>**FIGURE 19.** Average Exploration Time (in seconds) for the 4 explored room layouts: free exploration times (blue), and free exploration time after the guided exploration (red).</figcaption></figure>

The second group were first guided in the virtual room by the F2T's guidance mode; then, they could freely explore the room until they felt confident enough to be able to draw its shape.

The room's shape and the door's locations were perceived and reproduced correctly in most trials. The layout of right angled rooms was the most easily apprehended, whereas rooms with diagonal walls usually proved more confusing (Figure 18).

Figure 19 shows the Average Exploration Times (AET) for each room for both experimental conditions. The AET in the guided condition is 44.4% faster than the free-exploration condition (53.0s vs 95.4s).

### 2) DISCUSSION

This second experiment shows that the F2T is an efficient interface to help to recognize layouts, and more generally, 2D graphical data. Indeed, the shapes of all rooms were recognized well, with an average exploration time of 74.2 seconds across both conditions.

We noticed a higher difficulty perceiving diagonals and acute angles (i.e. sharp corners), which may be partly
explained by the thumbstick producing non-smooth and slightly curved diagonal movements, a consequence of the non-uniform friction of the 3D printed pieces on the X and Y axes. The participants also mentioned the difficulty of memorizing the shapes of more complex rooms (i.e. rooms with more diagonals and non-orthogonal angles), which might explain why the shape of these rooms are more difficult to reproduce.

Subjective reports gathered during post-test discussions highlighted that the participants found useful of the active exploration. That makes them to apprehend better and faster the investigated shape. However, they reported that the guiding speed was sometimes too high, and the prototype did not provide absolute distance information during exploration.

These results highlight the importance of active exploration to facilitate spatial comprehension using the F2T, which further confirms prior results and observations by others authors ([17], [27], [28]).

## VI. CONCLUSION

This paper proposed an original force-feedback-based architecture for haptic devices conveying 2D graphical data for VIP. This architecture relies on the user's movement acquisition through a flat thumbstick. This approach has many advantages:

- Firstly, this architecture removes the need for backdrivable mechanisms, which allows using less powerful, lighter and less expensive gearmotors. It is also possible to block user's movements without powering the motors and to move the thumbstick with greater precision. As the user moves the thumbstick cap only, it also reduces the problem of the perceived inertia when manipulating the device.

- Secondly, the architecture can use an XY plotting mechanism with linear gears (e.g. pinion-rack or screwnut), which (1) simplifies the motion control, (2) provides a consistent maximum speed and force on the whole surface, and (3) makes the system design easily scalable in different sizes for different uses (e.g. large devices for home uses or smaller devices for mobile uses).

- Finally, the proposed motion control allows users to experience virtual surfaces through simulated haptic properties, such as perceiving their edges or textures. Moreover, this approach makes it possible to generate active effects influencing the movements of the user, allowing to represent dynamic elements (such as sea waves, whirlpools of the air, etc.) and propose more interactive tactile/haptic experiences. Such haptic effects may also guide and assist VIP in spatializing and understanding 2D data.

To validate the F2T architecture, we proposed two prototypes to test motion controls and possibilities offered by this architecture. After a proof of concept, the second prototype confirmed its capability to generate relevant haptic stimuli; the second prototype confirmed its relevance for assistance in performing basic mental spatialization tasks. Collected results with both prototypes indicate that the F2T can be used to convey graphical information to blind users

through force-feedback. The guidance and free exploration modes help to understand the displayed content.

The F2T could be applied in multiple educational, rehabilitational or recreational contexts for VIP, such as map reading and space awareness training [29], assisted text reading (combined with Text-To-Speech technologies), picture and art exploration, or video/serious games. The F2T brings a haptic dimension to paintings and images, and induces haptic aesthetic impressions for both sighted and blind people. It allows multimodal art discovery and new ways to interact with art.

Moreover, F2T provides the scientific community with a novel experimental platform upon which new computational and cognitive models can be tested such as human-machine interactions, spatial awareness, emotion emergence from stimulations and proprioception.

Additional tests with end users will be continuously performed in order to improve the future versions of prototypes.

## ACKNOWLEDGMENT

The authors would like to thank the test participants from Universities of Rouen Normandy, Le Havre Normandy and Technical University of Lodz, and the members of Normandie-Lorraine, Fédération des Aveugles de France (FAF), The Association Valentin Haüy, and Association PERCEVOIR.

## REFERENCES

[1] *Blindness and Vision Impairment*. Accessed: Jun. 2021. [Online]. Available: https://www.who.int/fr/news-room/fact-sheets/detail/blindness-and-visual-impairment

[2] W. Elmannai and K. Elleithy, "Sensor-based assistive devices for visually-impaired people: Current status, challenges, and future directions," *Sensors*, vol. 17, no. 3, p. 565, Mar. 2017.

[3] S. O'Modhrain, N. A. Giudice, J. A. Gardner, and G. E. Legge, "Designing media for visually-impaired users of refreshable touch displays: Possibilities and pitfalls," *IEEE Trans. Haptics*, vol. 8, no. 3, pp. 248-257, Jul. 2015.

[4] J. K. O'Regan, *Why Red Doesn't Sound Like a Bell: Understanding the Feel of Consciousness*. New York, NY, USA: Oxford Univ. Press, 2011.

[5] J. K. O'Regan and A. Noë, "A sensorimotor account of vision and visual consciousness," *Behav. Brain Sci.*, vol. 24, no. 5, pp. 939-973, Oct. 2001.

[6] M. Gori, G. Cappagli, A. Tonelli, G. Baud-Bovy, and S. Finocchietti, "Devices for visually impaired people: High technological devices with low user acceptance and no adaptability for children," *Neurosci. Biobehav. Rev.*, vol. 69, pp. 79-88, Oct. 2016.

[7] A. Bicchi, D. Dente, and E. P. Scilingo, "Haptic illusions induced by tactile flow," in *Proc. Eurohaptics*, 2003, pp. 2412-2417.

[8] S. Choi and K. J. Kuchenbecker, "Vibrotactile display: Perception, technology, and applications," *Proc. IEEE*, vol. 101, no. 9, pp. 2093-2104, Sep. 2013.

[9] L. A. Jones and N. B. Sarter, "Tactile displays: Guidance for their design and application," *Hum. Factors, J. Hum. Factors Ergonom. Soc.*, vol. 50, no. 1, pp. 90-111, Feb. 2008.

[10] S. Simeonov and N. Simeonova, "Graphical interface for visually impaired people based on bi-stable solenoids," *Int. J. Soft Comput. Softw. Eng.*, vol. 3, pp. 128-131, Jan. 2014.

[11] J. J. Zarate, O. Gudozhnik, A. S. Ruch, and H. Shea, "Keep in touch: Portable haptic display with 192 high speed taxels," in *Proc. CHI Conf. Extended Abstr. Hum. Factors Comput. Syst.*, May 2017, pp. 349-352.

[12] D. Prescher, J. Bornschein, W. Köhlmann, and G. Weber, "Touching graphical applications: Bimanual tactile interaction on the HyperBraille pin-matrix display," *Universal Access Inf. Soc.*, vol. 17, no. 2, pp. 391-409, Jun. 2018.
[13] O. Bau, I. Poupyrev, A. Israr, and C. Harrison, "TeslaTouch: Electrovibration for touch surfaces," in *Proc. Annu. ACM Symp. User Interface Softw. Technol.*, 2010, pp. 283–292.

[14] M. Amberg, F. Giraud, B. Semail, P. Olivo, G. Casiez, and N. Roussel, "STIMTAC: A tactile input device with programmable friction," in *Proc. 24th Annu. ACM Symp. Adjunct User Interface Softw. Technol.*, 2011, pp. 7–8.

[15] I. Mo Koo, K. Jung, J. C. Koo, J.-D. Nam, Y. K. Lee, and H. R. Choi, "Development of soft-actuator-based wearable tactile display," *IEEE Trans. Robot.*, vol. 24, no. 3, pp. 549–558, Jun. 2008.

[16] T. Nakamura and A. Yamamoto, "Multi-finger surface visuo-haptic rendering using electrostatic stimulation with force-direction sensing gloves," in *Proc. IEEE Haptics Symp. (HAPTICS)*, Feb. 2014, pp. 489–491.

[17] S.-C. Kim, B.-K. Han, J. Seo, and D.-S. Kwon, "Haptic interaction with virtual geometry on robotic touch surface," in *Proc. SIGGRAPH Asia Emerg. Technol.*, 2014, pp. 1–3.

[18] T. Hausberger, M. Terzer, F. Enneking, Z. Jonas, and Y. Kim, "SurfTics—Kinesthetic and tactile feedback on a touchscreen device," in *Proc. IEEE World Haptics Conf. (WHC)*, Jun. 2017, pp. 472–477.

[19] J. Maiero, E. Kruijff, A. Hinkenjann, and G. Ghinea, "ForceTab: Visuo-haptic interaction with a force-sensitive actuated tablet," in *Proc. IEEE Int. Conf. Multimedia Expo (ICME)*, Jul. 2017, pp. 169–174.

[20] J. Solis, S. Marcheschi, O. Portillo, M. Raspolli, C. A. Avizzano, and M. Bergamasco, "The haptic desktop: A novel 2D multimodal device," in *Proc. 13th IEEE Int. Workshop Robot Hum. Interact. Commun.*, Sep. 2004, pp. 521–526.

[21] S. Rasool and A. Sourin, "Haptic interaction with 2D images," in *Proc. 10th Int. Conf. Virtual Reality Continuum Appl. Ind.*, 2011, pp. 13–22.

[22] G. Campion, Q. Wang, and V. Hayward, "The pantograph MK-II: A haptic instrument," in *Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst.*, Aug. 2005, pp. 193–198.

[23] M. O. Martinez, J. Campion, T. Gholami, M. K. Rittikaidachar, A. C. Barron, and A. M. Okamura, "Open source, modular, customizable, 3-D printed kinesthetic haptic devices," in *Proc. IEEE World Haptics Conf. (WHC)*, Jun. 2017, pp. 142–147.

[24] O. Schneider, J. Shigeyama, R. Kovacs, T. J. Roumen, S. Marwecki, N. Boeckhoff, D. A. Gloeckner, J. Bounama, and P. Baudisch, "DualPanto: A haptic device that enables blind users to continuously interact with virtual worlds," in *Proc. 31st Annu. ACM Symp. User Interface Softw. Technol.*, Oct. 2018, pp. 877–887.

[25] S. Saga and R. Raskar, "Simultaneous geometry and texture display based on lateral force for touchscreen," in *Proc. World Haptics Conf. (WHC)*, Apr. 2013, pp. 437–442.

[26] K. Kostopoulos, K. Moustakas, D. Tzovaras, G. Nikolakis, C. Thillou, and B. Gosselin, "Haptic access to conventional 2D maps for the visually impaired," *J. Multimodal User Interfaces*, vol. 1, no. 2, pp. 13–19, Jun. 2007.

[27] C. Magnusson, H. Caltenco, S. Finocchietti, G. Cappagli, G. Wilson, and M. Gori, "What do you like? Early design explorations of sound and haptic preferences," in *Proc. 17th Int. Conf. Hum.-Comput. Interact. With Mobile Devices Services Adjunct (MobileHCI)*, Aug. 2015, pp. 766–773.

[28] E. E. Pissaloux, R. Velazquez, and F. Maingreaud, "A new framework for cognitive mobility of visually impaired users in using tactile device," *IEEE Trans. Hum.-Machine Syst.*, vol. 47, no. 6, pp. 1040–1051, Dec. 2017.

[29] Y. Tao, L. Ding, and A. Ganz, "Indoor navigation validation framework for visually impaired users," *IEEE Access*, vol. 5, pp. 21763–21773, 2017.

<figure><img src="image_22.png" /></figure>

**SIMON L. GAY** received the degree in computer science engineering from ENSEEIHT, Toulouse, France, in 2010, and the Ph.D. degree from Claude Bernard University, Lyon, France, in 2014.

From 2017 to 2020, he was a Postdoctoral Fellow with LITIS, University of Rouen Normandy, France. Since 2020, he has been an Associate Professor at the University of Grenoble-Alpes, France. His research interests include developmental learning, artificial knowledge building and behavior emergence through interaction with the environment, space integration, and navigation. From 2017 to 2020, he applies his researches on assistance for visually impaired people navigation and space awareness. He proposed the force feedback tablet (F2T) as a solution to e-data accessibility, such as navigation map.

**EDWIGE PISSALOUX**, photograph and biography not available at the time of publication.

<figure><img src="image_23.png" /></figure>

**KATERINE ROMEO** received the B.S. degree in electrical engineering from Bosphorus University, Istanbul, Turkey, in 1981, and the Ph.D. degree from the Polytechnical Institute of Grenoble, France, in 1985.

She is currently an Associate Professor at the Department of Multimedia and Internet Professions, Institute of Technology of Rouen, University of Rouen Normandy, France. She works on accessibility of websites and accessible writing for visually impaired and blind. She participates in the evaluation of tactile stimulators and the analysis of multimodal perception and navigation assistance. Her research interests include developing multimodal assistive devices with inclusive technologies for sighted and visually impaired people, human-machine interaction, image processing and segmentation, and computer vision.

<figure><img src="image_24.png" /></figure>

**NGOC-TAN TRUONG** received the master's degree from the University of Paris Sud, in 2016, the second master's degree from the University of Rennes 1, in 2017, and the Ph.D. degree in electronics from ENSTA-Bretagne, in 2020.

Among other subjects, he worked on remote sensing, navigation, and data fusion. He currently holds a postdoctoral position with the LITIS, University of Rouen Normandy. His research interests include human-computer interaction, sensory substitution, psychophysics, computer vision, and computational models of memory, perception, and navigation.

...