# HaptiComm: A Touch-Mediated Communication Device for Deafblind Individuals

Basil Duvernoy, Zhanat Kappassov, Sven Topp, Jeraldine Milroy, Shuangshuang Xiao, Inès Lacôte, Azamat Abdikarimov, Vincent Hayward, Mounia Ziat

► To cite this version:

Basil Duvernoy, Zhanat Kappassov, Sven Topp, Jeraldine Milroy, Shuangshuang Xiao, et al.. HaptiComm: A Touch-Mediated Communication Device for Deafblind Individuals. IEEE Robotics and Automation Letters, 2023, 8 (4), pp.2014-2021. 10.1109/LRA.2023.3241758. hal-04218964

HAL Id: hal-04218964

https://hal.science/hal-04218964v1

Submitted on 26 Sep 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# HaptiComm: A Touch-Mediated Communication Device for Deafblind Individuals

Basil Duvernoy¹,⁶ ID, Zhanat Kappassov ²,⁵ ID, Senior Member, IEEE, Sven Topp³, Jeraldine Milroy⁴,

Shuangshuang Xiao², Inès Lacôte¹,⁷, Azamat Abdikarimov⁵ ID, Vincent Hayward¹,⁸ ID, Life Fellow, IEEE,

Mounia Ziat² ID

**Abstract**—Deafblindness is a unique disability characterized by a dual sensory reduction of both hearing and vision. For some Deafblind individuals, communication via touch may be their most accessible sensory channel. Multiple techniques that rely purely on touch exist within the Deafblind community. One method of interest referred to as “Deafblind Tactile Finger-spelling Alphabet” in Australia or “Deafblind Manual Alphabet” in the UK, comprises twenty-six tactile symbols representing the letters of the Latin alphabet. This paper describes the Hapti-Comm, a device designed to reproduce the sensations generated during fingerspelling communication. The HaptiComm comprises an array of twenty-four strategically placed electrodynamic actuators specifically designed to produce distinct tactile sensations upon which the fingerspelling alphabet is constructed. The first experimental evaluation showed promising results suggesting further investigations related to the timing and the pace at which the stimuli are produced.

**Index Terms**—Haptic Interfaces, Deafblindness, Tactile Communication, Fingerspelling.

## I. INTRODUCTION

There is a myriad of ways to define Deafblindness [1]. According to the World Federation of the Deafblind (WFDB), it is “a combined vision and hearing impairment of such severity that it is hard for the impaired senses to compensate for each other” [2]. Overall impact depends on the critical period of brain plasticity at the time of impairment, severity of loss, and order in which hearing and visual impairments occur [3]. Depending on varying constraints regarding the definition, the estimated world population of Deafblind individuals varies from 0.2% (16 million) to 2% (160 million) [2].

When neither vision nor hearing sufficiently convey information, the sense of touch is transformed into an alterna-

tive modality for communication. Several methods of tactile communication exist; Refreshable Braille [4] is the most widely used and commercially available technology for tactile communication. Assistive technologies targeting Deafblind people are inordinately rare and when available, they are often individually designed. Additionally, 45% of individuals using custom-made devices reported difficulty using them [5]. In most cases, the technology used is intended for blind or Deaf people with single sensory impairment. In general, assistive technologies often suffer from extremely steep learning curves and thus, fail to transition to a development phase. Hence the importance of user studies during the research and development cycle of these devices.

To contribute to this field, we introduce the HaptiComm, a tactile communication device that focuses on tactile finger-spelling (TFS), a communication method using tactile symbols to represent letters, words, ideas, or concepts. The device includes an array of twenty-four specifically designed actuators that stimulate the left hand while supported in a comfortable resting position. The elicited tactile sensations represent the building blocks of the Deafblind TFS Alphabet but can also be configured via software to produce other methods of tactile communication that use the volar surface of the hand such as the Lorm and Malossi alphabets.

The following sections can be summarized as follows: Sec. II reviews existing communication devices, Sec. III includes a summary of the device requirements, Sec. IV describes the tactile interface design and its limitation, Sec. V reviews the overall device and its implementation, and Sec. VI describes an initial evaluation with human participants.

## II. RELATED WORK

Communication devices for the Deafblind community can be categorized into two groups, Braille based devices and those focused on natural language reproduction. The development of devices based on natural tactile communication methods appeared in 1977 with the publication of a robotic fingerspelling hand by the Southwest Research Institute in San Antonio [6].

Between 1985 and 1993, the Department of Mechanical Engineering at Stanford University developed Dexter, a robotic fingerspelling hand designed to reproduce fingerspelling in American Sign Language (ASL) [7]. These devices focused on the visual aspect of sign language and were unsuccessful in generating sufficient interest within the Deafblind community. Reasons include cost, complexity and inadequate dexterity compared to human fingerspelling. ASL is grammatically different from English with fingerspelling only playing a very

Manuscript received: Month day, YEAR; Accepted Month day, YEAR.

This paper was recommended for publication by Editor NAME upon evaluation of the Associate Editor and Reviewers' comments. This work was supported by Google's Computer Science Education Research (CS-ER).

¹ BD, IL, and VH are with Sorbonne Université, Institut des Systèmes Intelligents et de Robotique (ISIR), 4 Place Jussieu, 75252 Paris, France.

² MZ, ZK and SX are with the Dept. of Information Design and Corporate Communication at Bentley University, Waltham, MA, USA.

³ ST is with Univ. of Sydney, Sydney, Australia

⁴ JM is Univ. of New England, NSW Australia

⁵ ZK, and AA are with Dept. of Robotics and Mechatronics, School of Engineering and Digital Sciences, Nazarbayev University, Nur-Sultan, Kazakhstan.

⁶ BD is with Center of Social and Affective Neuroscience, Linköping University, Sweden.

⁷ IL is with Université de Rennes, Inria, INSA, IRISA, CNRS, Rennes, France.

⁸ VH is with Actronika SAS, 68 Boulevard de Courcelles, 75017 Paris, France.

Digital Object Identifier (DOI): see top of this page.

contact: duvernoy@liu.se, mziat@bentley.edu
minor role in the language. This leads to a reduction in
recognition of spelled words visual sign language relies on far
more features such as location, direction, and facial expression
than a disembodied robotic hand can provide.

In 1993, Eberhardt et al. described a device based on the Tadoma method [8]. This device replicated the human vocal tract and was controlled by ten servo motors. Users placed their hands on the device to feel the effect of speech through movement and vibration. However, Tadoma has an incredibly steep learning curve and is not commonly used.

More recent works, such as the DB-HAND [9] and LORM Glove [10], focused on vibrotactile actuation to reproduce the sensation of the Malossi and Lorm methods, respectively. These devices take the form of a glove containing a set of actuators in continual contact with the dorsal or volar region of the hand. The DB-HAND device was the first vibrotactile system aimed at reproducing the Malossi alphabet, predominantly used in Italy. Developed in 2008 by Caporusso et al. [9], this bi-directional communication device comprised sixteen pairs of pressure sensors and Eccentric Rotating Mass (ERM) actuators embedded in a glove less than 10 mm thick and connected to a box housing a microcontroller and a battery. In 2012, Gollner et al. [10] proposed a similar device, the LORM Glove, using the Lorm alphabet, which is primarily used in German-speaking countries. Given the importance of working with the target community, the system was developed in collaboration with Deafblind organizations. This device was also a bi-directional system, embedding twenty-five pressure sensors and thirty-two ERMs into a fabric glove. The sensors were placed on the volar region of the hand, with the actuators placed on the dorsal region. This configuration enabled users to engage with objects while wearing the glove.

Several tactile communication technologies for accessibil-
ity based on Finger-Braille have been developed offering a
straightforward reproduction of Finger-Braille symbols, low
complexity, and compact design. These types of communi-
cation are outside the scope of the present article; for more
information readers can refer to [11], [12], [13].

III. REQUIREMENTS

In this section, the requirements evaluated as essential
for developing a Deafblind communication device are listed:
human factors, hardware constraints, and flexibility.

A. Human Factors

It is essential to convey tactile information in the clearest
and most natural manner. Thus, the most crucial requirement
is the perceptual quality of the signals [14]. To achieve
this, the device must generate sensations resembling hand-to-
hand contact during TFS. Tactile output from the HaptiComm
should be adjustable to the user's preferred speed and intensity.
The device must also minimize delays that could jeopardize
interactivity.

The design philosophy stems from the need to create a
device capable of supporting prolonged use without impeding
freedom of movement, which is a restriction of glove-based
configurations. In practice, the user's hand must remain free

to complete competing tasks that often call for immediate
attention. Slight postural readjustments of the hand must also
be allowed to support prolonged use. Hence, the hand should
not be constrained with straps, cradles, or similar restrictions;
but should be placed at rest in an ergonomic position. Its
design should induce minimal postural stress; to avoid injuries
or discomfort during prolonged use. Sustained skin contact
with impermeable surfaces imprisons sweat and does not allow
the skin to cool down. Therefore, the design must allow for
air circulation between points of contact.

B. Hardware Constraints

The design of the HaptiComm should be straightforward
and could ideally be constructed by non-experts with basic
engineering skills. Components should be simple, inexpensive,
and widely available. The manufacturing processes should be
well-documented for easy reproduction when components are
not available for purchase. Preferably, the device should be
light, portable, and robust, with materials and components
integrated into one single unit. Such a device is intended to
be used intensively. Accordingly, it must be safe (i.e., rounded
edges, low voltage), as well as easy to clean and maintain.

C. Flexibility

The letters of the Australian Deafblind TFS Alphabet may
vary over time and/or individual preferences. While this paper
focuses primarily on alphabetical symbols, it is noted that
"short-cut" tactile signs may be used to represent whole
words, phrases, or concepts. To incorporate new linguistic
features or update existing ones, the software design must be
flexible. Furthermore, the mechanical interface must account
for different hand sizes without compromising portability.

IV. DESIGN AND TACTILE CAPACITIES

Deafblind TFS is a communication system that haptically
represents letters, words, ideas, and concepts. The perception
of this system is predominantly passive, with signs placed
onto the volar surface of the hand. Sign generation is defined
by signer's hand shape, location on the receiver's hand,
orientation, motion, speed, direction and intensity of sustained
contact and other expressive features such as duration, rhythm,
pressure, and vibration. Evaluation of previous work (see
Section II) suggests development of a device from the visual
perspective of a signer would be prohibitive on the grounds
of cost and mechanical complexity. Instead, we focus only on
tactile aspects of Deafblind TFS that occur at the point of
contact between the signer's and receiver's hand. Focusing
solely on the properties of skin deformation allows us to
extract five key features that in combination generates the
tactile capacity for the creation of unique signs.

A. Key Features of the Deafblind TFS

This section describes the key features of the Deafblind TFS
alphabet resulting from observation, physical demonstration,
and discussion with interpreters and peers. These features
include contact size, shape, duration, location, and force.
<figure><img src="image_2.png" /><figcaption>Fig. 1: Letters of the Australian Deafblind Manual Alphabet.</figcaption></figure>

* **Contact Area Size** is defined by the physical contact created on the signer's hand. Contact area size varies between approximately 0.5 cm² and 150 cm². Evaluation identified four key categories of size: Single fingertip (a < 1 cm²); Small (1 cm² < a < 5 cm²); Medium (5 cm² < a < 15 cm²); and Large (a > 15 cm²).

* **Contact Shape** is defined by the overall shape of the contact being made. Examples may include round, rectangle, or a hook.

* **Contact Location** is the point(s) at which the hand is stimulated. Examples may include fingertips, a palm, wrist, or the entire hand.

* **Contact Duration** is the amount of time that the contact occurs measured in milliseconds. Duration occurs between 10 ms and 500 ms (10 ms < t < 500 ms).

* **Contact Force** is the amount of force exerted on the hand measured in Newtons. Force is usually constant with minor variation and is heavily influenced by personal preference.

## B. Contact Types of the Deafblind TFS Alphabet

As Deafblind TFS is growing and evolving over time, an exhaustive evaluation of all Deafblind TFS signs is prohibitive. In an attempt to provide meaningful evaluation criteria for the key haptic properties this study is restricted to the Deafblind TFS Alphabet (see Fig. 1). The alphabet comprises twenty-six signs representing the letters of the English alphabet and principally uses locations on, or in conjunction with, the volar region of the hand. Size is usually equivalent to a single fingertip while the remainder are evenly distributed between the size categories aforestated. Additionally, five contact types were identified: *Tap*, *Tap-and-Hold*, *Slide* (motion), *Squeeze*, and *Pinch*.

* **Tap** is defined as occurring at one or more discrete locations and with a duration of less than 10 ms.

* **Tap-and-Hold** may occur at one or more discrete locations. Duration is extended from that of the tap and is typically 75 ms.

* **Slide** is defined by a start location and one or more successive locations that the contact moves to while retaining contact force, shape, and size. Duration is dependent on the number of locations the contact must pass through and the speed of the movement.

* **Squeeze** has a constant location and an extended duration of 100 ms. A squeeze comes into initial contact with two or more sides of the hand and gently increases the amount of force being exerted.

* **Pinch** has the same properties as a squeeze except that force must be applied on only opposing sides of the location.

## C. Tactile Interface Design

The design was guided by the aforementioned requirements, including the desire to recreate natural tactile sensations described by the previously cited key features of Deafblind TFS. The result is a tactile interface consisting of an array of twenty-four actuators specifically designed for tactile communication.

1) **Actuator Design:** The actuators adopt the simplest configuration of a linear, radial tubular magnet machine [15]. They enable two modes of operation: (i) when activated with a current pulse, the actuator produces an impact on the skin which is similar to the impact energy made by a human finger; (ii) when activated with an oscillatory current combined with DC offset, the end of the actuator gently presses on the skin and vibrates within a broad range of frequencies [16]. Individual actuator tips are covered by a 10 mm hemispherical elastomeric dome made of EcoFlex™ 00-30 to approximate the shape and viscoelasticity of a human fingertip. For further technical details, readers can refer to [16].

2) **Actuators Distribution:** The twenty-four actuators have been placed strategically under the hand. Fig. 2a shows the actuator distribution. This arrangement comprises an array of 3-by-3 actuators located in the middle of the palm, and one actuator for each phalanx except for the proximal and intermediate phalanges of the little finger where one actuator was located between them. This exception was made as the letter *S* could be optimally elicited without increasing the overall device complexity. Similarly, one actuator was placed close to the wrist (*J*) and one near the ulnar side of the hand (*T*). Due to Mechanical complexity, only the volar region of the hand is stimulated.

## D. Performances and Limitations

1) **Signs Reproduction:** By the actuator design and distribution, it is possible to elicit three of the five contact
<figure><img src="image_3.png" /><figcaption>Fig. 2: Definition of the array of actuators and its function: a. Locations of the twenty-four actuators regarding the receiver's hand; b. Description of the actuator behavior and actuators orchestration regarding contact types and contact sizes; c. Simplified configuration of six combinations resulting from three contact types and two contact sizes.</figcaption></figure>

types. Additionally, sizes ranging from a fingertip to a whole hand are feasible. While different shapes are possible, they are composed of discrete points and not continuous contact. The logic of these combinations is shown in Fig. 2b and c. The contact types are elicited as follows. *Tap* sensations are produced by activating the actuators with current pulses. The actuator rotor starts from a resting position where it is lodged inside the actuator bore and rises to a stable position, 5 mm above the recessed position. When a hand rests on the device, an impact occurs producing a *Tap* sensation of an intensity modulated by the amplitude of the current pulse. If the current pulse has extended duration, a *Tap-and-Hold* sensation is produced. When the end-effectors are recessed they do not stimulate the skin, enhancing contrast. To achieve Slide sensations, the device takes advantage of the actuator's vibration mode to create tactile apparent motion effect [17], wherein the brain perceives motion despite activation only occurring in successively activated discrete loci. To increase the size and modulate the shape of the contact, it was found that simultaneously activated neighboring actuators separated by 10 mm cannot be felt individually. Simultaneous activation of multiple, contiguous actuators creates larger virtual contact areas. This effect is owed to the fact that static two-points discrimination thresholds smaller than 10 mm do not apply to dynamic contacts.

2) *Validation of the Other Requirements:* This approach keeps the complexity low, ensuring easy manufacturing and critical tolerances around a millimeter, robustness, and repeatability of the tactile stimulation. The delay between two consecutive signs at two extreme locations on the receiver's hand is minimal compared with a robotic solution. Moreover, the device can easily maintain the pace of a normal conversation and can even deliver signs as quickly as the user requests. The current intensity provided to the actuators can also be controlled, allowing the user to modulate the amplitude of the received force. The actuators are embedded in a rigid structure equal to the size of a hand. The device must be placed on a horizontal surface to function, for instance, a table, which is the main difference compared to glove-based approaches. If the user wants tactile information, they decide

when to place their hand on top of the device. Thus, the hand is not constrained, and the user is free to complete competing tasks. The shape of the user interface follows the curvature of a hand at rest to minimize stress during prolonged use and facilitate minor postural adjustments. Surfaces in contact with the receiver's skin are minimized to facilitate air circulation, reducing the accumulation of heat and sweat.

3) *Signs Reproduction Limitations and Gains:* Some design decisions were made based on our requirements, but they led to a loss in spatial resolution density and the ability to stimulate non-volar surfaces of the hand. In most cases, the limitation impact is minimal and does not affect the legibility of the device. For example, *W* retains its location which is unique across the fingers and is not impacted by the inability to stimulate the non-volar regions. In cases where understandability might be impacted such as identifying the letters *G* and *B* which are reduced to a contact area, we propose harnessing unique features offered by the device to enhance recognizability. These may include creating larger areas or modifying the type of contact by taking advantage of the actuator's vibratory function.

## V. SYSTEM

HaptiComm comprises the user interface (receiver's side), a custom-made electronics driver, and software translating text or voice to tactile output (interpreter's side). Table I shows a detailed cost analysis of the HaptiComm. The overall system weighs 2.5 kg and can be carried in a 20 × 25 × 30 cm³ box. The following section highlights the main features of the system.

### A. User Interface

1) *Hand rest:* This hand rest has been empirically modelled and was designed to tolerate minor hand readjustments for comfort purposes during prolonged use. Most critically, is the ability for the users to quickly release their hand from the device if needed, see Fig. 3a.
<table><caption>TABLE I: HaptiComm Cost-Analysis</caption><thead><tr><th>Item</th><th>Details</th><th>Cost</th><th>Qty</th><th>Total</th></tr></thead><tbody><tr><td>Actuators</td><td>Magnets, coils, and reels</td><td>$10.02</td><td>24</td><td>$240.50</td></tr><tr><td>Enclosure</td><td>3D printed parts and instrument cases</td><td>$538.00</td><td>1</td><td>$538.00</td></tr><tr><td>Silicon rubber</td><td>To make silicon tips</td><td>$75.00</td><td>1</td><td>$75.00</td></tr><tr><td>Nuts and bolts</td><td>Fasten and secure the system</td><td>$3.00</td><td>68</td><td>$204.00</td></tr><tr><td>Motor driving PCB</td><td>Customized PCB to control 24 actuators</td><td>$800.00</td><td>1</td><td>$800.00</td></tr><tr><td>Rasberry Pi</td><td>To control the system</td><td>$150.00</td><td>1</td><td>$150.00</td></tr><tr><td>5V power supply</td><td>Power the PCB</td><td>$100.00</td><td>1</td><td>$100.00</td></tr><tr><td>Cables and connectors</td><td>Connect system parts together</td><td>$73.00</td><td>1</td><td>$73.00</td></tr><tr><td>Shipping</td><td>Cost to ship all parts</td><td>$50.00</td><td>4</td><td>$200.00</td></tr></tbody><tfoot><tr><td colspan="3">Total cost</td><td></td><td>$2,380.50</td></tr></tfoot></table>

<figure><img src="image_4.png" /><figcaption>Fig. 3: User interface: **a.** Hand rest shape; **b.** Actuators' planar location (X and Y axes, top view); **c.** Actuators' vertical locations optimization: initial locations and constraints due to the magnetic interference based on the shape of the hand rest; **d.** user interface architecture; **e.** Device overview (high-angle view): actuators support, actuators guide, and hand rest; **f.** Overall process of transforming voice or text into actuators motion.</figcaption></figure>

2) *Distributing the actuators in the horizontal plane:* The aim was to find an optimal distribution of the actuators that could accommodate an average-sized hand. To this end, twenty-two adult hands were photographed in a rest position using a camera placed above the handrest. Because the camera was relatively close to the hand, each picture was processed to remove this distortion. That is, a sheet of paper was placed beneath the device (within the frame of the picture), and a homography transformation was applied based on the four corners of the page, thus normalizing the pictures. For each hand picture, actuator position was defined according to the organization displayed in Fig. 2a. The green crosses represent individual actuator locations in Fig. 3b. The twenty-four final locations were defined by the barycenter of the twenty-two locations of each actuator (illustrated by red dots in Fig. 3b).

3) *Distributing the actuators vertically:* The moving permanent magnets of two closely located actuators interfere, disrupting the expected behavior of both actuators. Shielding each actuator individually can be complex and increases cost. To cancel this magnetic interference, we chose to separate the actuators from each other spatially. By design, the actuators' horizontal location is fixed (see the previous section). As the remaining degree of freedom is the vertical placement of the actuators, some actuators' bodies were further away from the hand to increase their mutual distances globally. The minimal vertical distance between two actuators has been empirically estimated to be 27 mm. Since there is also a desire to minimize the total height of the device, the resulting optimization problem takes the form of a constrained Minimax optimization problem with a minimum acceptable distance set at 27 mm.
As the hand support is not flat, some actuators are more predisposed to increase the total height of the device than others. This has been considered in the optimization by initializing the vertical position of the actuators as close as possible to the hand support, see Fig. 3c. Using a solver based on [18], the initial vertical placements of the actuators were modified to satisfy the minimum distance. The optimization resulted in a maximum actuator displacement of 41.2 mm. The optimization also increased the total height of the device by the same amount. The actuators' body housing can also be seen in Fig. 3e.

## B. Signal-to-Noise Ratio

In order to minimize mechanical noise, thereby improving message clarity, three aspects were considered: consistent and uniform behavior for all actuators, minimal constant tactile noise, and minimal perceived actuators' noise. To address these considerations, three features have been added to the design: 1) the inclusion of an actuator guide in the system, 2) the reduction of the unnecessary sustained contact between the user interface and the hand, and 3) the division of the user interface into two main sub-assemblies.

1) *Actuator Guidance:* As in section V-A3, several actuator bodies were situated several centimeters below the hand to cancel the magnetic interference between close actuators. To keep the intended skin contacts, the rods connecting the magnets and the tip of the rotor were extended up to 41.2 mm. Without rod guidance, the HaptiComm was susceptible to shaking and vibrations from the deployment of actuators rising from their resting position inside the actuator bore. The guidance system consists of two discs of PLA, located as close as possible to both rod ends, see Fig. 3d (in blue) and 3e (blue mesh, middle picture).

2) *Reduction of the Permanent Skin Contact:* One issue related to mediated tactile communication compared to human touch is that it introduces unavoidable sustained contact with the skin during operation. To address this issue, the hand rest was designed as a mesh allowing airflow under the hand, thereby minimizing the accumulation of sweat and heat. The mesh also helped reduce the sensation of constant contact, which is especially important during prolonged use.

3) *Minimization of Mechanical Crosstalk:* The actuators generate mechanical noise when activated due to frictional contacts in the guidance mechanisms. This noise was within the tactile perceptual range and could pollute the intended signal. If not dampened, mechanical noise propagated in the rigid parts of the HaptiComm and affected the clarity of the messages sent by the device. To reduce mechanical crosstalk, the device was organized into two distinct sub-assemblies connected through elastomeric viscoelastic elements, see Fig. 3 and 3e. The first sub-assembly comprised the actuators, and the second served as the handrest. The noise was recorded while displaying the letters A and H to evaluate the effectiveness. The vibrations recorded on the hand rest were lower than 2% of the intensity recorded on the actuator sub-assembly (data recorded with a 3-axis accelerometer, ADXL335b).

## C. Drive Electronics and Software

A custom-made analog electronics drive was developed to control each actuator with a maximum signal of 4.5 V full swing from a 5 V power rail, see Fig. 3f. A separate DAC channel drove each power amplifier by steps of 2 mV. The DAC (AD5383) featured 32 outputs and was driven at 30 MHz via an SPI bus connected to a microcomputer (Raspberry Pi 3), ensuring a refresh rate of at least 2 kHz per channel. The microcontroller ran software that received inputs from a keyboard and converted them to DAC commands. The software used a configuration file (audio waveform stored in wav. files), which made it possible to easily design new tactile symbols and link them to specific letters. The software, written in C++, was double threaded, separating the high priority, 30 MHz DAC driver from the keyboard-to-DAC-commands conversion. The combination of the drive electronics and software allows control of both speed and intensity of the signal to match user expectations.

# VI. EXPERIMENTAL EVALUATION

Transitioning from human fingerspelling to mediated fingerspelling is not trivial. Machine actuation has no hedonic value (temperature, human skin, skin contact area, etc.). Since the HaptiComm has not undergone previous evaluation, the present study strives to ascertain its performance as described below. The experiment was designed to determine the recognition rate related to the types of contact on the volar region of the hand: *Tap*, *Tap-and-Hold*, and *Slide*. *Pinch* and *Squeeze* were not tested because they involved more than the volar region. Adjustments were made to some letters as explained in section V. For example; it is easy to distinguish a human finger *Tap* (letter *E*) from a human finger *Pinch* (letter *P*) on the index finger. However, the same task using a mediated device requires distinguished features to differentiate the letters (e.g., *Tap* for *E* and *Tap-and-Hold* for *P* on the same finger area with the same actuator). We also assessed the size of the virtual contact area determined by the number of activated actuators. The goal was to identify the perceptual factors that minimized the error rates and serve as a baseline for future device use.

## A. Participants

Ten adults (six F, mean age: 28.80, SD: 7.21) participated in this experiment. All participants were recruited from Bentley University and received an Amazon gift card for their involvement. The experiment conformed with the Declaration of Helsinki on the use of human subjects in research, and written informed consent was obtained from all participants. The experimental procedures were approved by the Institutional Review Board of Bentley University.

## B. Tactile Stimuli

The stimuli consisted of the three contact types: *Tap*, *Tap-and-Hold*, and *Slide*. These were combined with the number of activated actuators ranging between two and four (2-4), five and eight (5-8), and nine and twelve (9-12), corresponding to the size of the three contact areas Small, Medium, Large
<figure><img src="image_5.png" /><figcaption>Fig. 4: Percentage of correct responses: a. contact type; b. range of actuators. Error bars represent 95% confidence intervals.</figcaption></figure>

described in section IV-A. Fig. 2b illustrates the contact area expanding as a function of increased number of actuators. The combination of contact TYPE × actuator NUMBER resulted in nine different conditions with the independent variables TYPE and NUMBER.

### C. Procedure

After signing the consent form, participants put their left hand on the HaptiComm and used their right hand to press the space bar and select the question options. In a training phase, participants were exposed to the three contact types, where feedback was provided on their answers. A total of nine random trials were included in this phase. After the training phase, participants proceeded directly to the testing phase. This phase included one block of fifty-four trials, where the nine conditions were repeated six times in a randomized order. Each trial started with the participant pressing the space bar on the keyboard to trigger the tactile stimuli. After the trial, they were asked to answer two questions on their screen: (1) which type of stimulus was presented, in which participants were presented with three choices: *Tap*, *Tap-and-Hold*, and *Slide*, and (2) how many actuators were activated with three possible answers: 2 to 4, 5 to 8, or 9 to 12. A new trial was initiated only after the participant's response was recorded; thus, the experiment proceeded at a pace determined by the participant. On average, the experiment lasted between 15 and 20 minutes.

### D. Results

A two-way repeated-measures ANOVA was performed to analyze the effect of stimulus type (TYPE) and actuator number (NUMBER) on participants' correct responses to both questions.

1) Contact Types: For their probability of answering correctly about the stimulus type, the assumption of sphericity has not been violated (p > .05). The ANOVA revealed that there was a statistically significant effect of the variables TYPE [F(2, 18) = 15.47, p < .001] and NUMBER [F(2, 18) = 49.97, p < .001]. There was also a significant interaction between the effects TYPE and NUMBER [F(4, 36) = 17.97, p < .001]. To breakdown this interaction, pairwise comparisons of the main factors were performed. The post-hoc analysis revealed significant effects between 2-4 and 5-8, 2-4 and 9-12, 5-8

and 9-12 for the *Slide* condition (p values < .05). It also revealed a significant effect between 2-4 and both 5-8 and 9-12 numbers for the *Tap-and-Hold* condition. Fig. 4a shows participants' responses for each of the three stimulus types where performances for 2-4 and 5-8 were significantly lower than 9-12 for the *Slide* condition. Additionally, 2-4 obtained the lowest correct response for this same condition. Correct responses for the 2-4 condition were also significantly lower than the other range numbers when associated with *Tap-and-Hold*.

2) Range of Activated Actuators: In regard to the probability of producing a correct response to the number of activated actuators, Mauchly's test indicated that the assumption of sphericity had been violated (p > .05). The ANOVA revealed that there was a statistically significant effect of the variables TYPE [F(2, 18) = 6.56, p < .001] and NUMBER [F(2, 18) = 23.65, p < .001]. There was also a significant interaction between the two main factors TYPE and NUMBER [F(4, 36) = 6.30, p < .001]. Simple effect tests were conducted on each subset of data to breakdown the interaction. Comparison of the response for actuator range indicated a significant difference between *Tap* and *Slide* when the number of actuators was between 5 and 8. There was also a significant difference between *Slide* and the two other types for a range of actuators between 9 and 12. Fig. 4b shows participants' responses for each of the three ranges. Participants were less accurate with the *Tap* condition when associated with the 5-8 range. Similarly, *Slide* obtained the highest response accuracy when associated with the 9-12 range compared to *Tap* and *Tap-and-Hold*.

## VII. CONCLUSION

In this paper, we described the HaptiComm, a tactile communication device for individuals with dual sensory impairments. It was designed to reproduce tactile fingerspelling techniques that focus on stimulating the volar surface of the hand. While configured for the Australian, New Zealand and British Deafblind Fingerspelling Alphabet, its usage can be expanded to other symbols that can be created and personalized by the user. The device comprises an array of twenty-four actuators specifically designed for this purpose. The focus concentrated on improving the clarity of the tactile signals received by users, realistic tactile stimuli, and minimization
of unwanted mechanical noise. The device was composed of
a user interface, drive electronics and a microcomputer. It is
compact, lightweight, and mainly constructed from off-the-
shelf components. The design does not constrain the hand
of the user, leaving them free to complete competing tasks
that often call for immediate attention while receiving tactile
input. The hand support shape was ergonomically designed to
reduce muscular stress, discomfort, and potential injury from
prolonged use while allowing for adequate airflow to curtail
the accumulation of superfluous heat and sweat.

To ascertain the functionality of the device actuation, we
ran an evaluation with ten participants. Results showed that
participants could feel the differences between the types
of stimulation and the number of activated actuators. More
specifically, participants were more accurate in their responses
about the contact type when the number of actuators was
greater than five. However, the difficulty was evident for the
*Tap-and-Hold* and *Slide* conditions when associated with the
number of activated actuators between two and four. There was
less ambiguity for the *Tap* condition within the same range.
We consider the timing of *Tap-and-Hold* and *Slide* warrants
further investigation to distinguish them from each other while
preserving the favorable properties of tactile fingerspelling.
Additional testing related to multiple combinations of time
and speed are required, especially for small virtual contact
areas.

Participants were also less precise in finding the number
of activated actuators for higher ranges, specifically for the
*Tap* sensation when the number of actuators was higher than
five. Interestingly, participants were more accurate about *Tap*
when the range was small (2-4), suggesting a possible tradeoff
by exploring different timing conditions for *Tap* when the
virtual contact area is either small or large. More importantly,
it shows that participants were not counting the actuators but
were estimating the size of virtual contact. Numerosity studies
show a similar result, with a lower success rate associated with
larger numbers [19], [20], [21], [22]. Since we did not ask
participants to count the actuators but provide a range, these
results were expected.

We are planning to run additional studies by refining the
timing and speed parameters of actuation. We will endeavor
to estimate the device’s individual and sequenced letter recog-
nition rate compared to human fingerspelling and apply these
patterns in the context of communicating words, phrases, and
sentences to convey concepts and information. Quantifying
the learning curve is an essential element in the adoption
of assistive technology. This future step will further inform
and endorse the viability of the HaptiComm as a conduit for
naturalistic Deafblind communication.

ACKNOWLEDGEMENTS

We thank MES AP09058050 for their support and the
SMART project of Sorbonne Université. We are thankful
to Sarah McIntyre, Florian Richer, William Frier, Thomas
Daunizeau, Cyril Bounakoff, Mia Cirrincione, and Nurlan
Kabdyshev for their help during this project.

REFERENCES

[1] F. Ask Larsen and S. Damen, "Definitions of deafblindness and congenital deafblindness," *Research in Developmental Disabilities*, vol. 35, no. 10, pp. 2568-2576, 2014.

[2] W. F. of DeafBlind, "Global Report on the Situation of Persons with Deafblindness," 2018.

[3] J. Dammeyer, "Deafblindness: A review of the literature," *Scandinavian Journal of Public Health*, vol. 42, no. 7, pp. 554-562, 2014.

[4] S. Company, "Schoenherr Braillotron TI-2550 II," 1975. [Online]. Available: www.datamath.org/Related/Schoenherr/Braillotron.htm

[5] N. Southern and L. Drescher, "Technology and the needs of deafblind people," *International Congress Series*, vol. 1282, pp. 997-1001, 2005.

[6] C. J. Laenger and H. H. Peel, *Further Development and Test of an Artificial Hand for Communication with Deaf-blind People*. US Dep. of Health, Education and Welfare, Office of Education, 1978.

[7] D. L. Jaffe, "Evolution of mechanical fingerspelling hands for people who are deaf-blind," *J. of Rehab. Research and Development*, vol. 31, no. 3, pp. 236-244, 1994.

[8] S. P. Eberhardt, L. E. Bernstein, D. C. Coulter, and L. A. Hunckler, "OMAR a haptic display for speech perception by deaf and deaf-blind individuals," *IEEE Virt. Real. Annual Intern. Symp.*, pp. 195-201, 1993.

[9] N. Caporusso, "A wearable malossi alphabet interface for deafblind people," in *Working conf. on Adv. visual interfaces*, 2008, pp. 445-448.

[10] U. Gollner, T. Bieling, and G. Joost, "Mobile lorm glove: introducing a communication device for deaf-blind people," in *Intern. conf. on tangible, embedded and embodied inter.*, 2012, pp. 127-130.

[11] H. Nicolau, K. Montague, T. Guerreiro, A. Rodrigues, and V. L. Hanson, "HoliBraille: Multipoint vibrotactile feedback on mobile devices," *W4A 2015 - 12th Web for All Conference*, pp. 10-13, 2015.

[12] O. Ozioko, P. Karipoth, M. Hersh, and R. Dahiya, "Wearable assistive tactile communication interface based on integrated touch sensors and actuators," *IEEE Trans. Neural Syst. Rehab. Eng.*, vol. 28, no. 6, 2020.

[13] L. Pantera, C. Hudin, and S. Panéels, "Lotusbraille: Localised multifinger feedback on a surface for reading braille letters," in *IEEE World Hapt. Conf.*, 2021, pp. 973-978.

[14] M. Ziat, *Haptics for Human-Computer Interaction. From the Skin to the Brain*. Foundations and Trends® in Human-Computer Interaction, Now Publishers, 2022.

[15] J. Wang, G. W. Jewell, and D. Howe, "A general framework for the analysis and design of tubular linear permanent magnet machines," *IEEE Transactions on Magnetics*, vol. 35, no. 3 PART 2, pp. 1986-2000, 1999.

[16] B. Duvernoy, I. Farkhatdinov, S. Topp, and V. Hayward, "Electromagnetic Actuator for Tactile Communication," in *Haptics: Science, Technology, and Applications*, 2018, pp. 1-10.

[17] A. Israr and I. Poupyrev, "Control space of apparent haptic motion," 2011 IEEE World Haptics Conference, WHC 2011, pp. 457-462, 2011.

[18] F. W. Gembicki, "Vector Optimization for Control with Performance and Parameter Sensitivity Indices," Ph.D. dissertation, 1974.

[19] G. Mandler and B. J. Shebo, "Subitizing: An analysis of its component processes," *J Exp Psychol Gen.*, vol. 111, no. 1, pp. 1-22, 1982.

[20] A. Gallace, H. Z. Tan, and C. Spence, "Can tactile stimuli be subi-
tised? An unresolved controversy within the literature on numerosity judgments," *Perception*, vol. 37, no. 5, pp. 782-800, 2008.

[21] Z. Z. Cohen, S. Naparstek, and A. Henik, "Tactile enumeration of small quantities using one hand," *Acta Psychol.*, vol. 150, pp. 26-34, 2014.

[22] B. Duvernoy, S. Topp, J. Milroy, and V. Hayward, "Numerosity Identifi-
cation Used to Assess Tactile Stimulation Methods for Communication,"
*IEEE Transactions on Haptics*, vol. XX, no. Xx, pp. 1-8, 2020.