# TEXTile: Eyes-Free Text Input on Smart Glasses Using Touch Enabled Textile on the Forearm

Ilyasse Belkacem, Isabelle Pecci, Benoît Martin, Anthony Faiola

► To cite this version:

Ilyasse Belkacem, Isabelle Pecci, Benoît Martin, Anthony Faiola. TEXTile: Eyes-Free Text Input on Smart Glasses Using Touch Enabled Textile on the Forearm. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.351-371, 10.1007/978-3-030-29384-0_22. hal-02544630

HAL Id: hal-02544630

https://inria.hal.science/hal-02544630

Submitted on 16 Apr 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# TEXTile: Eyes-Free Text Input on Smart Glasses Using Touch Enabled Textile on the Forearm

Ilyasse Belkacem¹, Isabelle Pecci¹, Benoît Martin¹ and Anthony Faiola²

¹ Laboratoire de Conception, Optimisation et Modélisation des Systèmes, LCOMS EA 7306,
Université de Lorraine, Metz, FRANCE

² Department of Biomedical and Health Information Sciences, University of Illinois, Chicago,
IL USA

{ilyasse.belkacem, isabelle.pecci, benoit.martin}@univ-lorraine.fr, faiola@uic.edu

**Abstract.** Smart glasses are autonomous and efficient computers that can perform complex tasks through mobile applications. This paper focuses on text input for mobile context. We present a new connected fabric to smart glasses as a device for text entry. This device, integrated into clothing, provides a new interaction technique called TEXTile. It allows typing without requiring users to hold a device. Users can put fingers anywhere on the fabric surface and release them, without needing to look at the fabric or using markers on the fabric. The text entry technique is based on eight combinations of fingers in contact or released, identified as pleasant among 15 in a survey involving 74 participants. A first user's study with 20 participants establishes that the eight combinations for TEXTile were significantly reliable (98.95% recognition rate). A second study with nine participants evaluates the learning curve of TEXTile. Users achieved a mean typing of 8.11 WPM at the end of ten 12-minute sessions, which can be slow, but sufficient with short text compared to other advantages of the technique. Results show low error rates for tasks completed and good usability (76% in SUS questionnaire). The NASA-TLX questionnaire establishes there is no important mental or physical workload to accomplish the task.

**Keywords:** Wearable, smart glasses, textile, forearm interaction, text entry.

## 1 Introduction

Today, the smartphone is the most widespread device in the world offering access to a broad range of apps, tools, and services. However, the mobile device has some practical limitations: it is often in the pocket or bags [1], and it does not allow quick access. In addition, during use, the user's hands are occupied, even if they are not interacting with the device, which makes its use frustrating in some context.

Smart glasses can overcome these limitations. This device looks like conventional eyeglasses including electronic components: a processor, sensors (Wifi, Bluetooth, GPS, accelerometer, gyroscope, etc.) and a transparent display screen that allows a
simultaneous vision of physical and digital worlds. Current smart glasses are autono-
mous and efficient computers that can perform more complex tasks for mobile applica-
tions. When using smart glasses, users may need to enter text: to search for information,
to reply to a message or to write a note. Common text entry used with smartphones or
computers may be complex to use with smart glasses.

The first commercial smart glasses, the Google Glass [2], provided speech recogni-
tion to enter text. However, this technique is inappropriate in noisy environments [3],
and users didn't like it in public places [4]. There is a need to design a new text entry
system for use in conjunction with smart glasses beyond what is currently available [5].

In this paper, we propose a new text entry technique called "TEXTile". This tech-
nique uses a new one-dimensional tactile fabric device that is placed on the forearm
while being connected to the smart glasses. We did an online survey to identify the
most comfortable combinations of fingers in contact with the fabric device. With these
combinations, we built a grammar for our new text entry technique. Two user studies
were conducted: one demonstrated the performance and reliability of the device, and
one experimented the performance of the new text entry technique. The contribution of
our work includes:

1. A smart textile for input with smart glasses based on a literature review and conceptual modeling.

2. A low cost and reliable prototype integrated into the fabric of one's clothing, e.g., on the forearm.

3. A new text entry technique using a grammar based on a user study that matches with the conceptual framework of our prototype.

## 2 Motivating Goal And Design Rational

In a mobile environment, the text entry task can be done standing, walking or sitting. The user cannot hold additional devices because his hands can be busy, and he cannot always use audio input due to respect privacy. Such a context would include for example a pedestrian that sends a message while walking in the street, a worker attending a meeting and taking short notes, or a clinician accessing patient records while interacting with the patient. To meet the needs of this context, we have set the following requirements for our design:

* (Req 1) The location of the interaction: interaction with the device should not hamper the user work. He should not carry an extra device in his hand or wear a glove. Wrist and forearm interactions are the most socially acceptable [6].

* (Req 2) The discretion of the device: the device must be invisible and comfortable during a mobility situation. It should not be cumbersome to affect the user work (e.g., a smartphone placed on the forearm).

* (Req 3) Eyes-free interaction: since the smart glasses screen is in front of the eyes, the user must be able to interact while remaining engaged to both real and digital worlds.
# 3 Related Works

With the emergence of mobile and wearable technologies, many research has been in-
terested in text entry because this task requires effective hardware or software solutions
to compensate for the absence of classic keyboards. With smart glasses, the great ad-
vantage is to be able to interact while remaining attentive to the real world, since the
smart glasses screen is in front of the eyes. This advantage becomes a challenge for
tasks like text entry. Usually, the user looks at the keyboard to enter text and looks at
the screen to check for text accuracy. With smart glasses, the user can look at the real-
world, while walking and interacting with the virtual environment. In this context, it is
ineffective and cumbersome to look at a text entry device.

Many classifications exist for text entry techniques. In this paper, we divide text entry techniques into two groups: those that require a single action and those that require multiple actions to select a character. We define an action as a gesture, a flick, a selection by pointing one or more touch simultaneously, etc.

## 3.1 Text entry techniques using a single action to select a character

The best known single action technique is the standard keyboard: one character maps
to one physical key. It can be physical or virtual. Small versions of Qwerty keyboard
for wearable exist, but they failed to gain acceptance due to their lower text performance
and their fatigue level [7]. To overcome space limitation, chorded keyboards were in-
troduced to decrease the number of keys. A chord is a combination of keys pressed
simultaneously. One of the most common is the one-handed Twiddler keyboard [8].
Other research has proposed several chording techniques based on Braille encodings:
BrailleTouch [9] and Perkinput [10]. These techniques are implemented on
smartphones. Although designed primarily for visually impaired people, they can be
used for text entry on smart glasses. The disadvantage of chorded keyboards is the effort
needed to learn chords.

Single-action techniques involve also gestures based input. Gestures can be per-
formed with a two-dimensional device. To ease learning, Graffiti [11] and EdgeWrite
[12] use gestures that are more or less related to the shape of the Latin letters. Graffiti
[11] supports quick learning and aid retention through the similarity of most characters
to the Romain alphabet, but it requires accuracy. EdgeWrite [12] needs less accuracy
but requires edges to perform the gestures and gestures are less related to the shapes of
the letters. With QuikWriting [13] and 8pen [14], users trace gestures around a central
region of the layout to one or more zones around the center to enter a character. Since
the layout of the characters determines the gestures, it is challenging to support text
entry in a mobile situation: the gestures require users to find locations of the character.
Gestures can be performed as well in the air. A sensor captures the gesture, and the user
can perform a gesture in the air associated with a character as in Airwriting [15] or
Airstroke [16]. These techniques have some human factors limitations because it causes
fatigue in the arms [17]. This technology requires the user to make gestures publicly in
front of the face, but users do not appreciate [18]. PalmType [26] is a technique that
uses the palm of the hand as a keyboard to enter text. A virtual keyboard is displayed
on the smart glasses, and the user touches his palm. Infrared sensors located on the forearm detect each touch. This technique requires the user to locate the specific region on the palm to enter the characters. As some smart glasses have a tactile surface, Yu and al. [27] proposed a system of gestures on this surface: each character is a set of one-dimensional lines that mimic the representation of the character with handwriting. This system does not support numbers and special characters.

## 3.2 Text entry techniques using multiple actions to select a character

Multiple actions to select a character allow reducing the space to interact and simplify the learning. Actions are sequential and they allow navigating in the space of characters. Users first select a group of characters and a character among this group. MessageEase [19] was primarily designed for 12-button keypad of mobile phones. It uses a tap for the first action followed by a tap or a slide-lift for the second action. With Flick-Key [20] and ThumbText [21], the keyboard layout is divided into six large keys. The user selects a group of characters by pointing a position on the surface and make a flick for the second action to select the character. All of these three techniques may imply accuracy issues because they require to locate where to tap for the first action. More than two actions can be performed in H4-writer [22], H4-TEG [23], MDITIM [24] to select a character depending on the frequency of the character. The Huffman coding using four keys represent the characters, but it requires learning.

Swipezone [28] was adapted from SwipeBoard technique [29] to smart glasses. By using this technique, each character is encoded with two actions: a first swipe that selects the region on the keyboard, and a second swipe that selects the character. This technique is very efficient and does not require a lot of learning because it uses the Qwerty layout. Holdboard [30] uses the same interaction technique, but unlike in Swipezone, it merges the selection of the region and the selection of the character in one gesture. The location of the thumb touching the edge of the smartwatch selects a group of characters (starting point of the gesture), and then, a swipe selects the desired character. That's why we consider it in the multiple actions category. This technique has also a problem with localization.

## 3.3 Devices used for text entry

Many devices can be associated with a text entry technique on smart glasses.

**Handheld devices.** In this category, we can find small versions of the Qwerty keyboard or Twiddler chorded keyboard with different buttons [8]. Another technology is a gyroscope-based handheld device [31], and it consists in entering the text by using a handheld controller to select the character on a keyboard with hand movement. The user must hold the controller in the hand and tilt it to different angles to select targets (characters) on the screen. Holding an extra device does not satisfy our requirement.
**Hand wearables.** DigitTap [32] is a wrist-worn device with an accelerometer to detect tap time and a camera to detect tap location. The acquisition of tap location cannot be accurately recognized in the occlusion environment or with wrist movement. Another work, DigitSpace [33], includes a pair of magnetic-sensing nail-ring chains mounted in the hand of the user. The prototype is very cumbersome in some mobile contexts. Other works use thumb-to-fingers interaction, but by wearing a glove [23,34,35,36]. The more recent and advanced work is DigiTouch[36], which senses a continuous touch position and detects the thumb position along the length of a finger by using conductive fabric strips and a conductive patch on the thumb pad. This technique uses one action by using the two hands. Though the technique is promising and can achieve high text entry speed, wearing a glove during the interaction may hamper the user in a mobile situation and may make his tasks difficult to master. Other wearable devices use eyes-free text entry, such as a ring [37], miniature touch surface [21] or even a nail-mounted device [38]. For example, asking the user to tap the finger that carries the ring [37] on a surface in different directions to select a character, has a variable performance that depends on the surface quality. Also, users of [21,38] may have difficulty because the interaction surface is limited to a small portion and can suffer from precision.

**Eye or head tracking.** With significant advances in eye-tracking performance, we can use eye movement for text entry [39] with smart glasses. In eye-tracking systems, people are usually asked to fix long enough on a character to select it. In a mobile context, the user could not keep visual attention to his environment. Moreover, interacting with the head using a gyroscope [40] requires learning and can be a source of fatigue if the text is long. It requires considerable visual attention while typing, along with constant head movement, which seems to be difficult in situations of needed mobility, making this technology also socially troublesome.

**Microphone.** Text entry using voice seems to be the most natural technique with a good performance outcome [25]. However, it raises issues of reliability and confidentiality in mobile situations.

**Interactive textile.** Many new inexpensive interactive textile materials are marketed. They can be made for invisible and ubiquitous interactions [41]. Several studies are interested in interactive textiles. One of the earliest is PocketTouch [42], that inserts a capacitive device grid in the pocket, but not integrated on the fabric of the pocket. Other works such as [43] and [44] have designed a resistive surface in the form of a matrix of pressure sensors, on the textile surface. Resistive technology does not detect multiple points in an efficient way. zPatch [45] combines resistive and capacitive technology, making a patch to detect hover, touch, and pressure on its interface. It does not yet allow multi-finger interactions.
# 4 The Text Entry Technique TEXTile

## 4.1 Description Of The Prototype

To meet the needs defined above (section 2), we chose to use an interactive textile that can be placed on clothing. We have designed a textile surface with a capacitive grid, which allows us the recognition of several simultaneous contact points, as outlined in detail below.

**Hardware Device.** Our prototype is a tactile fabric with a size of 11.9 cm x 10 cm. The prototype consists of 36 parallel conductive threads (thin, stainless, and washable), sewn in a linear array onto the fabric with a spacing of 3.5 mm. Three "12-cap-touch breakout MPR121" [46] cards are used to connect the threads. These cards allow us the capacitive sensing of touch on the tactile fabric. The cards manage the sensitivity. In addition, cards connect the 12 threads. All three cards are connected to a FLORA microcontroller (see Fig. 1).

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** The prototype has 36 parallel conductive threads sewn in a linear array connected to capacitive sensors.</figcaption></figure>

The microcontroller receives the touch events (contact or release) of each entry among the 36 entries. The device is connected to smart glasses with an OTG cable. A Bluetooth module can be integrated for a wireless solution if desired. The touch surface is framed by a red plastic rectangle so that the user can feel if their fingers are moving away from the touch surface.

To meet the first requirement (Req1), we explored the idea of attaching the device to the sleeve of a white lab coat, with snap buttons. Thus, when the user wears the coat, he can interact with the surface on the forearm.

**Processing Software.** The data transmitted by the MPR121 cards are processed at two levels. The first level, on the Arduino platform based on C/C++. This platform can detect touch events (contact or release) on the thread, and knows on which thread the event has been performed. After detection, the information is sent using the I2C serial bus communication protocol at the second level.
The second level is programmed in JAVA using the Android SDK because the smart glasses run under the Android system. This level interprets the state of the threads, in terms of fingers in contact or released. The combination of fingers is validated after a time without changing of states on the threads. The prototype can identify the numbers of fingers in contact with the textile surface by extracting the touch groups on the surface. Then, if the user releases fingers, the algorithm identifies which fingers are released by eliminating the touch groups from the start position. Once the operations are interpreted by the algorithm, in terms of interaction, specific actions on the system can be triggered.

**Prototype limitation.** Our prototype was simplified to perform 1D interactions anywhere on the surface in order to have reliability in fingers in contact and released detection. However, it has some limitations in the type of gestures that we can perform on the surface. The prototype does not identify which fingers are on the surface, that is why we need a start position to detect released fingers. Another limitation to the detriment of the simplicity of the prototype is that we cannot perform two-dimensional gestures as in many common techniques.

## 4.2 Preliminary Study

Our goal was to find a language of interaction based on simple contacts-and-releases of the four fingers (forefinger, middle finger, ring finger and little finger). The thumb was ignored because of its shifted position from the other fingers. We define a combination of 4 values $d_1d_2d_3d_4$ where the state of each finger $d_i$ is defined by :

$$d_i = \begin{cases} 0, & \text{if finger } i \text{ is released} \\ 1, & \text{if finger } i \text{ is in contact with the device} \end{cases}$$

$$\text{forefinger} = 1 \le i \le 4 = \text{little finger}$$

We started by conducting an online survey to identify comfortable combinations by using our prototype. The online survey was an effective tool for obtaining quick feedback on acceptability, as well as related to comfort with a gestural interface [47].

**Participants.** An email was sent requesting volunteers for the study. The email was dispatched on a university mailing list. Participants were encouraged to send it to others to participate in the study. We sent it to our families and friends too. 74 participants (37 females, 64 right-handed) answered to the online survey, 62% aged between 15-24, 18% between 25-34, 5% between 35-44, 4% between 45-54, 3% between 55-64 and 7% more than 65 years old. 69% were students, 19% workers, 5% retired persons, 3% unemployed, 3% housewife and 1% at college student.

**Task and Procedure.** The online survey examined the possible types of interaction while using our prototype: the RELEASE, i.e. release fingers “simultaneously” from a
start combination. We have chosen as the start combination the one where the four fingers (index, middle, ring and little) are in contact with the surface. The thumb was ignored because of its position being too offset from the others. We have 15 possible combinations of finger released. We have explored this “RELEASES” because they can (1) be performed in our 1D interaction surface, (2) be detected in an accurate way on our device, and (3) meet the eyes-free interaction without needing to look at the fabric or using markers on the fabric.

Before starting the survey, we inform the participants that the purpose of the study was to evaluate the comfort of these combinations to be used as part of an interface.

The survey illustrated the 15 combinations, with animated demonstrations in the form of a hand that performs the combination on the surface and with a brief description. In this way, the participants of the study could reproduce the combination. The videoed action took about two seconds, played in a loop. Participants were asked to reproduce and rate each combination, according to the comfort level they felt by using a five-point Likert scale—from “Not at all comfortable” to “Very comfortable”. All combinations are displayed in the same order on a single page with scrolling. The user can go back and correct a rate he gave.

Each combination was coded as follows: the finger in contact with the surface is represented by a 1 (or 0 if it is released). We presented the four fingers by a sequence of 4 digits where the high weight is associated with the index and the low weight with the little finger. For example, “RELEASE_0110” maps to “release the forefinger and the little finger from the start combination”.

**Results.** Figure 2 shows the average scores for the 15 combinations.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Mean score of each combination for the RELEASE operation.</figcaption></figure>

We only retained the comfortable combinations that had a score of more than 3.5 to eliminate the combinations with negative or neutral score—giving us 8 combinations:

* 3 combinations with a one released finger: 0111, 1011, 1110.

* 2 combinations with two released fingers: 0011 et 1100.
* 2 combinations with three released fingers: 0001, 1000.

* 1 combination with four released fingers: 0000.

From our analysis, we learned that comfortable combinations do not require the user to release non-adjacent fingers.

## 4.3 Evaluation Of Device Reliability

The purpose of the experiment is to evaluate the reliability of the device.

**Participants.** Twenty unpaid volunteer participants (average age 21.6 years, all right-handed) were recruited from the University's Computer Science PhD program. Participants never used smart glasses neither connected textile surfaces in the past.

**Task.** The goal is to evaluate the performance of the detection of the start combination (four fingers in contact with the textile surface) and the number of fingers released detected from the start combination. Participants had to interact with the textile surface in response to a visual stimulus displayed on the smart glasses. The stimulus (see Fig. 3) asked the user to put the four fingers in contact. Once the fingers are in contact with the surface, arrows indicates the fingers to release.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Screenshot of the task</figcaption></figure>

**Procedure.** Our study follows a single-variable plan: the operation (start combination, RELEASE) with an intra-subject configuration. The RELEASE operation had 8 possibilities (the selected combinations). Each combination was tested 10 times. Each participant performed 80 tests as follows: 0000 → 1111 → RELEASE. We collected data for 0000→1111 (8x10) and 1111→RELEASE (8x10). The task order was randomized with only one constraint: a RELEASE task always comes after a starting position task.

Before starting the tests, participants were asked to read a document that specifies the purpose of the experiment, the task to be performed, and the devices to use. They also fill out a pre-questionnaire for demographic information (age, gender, laterality). Participants began the test with a visual feedback on the smart glasses that gives them instructions.
**Apparatus.** The experiment was performed using Epson binocular transparent smart glasses. This model offers very good image quality with its OLED display at a 1280x720 pixels resolution. The screen of smart glasses maps to a screen of 80 inches by 5 meters. The smart glasses can be worn over eyeglasses. They embed various sensors (GPS, compass, gyroscope, accelerometer, etc.). They are connected to a controller through a cable. This controller contains buttons and a tactile surface used as a trackpad. The smart glasses run under an Android 5.1 operating system.

Our textile surface is connected to the glasses and placed on the left sleeve of a lab coat worn by the participant. The participant is standing, wearing the glasses and interacting with the textile surface with his right hand. The experiment took place in the same room with the same lighting conditions (closed window shutters and interior lights under the ceiling).

**Collected Data.** During the study, we recorded each interaction task with the textile surface (the expected interaction and the interaction detected), i.e.1600 (20 participants x 80 operations) of start combinations and 1600 operations release.

**Results.** Overall, our prototype successfully identified 99.19% of starting positions and 98.69% of RELEASE operations performed. Table 2 summarizes the detailed recognition rates.

The detailed error analysis revealed two causes of error. In some situations, the user's finger was not sufficiently in contact with the surface. In addition, the user sometimes hesitated and did not release the requested fingers simultaneously, which was interpreted as more than one action. For example, if it requested to release the middle finger, the ring finger and the little finger (RELEASE_1000), the user releases the middle finger and the ring finger and after a time longer than he/she releases the little finger, which considered as (RELEASE_1001). We distinguish device errors and participants errors by observation.

In addition, the reliability test showed that our textile surface can be used to put in contact four fingers (forefinger, middle, ring and little fingers) on the surface and release them. Hence, it could be used to test our text input technique.

<table><caption>**Table 1.** Recognition operations' rate by observation.</caption><thead><tr><th></th><th>Device errors</th><th>Participant errors</th><th>Reliability of the device</th></tr></thead><tbody><tr><td>Starting position</td><td>13/1600<br>0.81%</td><td>0/1600<br>0%</td><td>99.19%</td></tr><tr><td>Release</td><td>21/1600<br>1.31%</td><td>94/1600<br>5.87%</td><td>98.69%</td></tr></tbody></table>

## 4.4 Definition Of TEXTile

TEXTile is a text entry technique that encodes each character into a sequence of two actions (see Fig. 4). For each action, the user needs to put in contact the four fingers on the textile surface (called start combination) and to release between one and three fingers. If the user releases the four fingers, the text entry interaction is cancelled.
The advantage of this technique is that it does not require the visual attention of the user to look at the tactile surface which is being typed on. As the user has the glasses screen in front of their eyes, a view of the keyboard is displayed to help him to perform the right actions.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Two actions to perform the typing of a character.</figcaption></figure>

As the goal is to support novice users to enter a short text, we opted for a Qwerty layout as in Swipezone [28]. The keyboard is divided into two regions of three rows, which gives a distribution of characters on six rows. Each row is divided into two regions of three characters. This division allows us to reach all the characters in two actions as mentioned in Figure 4. To facilitate learning, we chose an intuitive method to select a row or a character:

* The number of the row to be selected maps to the number of fingers to release. If the row is located on the left region, the user needs to release its fingers by starting from forefinger. If the row is located on the right region, the user needs to release its fingers from little finger to forefinger.

* To choose the character in the row selected, it's the same technique as row selection. The user just needs to release the number of fingers matching to the position of the character on the region of the row. If the character is located on the left region of the row, the user needs to release its fingers by starting from forefinger. If the character is located on the right region, the user needs to release its fingers from little finger to forefinger.

* To select another character, the user just needs to do the start combination (put the four fingers in contact with the textile surface).

A visual help is displayed on the keyboard that user can see on smartglasses to help him which fingers to release (see Fig. 5): the numbers indicate how many fingers to release.
release and the arrows under the numbers indicate from which finger to start. For example, to select the second row of the right region, the user must release two fingers starting from the right: little and ring fingers (RELEASE_1100). Table 2 summarizes how to select each region from the start combination. Fig. 5 shows how to write the word “CAP” with visual feedback on the smart glasses.

<figure><img src="image_6.png" /><figcaption>Fig. 5. Example of text entry: the word “CAP”.</figcaption></figure>

<table><caption>**Table 2.** The combinations to select a row or a column (character) on the keyboard.</caption><thead><tr><th>Number of fingers to release</th><th>Combination</th><th>Result of action 1</th><th>Result of action 2</th></tr></thead><tbody><tr><td>0 finger</td><td>RELEASE_0000</td><td>No row selected</td><td>No column selected</td></tr><tr><td>1 finger</td><td>RELEASE_0111</td><td>Left row #1</td><td>Left column #1</td></tr><tr><td>1 finger</td><td>RELEASE_1110</td><td>Right row #1</td><td>Right column #1</td></tr><tr><td>2 fingers</td><td>RELEASE_0011</td><td>Left row 2 left</td><td>Left column #2</td></tr><tr><td>2 fingers</td><td>RELEASE_1100</td><td>Right row 2 right</td><td>Right column #2</td></tr><tr><td>3 fingers</td><td>RELEASE_0001</td><td>Left row 3 left</td><td>Left column #3</td></tr><tr><td>3 fingers</td><td>RELEASE_1000</td><td>Right row 3 right</td><td>Right column #3</td></tr></tbody></table>
Among the eight combinations selected during the preliminary study, RELEASE_1011 (release middle finger) combination remains available. This combination was used to switch to the numeric and special characters view of the keyboard. Initially, the QWERTY view was displayed.

# 5 Experiment Of TEXTile: Evaluation Of The Learning Curve

The purpose of this experiment was to evaluate novice performance and the learning curve of the TEXTile technique on smart glasses.

## 5.1 Participants

Nine unpaid volunteers (two female) participated in the experiment (average age 24 years, all right-handed) were recruited from the University's Computer Science program for a five-days study. Participants never used smart glasses neither connected textile surfaces in the past. In addition, they did not participate in any of the previous studies we conducted.

## 5.2 Task

Participants completed a text entry task with the TEXTile technique. The experiment was structured into 10 sessions that were split into five days, two sessions each day, to observe the learning curve and fatigue levels. Each session lasted 12 minutes. During this time the participants entered short text phrases as quickly and accurately as possible randomly selected from the MacKenzie and Soukoreff corpus [48].

## 5.3 Procedure

The experiment was a within-subjects design, with one independent variable, the session. The sessions (1-10) were used to check the learning effect and the novice performances. On the first session and before starting the tests, participants are asked to read a document specifying the purpose of the experiment, the tasks to be performed, and the devices to be used. The document explains also the text entry technique and how to perform typing and releasing to write letters based on visual aid on smart glasses. After that, they completed a training session where they had to enter the sentence "hello world" to get familiar with the smart glasses, the fabric and the technique. No prior training was performed because the goal of our evaluation is to observe how the performance varied over time. During this pre-test, if a participant had a problem, then they could ask the experimenter for help. Data from this session were discarded and this was done only one time, at the beginning of the first session.

Following the training session and after a short break, the test application started. Each participant typed sentences for 12 minutes. The timing started after the participants typed the first letter. A sentence was validated once typing the "enter" key. Error correction was allowed using the "back" key. It is up to the user to make as few mistakes
as possible while keeping a good speed of data entry. The test procedure for other sessions was the same as the first session, not including the training session. During the experiment, the experimenter took notes from the observations of the behavior of the participants.

At the end of the experiment (i.e., after session 10), the participants shared their comments on the experiment and the tested device. They also filled out the NASA-TLX evaluation, a subjective feedback form to assess perceived workload. They also complete a SUS questionnaire to measure the usability of the tested technique.

## 5.4 Apparatus

We conducted this experiment under the same conditions as the previous experiment, with the same equipment.

## 5.5 Collected Data

During this study, we recorded the expected sentence, each operation on the surface (Start position, RELEASE) with the time of the event. In addition, to the data collected during the interaction, we collected all questionnaire answers (SUS, NASA-TLX), the informal comments from the participants and the observations we noted during the experiment.

## 5.6 Results

**Text entry speed - the learning curve.** Fig. 6 shows the text entry speed overage over each session. The one-way repeated measures ANOVA showed a significant effect of the sessions ($F_{8,72} = 19,53, p < .001$). Tukey's HSD post hoc tests on sessions were carried out for text entry speed. Comparisons using the Tukey HSD test indicated that sessions six and nine, seven and ten were significantly different.

We derive the regression model in the form of the power law of learning. The equation and the squared correlation coefficients are shown in Fig. 6.

<figure><img src="image_7.png" /><figcaption>**Fig. 6.** Text entry speed by session.</figcaption></figure>
**Errors.** We distinguish two levels of errors:

* Corrected error: error that caused an entry of a wrong character with correction.

* Uncorrected error: error that caused an entry of a wrong character without correction.
Fig. 7 shows the average rate of the corrected errors over each session and Fig. 8 shows the average rate of the uncorrected errors by session.

The one-way repeated measures ANOVA showed that the difference between the ses-
sion is significant for corrected errors and for uncorrected errors.

Another type of error that was reported is soft error when the users make a row selection error (i.e., an error during the first action and the user cancels without causing an entry of a character, the passage S2→S0 and S3→S0 in Fig. 4). The soft error rate was 5.65% (SD 0.74) in the first session to 4.17% (SD 0.51) in the last session.

<figure><img src="image_8.png" /><figcaption>Fig. 7. Corrected text entry errors by session.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 8. Uncorrected text entry errors by session.</figcaption></figure>
**Participant feedback.** The results of the SUS questionnaires demonstrated an average of 76.5 (SD 6.89) for the TEXTile technique. The NASA-TLX questionnaires administered after the last session produced subjective ratings. Responses to these questions are summarized in Fig. 9.

When interviewing the participants about the TEXTile technique, they stated that: “at the beginning we look for how to reach a character, and it becomes a reflex action”, “it requires a little concentration at first, but we take over quickly”, “As we go along, the position of the letters are retained and automatically actions are made without looking at the keyboard” and “it takes a little time to get used with.”

Our primary findings from study 2 is related to the position of the arm carrying the textile. The participants who started the test with the arm in front of the eyes to write finished with their arm along the body.

<figure><img src="image_10.png" /><figcaption>Fig. 9. Average NASA-TLX score of TEXTile technique.</figcaption></figure>

# 6 Discussion

The results of our study show that although the TEXTile technique started with a relatively low speed (4.76 WPM; SD 0.58), the participants were still able to learn the technique as the study progressed. That is, they eventually achieved an entry speed of 8.11 WPM (SD 0.29) in the last session, after 2 hours of practice. These results are not surprising, because participants never practiced text input with the TEXTile technique. They were also not in a comfortable position because they were standing to simulate a standing mobility context.

Although our longitudinal study lasted 10 sessions, it did not last long, so the participants never became experts. It is hard to judge the performance of TEXTile technique after 2 hours of use but the results of our study show that users were able to type using TEXTile and meet the requirement of mobility context.

The error rate is relatively low (see Fig. 7). Increasing speed through sessions is not done at the expense of error. We also noticed a soft error rate from 5.65% (SD 0.74) in the first session to 4.10% (SD 0.51) in the last session. These errors can be explained
by the fact that the participants choose a combination and then cancelled. Part of the
errors was due to a combination that was not done correctly. For example, if it requested
to release the middle finger, the ring finger and the little finger (RELEASE_1000), then
the user releases the middle finger and the ring finger and after a time longer than the
validation time he/she releases the little finger, and thus, the combinations were per-
ceived as sequentially, which caused a misinterpretation of the combination and con-
sidered as (RELEASE_1001).

The results of the SUS questionnaire indicated a score of 76% for the TEXTile tech-
nique. This value can be considered as reaching a good level of satisfaction [49]. The
results of the NASA-TLX questionnaire were also very encouraging in the absence of
negative scores, while highlighting the mental and physical requirements necessary to
accomplish the task.

Although the TEXTile technique was new to the participants, they found it very in-
structive. Participants also noted that their learning of the TEXTile technique text entry
became quite eventually automatic, without the need to look at the keyboard. As our
goal was to provide eyes-free interaction with the device and not with the visual help
displayed on the smartglasses, another study will be necessary to measure when this
visual help is no more useful.

7 Conclusion And Future Work

We designed a new text entry technique referred to as TEXTile, in conjunction with the use of smart glasses. This technique uses a new connected fabric as a device that can be integrated into one's clothing. It provides typing without requiring users to hold a device. The user performs multi-touch of fingers anywhere on the fabric surface with no anchored targets for typing. The results of our user study demonstrated that the TEX-Tile technique had a good performance by reaching 8.11 WPM showing the feasibility of using TEXTile for text entry. After 2 hours of practice, the participants speed of input continued to grow throughout the study.

There are several limitations to the study that should be noted. First, the participants
of the study were limited with high degrees of experience with technology and compu-
ting. However, the research findings presented here were only intended to be explora-
tory; and as such, we used a convenience sample of users, i.e., students. Next steps will
include testing using a more general population, to assess the value of the technology
in other real-world settings. We hope to test TEXTile on a broader participant popula-
tion to assess usability and learning, in a fairer context. Another perspective is to con-
duct a blind experiment setting for experts where users do not have visual feedback to
limit the visual attention the user devotes to smart glasses visual feedback.

For our future work, we plan to make improvements to TEXTile, in order to address
several issues. The first would be to improve our prototype and to keep up with new
advances in digital fabrics technology. One possibility will be to have a two-dimen-
sional textile surface to compare our text entry technique to other techniques like Graf-
fiti [11] or EdgeWrite [12]. This improvement will allow us to use the textile surface
for other type of interaction, other than the text entry like pointing and manipulating interfaces in smart glasses.

We plan to permit users to edit their text (selection, copy, paste, etc.) and control the interface. We still have several possibilities with combinations of fingers that have not been explored, e.g., combinations with three fingers as starting position. Now, our technique uses two actions to enter a character. As such, we will increase the input speed by evaluating our technique using the first action to select the region and a prediction model to replace the second action to select the character. Lastly, we will also focus our revised version of TEXTile on providing an auto-completion and auto-correction function.

## References

1. Wiese, J., Saponas, T. S., & Brush, A. J. (2013, April). Phoneprioception: enabling mobile phones to infer where they are kept. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (pp. 2157-2166). ACM.

2. Google Glass Project. (2017). Retrieved November 5, 2018, from https://en.wikipedia.org/wiki/GoogleGlass/.

3. Yi, S., Qin, Z., Novak, E., Yin, Y., & Li, Q. (2016, April). Glassgesture: Exploring head gesture interface of smart glasses. In *Computer Communications, IEEE INFOCOM 2016-The 35th Annual IEEE International Conference on* (pp. 1-9). IEEE.

4. Kollee, B., Kratz, S., & Dunnigan, A. (2014, October). Exploring gestural interaction in smart spaces using head mounted devices with ego-centric sensing. In *Proceedings of the 2nd ACM symposium on Spatial user interaction* (pp. 40-49). ACM.

5. McCall, R., Martin, B., Popleteev, A., Louveton, N., & Engel, T. (2015, June). Text entry on smart glasses. In *Human System Interactions (HSI), 2015 8th International Conf. on* (pp. 195-200). IEEE.

6. Profita, H. P., Clawson, J., Gilliland, S., Zeagler, C., Starner, T., Budd, J., & Do, E. Y. L. (2013, September). Don't mind me touching my wrist: a case study of interacting with on-body technology in public. In *Proceedings of the 2013 International Sym. on Wearable Computers* (pp. 89-96). ACM.

7. Kim, H., Sohn, M., Kim, S., Pak, J., & Lee, W. (2007, September). Button keyboard: a very small keyboard with universal usability for wearable computing. In *IFIP Conference on Human-Computer Interaction* (pp. 343-346). Springer, Berlin, Heidelberg.

8. Lyons, K., Starner, T., Plaisted, D., Fusia, J., Lyons, A., Drew, A., & Looney, E. W. (2004, April). Twiddler typing: one-handed chording text entry for mobile phones. In *Proceedings of the SIGCHI conference on Human factors in computing systems* (pp. 671-678). ACM.

9. Frey, B., Southern, C., & Romero, M. (2011, July). Brailletouch: mobile texting for the visually impaired. In *International Conference on Universal Access in Human-Computer Interaction* (pp. 19-25). Springer, Berlin, Heidelberg.

10. Azenkot, S., Wobbrock, J. O., Prasain, S., & Ladner, R. E. (2012, May). Input finger detection for nonvisual touch screen text entry in Perkininput. In *Proceedings of Graphics Interface 2012* (pp. 121-129). Canadian Information Processing Society.

11. Tinwala, H., & MacKenzie, I. S. (2009, September). Eyes-free text entry on a touchscreen phone. In *Science and Technology for Humanity (TIC-STH), 2009 IEEE Toronto Int. Conf.* (pp. 83-88). IEEE.
12. Wobbrock, J. O., Myers, B. A., & Kembel, J. A. (2003, November). EdgeWrite: a stylus-based text entry method designed for high accuracy and stability of motion. In *Proceedings of the 16th annual ACM symposium on User interface software and technology* (pp. 61-70). ACM.

13. Perlin, K. (1998, November). Quikwriting: continuous stylus-based text entry. In *ACM Symposium on User Interface Software and Technology* (pp. 215-216).

14. 8pen Project. (2019). Retrieved April 14th, 2019, from http://www.8pen.com/.

15. Amma, C., Georgi, M., & Schultz, T. (2012, June). Airwriting: Hands-free mobile text input by spotting and continuous recognition of 3D-space handwriting with inertial sensors. In *Wearable Computers (ISWC), 2012 16th International Symposium on* (pp. 52-59). IEEE.

16. Ni, T., Bowman, D., & North, C. (2011, May). AirStroke: bringing unistroke text entry to freehand gesture interfaces. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (pp. 2473-2476). ACM.

17. Hincapié-Ramos, J. D., Guo, X., Moghadasian, P., & Irani, P. (2014, April). Consumed endurance: a metric to quantify arm fatigue of mid-air interactions. In *Proceedings of the 32nd annual ACM conference on Human factors in computing systems* (pp. 1063-1072). ACM.

18. Hsieh, Y. T., Jylhä, A., Orso, V., Gamberini, L., & Jacucci, G. (2016, May). Designing a willing-to-use-in-public hand gestural interaction technique for smart glasses. In *Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems* (pp. 4203-4215). ACM.

19. Nesbat, S. B. (2003, November). A system for fast, full-text entry for small electronic devices. In *Proceedings of the 5th international conference on Multimodal interfaces* (pp. 4-11). ACM.

20. FlickKey keyboard. (2017). Retrieved April 14th, 2019, from http://www.flickkey.com/.

21. Kim, J., Delamare, W., & Irani, P. ThumbText: Text Entry for Wearable Devices Using a Miniature Ring. In *Proceedings of Graphics interface Conference*.

22. MacKenzie, I. S., Soukoreff, R. W., & Helga, J. (2011, October). 1 thumb, 4 buttons, 20 words per minute: Design and evaluation of H4-Writer. In *Proceedings of the 24th annual ACM symposium on User interface software and technology* (pp. 471-480). ACM.

23. R Bajer, B., MacKenzie, I. S., & Baljko, M. (2012, June). Huffman base-4 text entry glove (H4 TEG). In *Wearable Computers (ISWC), 2012 16th International Symposium on* (pp. 41-47). IEEE.

24. Isokoski, P., & Raisamo, R. (2000, May). Device independent text input: A rationale and an example. In *Proceedings of the working conference on Advanced visual interfaces* (pp. 76-83). ACM.

25. Hoste, L., Dumas, B., & Signer, B. (2012, May). SpeeG: a multimodal speech-and gesture-based text input solution. In *Proceedings of the International working conference on advanced visual interfaces* (pp. 156-163). ACM.

26. Wang, C. Y., Chu, W. C., Chiu, P. T., Hsiu, M. C., Chiang, Y. H., & Chen, M. Y. (2015, August). PalmType: Using palms as keyboards for smart glasses. In *Proceedings of the 17th International Conf. on Human-Computer Interaction with Mobile Devices and Services* (pp. 153-160). ACM.

27. Yu, C., Sun, K., Zhong, M., Li, X., Zhao, P., & Shi, Y. (2016, May). One-dimensional handwriting: inputting letters and words on smart glasses. In *Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems* (pp. 71-82). ACM.

28. Grossman, T., Chen, X. A., & Fitzmaurice, G. (2015, August). Typing on glasses: adapting text entry to smart eyewear. In *Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services* (pp. 144-152). ACM.

29. Chen, X. A., Grossman, T., & Fitzmaurice, G. (2014, October). Swipeboard: a text entry technique for ultra-small interfaces that supports novice to expert transitions. In *Proceedings of the 27th annual ACM symposium on User interface software and technology* (pp. 615-620). ACM.
30. Ahn, S., Heo, S., & Lee, G. (2017, October). Typing on a Smartwatch for Smart Glasses. In *Proceedings of the 2017 ACM Int. Conf. on Interactive Surfaces and Spaces* (pp. 201-209). ACM.

31. Jones, E., Alexander, J., Andreou, A., Irani, P., & Subramanian, S. (2010, April). GesText: accelerometer-based gestural text-entry systems. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (pp. 2173-2182). ACM.

32. Prätorius, M., Valkov, D., Burgbacher, U., & Hinrichs, K. (2014, November). DigiTap: an eyes-free VR/AR symbolic input device. In *Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology* (pp. 9-18). ACM.

33. Huang, D. Y., Chan, L., Yang, S., Wang, F., Liang, R. H., Yang, D. N., ... & Chen, B. Y. (2016, May). Digitspace: designing thumb-to-fingers touch interfaces for one-handed and eyes-free interactions. In *Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems* (pp. 1526-1537). ACM.

34. Kuester, F., Chen, M., Phair, M. E., & Mehring, C. (2005, November). Towards keyboard independent touch typing in VR. In *Proceedings of the ACM symposium on Virtual reality software and technology* (pp. 86-95). ACM.

35. Seongil Lee, Sang Hyuk Hong, and Jae Wook Jeon. 2003. Designing a Universal Keyboard Using Chording Gloves. In *Proceedings of the 2003 Conference on Universal Usability (CUU '03)*. ACM, New York, NY, USA, 142-147.

36. Whitmire, E., Jain, M., Jain, D., Nelson, G., Karkar, R., Patel, S., & Goel, M. (2017). Digi-Touch: Reconfigurable Thumb-to-Finger Input and Text Entry on Head-mounted Displays. *Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies*, 1(3), 113.

37. Nirjon, S., Gummeson, J., Gelb, D., & Kim, K. H. (2015, May). Typingring: A wearable ring platform for text input. In *Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services* (pp. 227-239). ACM.

38. R Chan, L., Liang, R. H., Tsai, M. C., Cheng, K. Y., Su, C. H., Chen, M. Y., ... & Chen, B. Y. (2013, October). FingerPad: private and subtle interaction using fingertips. In *Proceedings of the 26th annual ACM symposium on User interface software and technology* (pp. 255-260). ACM.

39. Majaranta, P., Aula, A., & Räihä, K. J. (2004, March). Effects of feedback on eye typing with a short dwell time. In *Proceedings of the 2004 symposium on Eye tracking research & applications* (pp. 139-146). ACM.

40. Yu, C., Gu, Y., Yang, Z., Yi, X., Luo, H., & Shi, Y. (2017, May). Tap, dwell or gesture?: Exploring head-based text entry techniques for HMDs. In *Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems* (pp. 4479-4488). ACM.

41. Poupyrev, I., Gong, N. W., Fukuhara, S., Karagozler, M. E., Schwesig, C., & Robinson, K. E. (2016, May). Project Jacquard: interactive digital textiles at scale. In *Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems* (pp. 4216-4227). ACM.

42. Saponas, T. S., Harrison, C., & Benko, H. (2011, October). PocketTouch: through-fabric capacitive touch input. In *Proceedings of the 24th annual ACM symposium on User interface software and technology* (pp. 303-308). ACM.

43. Zhou, B., Cheng, J., Sundholm, M., & Lukowicz, P. (2014, February). From smart clothing to smart table cloth: Design and implementation of a large scale, textile pressure matrix sensor. In *International Conference on Architecture of Computing Systems* (pp. 159-170). Springer, Cham.

44. Schneegass, S., & Voit, A. (2016, September). GestureSleeve: using touch sensitive fabrics for gestural input on the forearm for controlling smartwatches. In *Proceedings of the 2016 ACM International Symposium on Wearable Computers* (pp. 108-115). ACM.
45. Strohmeier, P., Knibbe, J., Boring, S., & Hornbæk, K. (2018, March). zPatch: Hybrid Resistive/Capacitive eTextile Input. In *Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction* (pp. 188-198). ACM.

46. Ian Adafruit MPR121 12-Key Capacitive Touch Sensor Breakout (2019). https://learn.adafruit.com/adafruit-mpr121-12-key-capacitive-touch-sensor-breakout-tutorial/overview, visited April 14th, 2019.

47. Rico, J., & Brewster, S. (2010, April). Usable gestures for mobile interfaces: evaluating social acceptability. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (pp. 887-896). ACM.

48. MacKenzie, I. S., & Soukoreff, R. W. (2003, April). Phrase sets for evaluating text entry techniques. In *CHI'03 extended abstracts on Human factors in computing systems* (pp. 754-755). ACM.

49. Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. *Intl. Journal of Human-Computer Interaction, 24*(6), 574-594.