# Investigating Feedback for Two-Handed Exploration of Digital Maps Without Vision

Sandra Bardot, Marcos Serrano, Simon Perrault, Shengdong Zhao, Christophe Jouffrais

► To cite this version:

Sandra Bardot, Marcos Serrano, Simon Perrault, Shengdong Zhao, Christophe Jouffrais. Investigating Feedback for Two-Handed Exploration of Digital Maps Without Vision. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.305-324, 10.1007/978-3-030-29381-9_19. hal-02544543

HAL Id: hal-02544543

https://inria.hal.science/hal-02544543

Submitted on 16 Apr 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Investigating Feedback for Two-Handed Exploration of Digital Maps without Vision

Sandra Bardot<fnref n="1" />, Marcos Serrano<fnref n="1" />, Simon Perrault<fnref n="2" />, Shengdong Zhao<fnref n="3" /> and Christophe Jouffrais<fnref n="1" />,<fnref n="4" />,<fnref n="5" />

**Abstract.** Digital Interactive Maps on touch surfaces are a convenient alternative to physical raised-line maps for users with visual impairments. To compensate for the absence of passive tactile information, they provide vibrotactile and auditory feedback. However, this feedback is ambiguous when using multiple fingers since users may not identify which finger triggered it. To address this issue, we explored the use of bilateral feedback, i.e. collocated with each hand, for two-handed map exploration. We first introduced a design space of feedback for two-handed interaction combining two dimensions: spatial location (unilateral vs. bilateral feedback) and similarity (same vs. different feedback). We implemented four techniques resulting from our design space, using one or two smartwatches worn on the wrist (unilateral vs. bilateral feedback respectively). A first study with fifteen blindfolded participants showed that bilateral feedback outperformed unilateral feedback and that feedback similarity has little influence on exploration performance. Then we did a second study with twelve users with visual impairments, which confirmed the advantage of two-handed vs. one-handed exploration, and of bilateral vs. unilateral feedback. The results also bring to light the impact of feedback on exploration strategies.

**Keywords:** Users with Visual Impairment, Accessibility, Wearable Devices, Smartwatches, Multimodal Feedback, Map Exploration.

## 1 Introduction

In special education centers for people with visual impairments, raised-line maps are particularly important. An advantage of these maps is that they can be explored with both hands, allowing efficient two-handed exploration strategies [4]. However, making raised-line maps is a tedious process, which requires tactile document specialists to adapt and simplify visual maps. Moreover, maps have to be printed out on a special

<footnote n="1">¹ IRIT, University of Toulouse, Toulouse, France</footnote>

<footnote n="2">² Singapore University of Technology and Design (SUTD), Singapore</footnote>

<footnote n="3">³ NUS-HCI lab, National University of Singapore, Singapore</footnote>

<footnote n="4">⁴ IRIT, CNRS, Toulouse, France</footnote>

<footnote n="5">⁵ IPAL, CNRS, Singapore</footnote>

<footnote>`{firstname}.{lastname}@irit.fr`</footnote>
paper that generates raised lines when heated. As a result, raised-line maps cannot be
changed or updated easily when the content is out-of-date.

Digital Interactive Maps (DIMs) are a good alternative to raised line maps [13]. They can be generated on demand and are easily modifiable. DIMs can be displayed on a touch sensitive surface (e.g. a tablet) and explored with the fingers. DIMs are not combined with any physical objects, i.e. they are entirely digital. During exploration, they provide auditory and/or vibrotactile feedback according to touch positions on the map [13]. Previous work has proposed to explore DIMs on multitouch surfaces using one [39] or several fingers (usually one finger of each hand) [5, 19]. However, single-finger exploration raises perceptual and cognitive issues [29]. Two-handed interaction allows for preserving exploration strategies [19] but raises the challenge of providing the appropriate feedback for each hand without ambiguity. As bimanual interactions support cognitive processes related to spatial perceptions, it is then necessary to design and evaluate them to explore DIMs in ubiquitous context (e.g. with the use of an Xperia Touch). To the best of our knowledge, no previous work has: 1) explored the design of appropriate feedback for two-handed exploration of digital maps 2) compared the impact of feedback on the performance and strategies used during one- vs. two-handed map exploration.

In this work, we investigated these two research questions. First, we studied the design of unilateral vs. bilateral feedback for two-handed exploration of digital maps without vision. To this end, we proposed a design space for two-handed exploration feedback (see Fig. 1), which combines two axes: location of the feedback (unilateral vs. bilateral), and feedback similarity (same vs. different feedback). To compare the usability of the different techniques, we conducted a first study with 15 blindfolded subjects. Results revealed that bilateral feedback outperforms unilateral feedback, and that the feedback similarity has little influence on exploration performance.

Then, we explored the impact of unilateral vs. bilateral feedback on performance and exploration strategies in a study with 12 people with visual impairments. In that study, the one-handed technique with unilateral feedback is similar to existing map exploration techniques on a tablet with accessibility feature activated. The results show that the two-handed technique with bilateral feedback is between 30% and 43% faster than the other techniques, and is preferred. We also identified, for each study and task, the two-handed exploration strategies used by the participants, revealing the impact of the feedback on such strategies.

To summarize, we explored the design and the evaluation of two-handed exploration
techniques with different feedback on digital maps, and compared it with one-handed
exploration. Our contributions are: 1) the exploration of all techniques resulting from
the combination of our two axes (spatial location of the feedback and feedback simi-
larity); 2) the results of an evaluation of this design space with 15 blindfolded people;
3) the results of an evaluation with 12 users with visual impairments comparing one-
handed exploration vs. two-handed exploration techniques (with unilateral and bilat-
eral feedback) for map exploration.
# 2 Related Work

## 2.1 Exploring DIMs on touch sensitive surfaces

Digital interactive maps (DIMs) can be explored by directly moving the finger over the display surface. Important elements of the map (buildings, streets, regions, etc.) are rendered by auditory and/or vibratory feedback. For instance, TouchOverMap was designed for a smartphone [33] and provided a basic overview of a map layout. Other prototypes were designed on tablets and provided access to maritime, choropleth or city maps [8,25,37]. Some projects have also been developed on large tabletops [26]. One persistent issue with DIMs based on multitouch is that feedback can be ambiguous: when multiple fingers are moving on the surface, the user does not know which finger triggers the feedback. To release ambiguity, the user must raise up all the fingers and touch back, one finger after the other, to locate the rendered elements. Another possibility is to use only one finger to explore the map [31]. For instance, Bardot et al. [3] designed a one-handed technique to explore DIMs using a smartwatch. Vocal and vibratory feedback was provided by the smartwatch according to the finger location on the DIM. In this technique, the feedback was collocated with the moving hand.

Although users can use gestures with several fingers (two, three or four fingers) to access menus, two-handed exploration is not possible with VoiceOver or TalkBack. Hence map applications depending on them (e.g. Plans, Google Maps, Ariadne GPS, etc.) do not rely on two-handed exploration. For instance, it is not possible to use one finger as a reference point (anchor) while the other is moving around, which can provide the relative location of different elements in the map.

## 2.2 Multimodal feedback

Giudice et al. [16] designed multimodal (audio and tactile) feedback that was triggered when graphics were explored by touch on a tablet. They showed that vibro-audio interface is a good multimodal solution for providing access to dynamic graphical information. They also showed that it supports the development of mental representations of the explored graphics. Goncu et al [17] designed a more complex device that allowed the user to use the index finger of both hands, and that provided vibrations with small vibrators attached to the fingers and controlled by the tablet. The results showed that blind participants were able to understand different graphics including tables, line graphs and floorplans. Although multimodal feedback distributed on both hands may intuitively seems more efficient that single multimodal feedback, it has never been evaluated systematically.

## 2.3 Parallel feedback

The “cocktail party effect” is a known research problem [1,22]. It is the ability for a person to focus on a single speaker among several parallel conversation or background noise. Results showed that the ability of a listener depend on different factors (such as
the type, the number or the location). How many information can be gather from several
conversations while focus on one is unclear. Guerreiro et al. [18] proposed a study using
several concurrent speech channels. They aimed to understand how people with VI can
glean relevant information over two, three or four channels. The results showed that
users where able to easily get the relevant information over two or three concurrent
channels.

## 2.4 Haptic exploration strategies

Single hand exploration of an unknown digital map relies on a few strategies [38, 43]. The perimeter strategy consists in exploring an area for the identification of shape, size and contour characteristics. The gridline strategy involves searching for elements of a configuration with a systematic exploration of the display. The cyclic strategy consists in moving from one element to another in the same order, and then returning to the starting point. The reference point strategy is a back and forth movement between two elements. Previous studies [38] showed that grid and perimeter strategies are specially used for the first step of the exploration called “discovery phase”, whereas cyclic and reference point strategies better serve the memorization of the map configuration.

In a recent study [19], Guerreiro et al. investigated strategies used during one-handed and two-handed target finding on a touchscreen. Complex behaviors involving both hands have been observed and six two-handed strategies have been identified: path scan consists in a structured scanning path; focused consists in searching a small sub-section; to-the-point corresponds to a path starting close to the target, which is already known by the user; freeform represents an unstructured strategy, while freeform symmetry describes the same strategy with two-hands; finally, trailing finger is a variant of freeform with an offset between hands. Although authors did not specifically compare one-handed vs. two-handed performance, they pointed out that they found no difference in exploration time. These strategies are consistent with previous work on how people with VI explore unknown spaces by foot [15,23] or using tactile input [28].

In our study, we designed different types of bilateral feedback for digital map explo-
ration. We then assessed the effects of unilateral vs. bilateral feedback on exploration
strategies used with different one- vs. two-handed interaction techniques.

3 Bilateral feedback for two-handed exploration

While unilateral feedback can be easily produced using a multitouch surface, bilateral
feedback requires locating both hands and providing unambiguous cues. The use of two
wearable devices such as smartwatches can comply with this requirement since they
have sound and vibration capabilities. A smartwatch or a vibrating bracelet can be worn
on each hand, providing bilateral feedback that is collocated with each hand. To render
the elements of a map (i.e. region, cities and borders), we used TTS to render the name
of cities or regions, and vibratory feedback to locate map elements, such as borders
between regions.
Interaction for one-handed exploration of digital maps (e.g. on tablets and smartphones) is based on audio and vibratory feedback [25,39]. Audio feedback is often used to render text and descriptions (e.g. name of regions) while vibratory feedback is preferred to render elements that must be located accurately (e.g. points or lines). We leveraged this approach for two-handed exploration. The challenge was to provide feedback corresponding to both hands actions, while still being cognitively manageable. We know that using different feedback for each hand, for instance using different voices (e.g. male or female) and different vibration patterns (e.g. single or dual), can help to identify which hands triggers it [32]. Then we proposed a design space based on the combination of two design factors: spatial location of the feedback (unilateral or bilateral) and similarity of the feedback (same or different).

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** The design space combines two feedback dimensions: spatial location and similarity.</figcaption></figure>

The combination of these two design factors resulted in four feedback techniques (see Fig. 1): 1) unilateral with the same feedback (SF) for both hands (U-SF); 2) unilateral with different feedback (DF) for each hand (U-DF); 3) bilateral with the same feedback for both hands (B-SF); and 4) bilateral with different feedbacks for each hand (B-DF). It is interesting to note that the condition U-SF is similar to exploring a DIM on a touch screen with two fingers. We also designed a Control condition including one hand only, which is similar to exploring a DIM with only one finger on a tablet with the accessibility feature activated.

For the conditions with different feedback (U-DF and B-DF), we used a female voice and one vibration for the left hand, and a male voice and two vibrations for the right hand. For the unilateral conditions (U-SF and U-DF), if the user moves both hands at the same time, the feedback corresponds to the latest detected movement.

# 4 Material and Methods

In the current paper, we present two studies on two-handed exploration of maps without vision. In this section, we detail the rationale for such an approach, as well as the experimental task and apparatus, which was the same for both studies.
## 4.1 Rationale

Including people with visual impairments (VI) in controlled studies is not a simple task due to limited availability [35]. Three different approaches might be adopted to evaluate interaction techniques for people with VI: (1) Gathering sighted people and people with VI in the same study, which is appropriate when the study is not too long because we cannot expose users with VI to long studies; (2) When the study is too long, it is possible to conduct separate evaluations: a first study with blindfolded people to reduce the design space, and a second study with people with VI to assess performance; (3) Conducting a study on a reduced group of users with VI, which fits well with evaluating qualitative user experience, but does not with evaluating performance.

We chose the best approach (2) for analyzing performance during a long study. We adopted a two-step approach with two distinct populations: blindfolded users are non-experts of tactile exploration. They were exposed to all the conditions (Study 1). On the other hand, people with VI are, in general, expert in tactile exploration and were exposed to a subset of conditions (Study 2).

## 4.2 Maps

We asked participants to explore digital maps similar to country maps, i.e. composed of regions. To create the digital maps, we used the Voronoi algorithm [14], configured to randomly generate 30 regions of different areas fitting an A3 format (29.7×42 cm). We included constraints in our Voronoi algorithm to define the targets regions: each map has the same complexity but with different target positions and size. The maps were different for each participant and each task.

## 4.3 Task and instructions

Each session included two tasks corresponding to the discovery and memorization phases observed during haptic map exploration [38]. The first task was to explore a digital map and find 4 regions as fast as possible. To simplify the task, the names of the 4 regions were the same for all trials but their location was pseudo randomly changed. After 2 minutes, if the participant did not find the 4 regions, we considered the trial as a failure. Before switching to the second task, we indicated the eventual missing regions to the participant.

The second task was to compare the distance between 3 regions among the 4 regions found in task 1. For each trial, the three target regions were pseudo randomly chosen (the regions were chosen in order to remove comparisons that were too easy; and the trials order was random). After 2 minutes, if the participant did not answer the question, we considered the trial as a failure.

During the training session, we explained the techniques and the tasks. Participants started with a training map for each technique. Participants notified the experimenter when they found a specific region, and when they identified the two regions with the shortest distance.
**4.4 Design and procedure**

The experiment followed a within-participant design with Interaction Technique as main factor. A block included three trials with the same technique. We counterbalanced the blocks order across participants using a Latin Square. We informed participants that they could take a break between blocks. Before using each technique, participants completed a familiarization session during which the technique was explained, and they completed a few trials of each task until they felt comfortable with the technique (around 5 min).

**4.5 Apparatus**

For finger tracking, we used TopCodes tags [24] attached to the index fingers of each hand (Fig. 2). We tracked them with a Logitech C270 webcam (1280×720px, 50Hz) located above the exploration surface. We used two SimValley AW-414 Android smartwatches (91 grams, 45×44×14mm), connected to the computer running the study with TCP sockets over a local Wi-Fi.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Tags on index fingers (left) and setup (right).</figcaption></figure>

**4.6 Collected data**

We logged all tracking data, completion times and success rates for the two tasks. At the end of each block, participants had to fill a NASA-TLX questionnaire [21] about the technique they just used. At the end of the study, users had to classify the techniques by preference order and subjective effectiveness.

**4.7 Identification of exploration strategies**

We coded the exploration strategies according to Simonnet and Vieilledent [38] and Guerreiro et al [19] (see Related Work section). We first unified equivalent strategies (i.e. gridline and path scan; back-and-forth and to-the-point). As a result, we had a set of eight strategies corresponding to the discovery (path scan, perimeter, freeform, freeform symmetry) and memorization (focused, back-and-forth, cyclic and point-of-reference) phases. During data analysis, we also included an additional discovery strategy named asymmetry (i.e. a different strategy with each hand).
The “Path scan” strategy corresponds to hand movements that clearly show a system-
atic and well organized exploration of the drawing. The “Perimeter” strategy is another
discovery strategy that consists in corresponds to displacements along the physical lim-
its of the interactive surface. The “Freeform” strategy represents an unstructured strat-
egy, which does not explicitly show any form of organization. The “Freeform sym-
metry” is another unstructured strategy, but with the two-hands moving in similar di-
rections (in mirror or in parallel). The “Focused” strategy consists in searching a small
sub-section of the drawing in order to relocate a previously detected element of the
drawing. The “Back-and-forth” strategy corresponds to back and forth movements be-
tween two elements of the drawing that have been previously located. The “Cyclic”
strategy is a movement between three or more elements of the drawing, which are re-
located one after the other before coming back to the starting element (e.g. A – B – C
– A). Finally, the “Point-of-reference” strategy consists in star-like movements between
three or more targets (e.g. A – B – A – C – A). The memorization strategies can help to
locate the elements relative to each other.

In order to detect the discovery strategies in each trial, we relied on three independent judges inspecting the exploration paths. We generated one image showing the complete exploration path, and a set of images corresponding to successive periods (30 seconds duration) extracted from the whole exploration path. The visual coding by the judges was divided in two steps: first, they coded 5% of the trials set, and then they compared their respective coding in order to agree on the coding. Finally, they encoded the entire set of trials, and they collectively decided (i.e. reach a consensus if needed) which discovery strategy was the most observed in each trial.

Since three memorization strategies (back-and-forth, cyclic and point-of-reference)
were difficult to identify visually, we developed three algorithms to perform the iden-
tification process. We validated the three algorithms with unit tests and with thorough
visual exploration of many subparts of the exploration traces. In the following sections
of the paper, when the memorization strategies were observed in every trials, we report
the average number of occurrences per trial. When the memorization strategies were
observed in some trials only, we report the percentage of trials in which these strategies
were observed.

**4.8 Analysis**

Recently, the null-hypothesis significance testing (NHST) has come under criticism within the statistics [2,9] and HCI communities [11-12]. We thus report our results using estimation techniques with confidence intervals (0.95), instead of p-value statistics, consistent with APA recommendations [40].

**5 Study 1: Exploration of the design space**

The goal of this first study was to compare the impact of the design factors during map
exploration.
**5.1 Participants**

We recruited 15 sighted participants (3 females) aged between 19 and 29 (M= 23, SD=
3). Participants were blindfolded for the study. They were recruited at the university:
12 of them were undergraduate students, 2 were PhD students, and one was research
assistant. Fourteen participants were right-handed and one left-handed. Six participants
owned a smartwatch. None of them had hearing problems.

**5.2 Interaction techniques**

We compared the four two-handed techniques from our design space (U-SF, U-DF, B-SF and B-DF) and the control technique. As mentioned earlier, the control technique is similar to exploring a DIM with one finger on a tablet with the accessibility feature activated; and U-SF condition is similar to exploring a DIM with two fingers.

**5.3 Results**

In total, we collected 5 techniques × 2 tasks × 3 repetitions × 15 participants = 450 trials.

**Task 1: completion time and success rate.** On average, participants took 64.4 s (CI [55.8, 72.4]) using one hand only (Control), 68.9 s (CI [61.4, 76.5]) using two hands with unilateral feedback (average completion time for U-SF and U-DF because they are similar), and 45.5 s (CI [40.2, 51.6]) using two hands with bilateral feedback (average completion time for B-SF and B-DF because they are similar). Hence, results show a difference between techniques concerning completion time (Fig. 3). The Control and unilateral techniques were slower than bilateral techniques. Results did not show any difference regarding the feedback similarity factor (i.e. between U-SF and U-DF, or between B-SF and B-DF). Among all the trials, only nine trials were considered as failure. For each condition, it represents: 2 for Control (4.44 %), 6 for U-SF (13.3%) and 1 for U-DF (2.22 %).

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Mean time in s to complete Task 1.</figcaption></figure>
**Task 2: completion time and success rate.** Concerning the comparison of map elements location (task 2), we observed that, very often, participants remembered the position of the regions to be compared from task 1, and hence answered directly without re-exploring the map (62.5% of the trials). For the remaining trials with map exploration, we observed no difference in completion time between techniques: on average it took participants 24.8 s to complete the task. Regarding success rates, only 2 U-SF trials were incorrect (4.44%).

**Strategies used.** During task 1, across the different conditions, the most used strategies were “Path scan” (71% of the trials) and “Freeform” (15.5%). The prevalence of the “Path scan” strategy was true for all the techniques (Control: 73.3%; U-SF: 61.1%; U-DF: 66.6%; B-SF: 76.7% and B-DF: 77.2%).

During task 2, when users re-explored the map, the most used discovery strategies were “Freeform” (75.7% of the trials) and “Path scan” (13.7%). Participants also used memorization strategies: “Back-and-forth” movements with the same hand between two regions were observed in 28.8% of the trials for Control, 20% for U-SF, 26.6% for U-DF, 22.2% for U-SF, 31.1% for B-SF, and 28.8% for B-DF. “Cyclic” strategy over three regions was observed in 4% of the trials for Control, and not observed with the other techniques. The participants did not use the “Point-of-reference” strategy.

**Simultaneous hand movements.** To further investigate two-handed exploration, we identified simultaneous movements for the different techniques. Simultaneous movements correspond to exploratory movements with both hands at the same time. On average, participants used simultaneous movements in 15.7% of the trials using U-SF, in 17.4% using U-DF, in 72.1% using B-SF, in 75.8% using B-DF. This clearly illustrates how the feedback impacts behavior. With bilateral feedback, participants did use their two hand at the same time without any difficulty to manage parallel audio and vibratory feedback.

**Subjective report. User preference.** Overall, participants preferred using the bilateral technique with different feedback (B-DF). Among the 15 participants, 10 chose this technique as their favorite, 3 chose the bilateral with same feedback (B-SF), and 2 the unilateral with different feedback (U-DF).

*User feedback.* All the participants appreciated moving both hands at the same time, and most of them added positive comments about the bilateral feedback. P3 appreciated not “wasting time on determining which hand I’m using”. P5 found that “feedback is stronger” with bilateral feedback. Regarding the B-DF technique, users enjoyed “listening to different voices because it allows me to easily identify which hand triggered the feedback” (P11). Concerning unilateral feedback, P8 subjectively reported exploring slowly, and explained that he had to “move his hands one after the other in order not to miss any regions”. P12 felt “less confident using U-SF and U-DF techniques because the exploration was less intuitive”.
**5.4 Discussion**

In this first experiment, we found out that bilateral feedback techniques allow participants to perform task 1 faster than the other techniques (control and both unilateral techniques). This result confirms that bilateral feedback makes it easier for participants to locate elements on the digital map. The type of feedback (same or different) did not affect performance, as we found no differences between B-SF and B-DF in terms of accuracy or completion time. However, a large majority of participants (10/15) preferred having a different feedback (B-DF) compared to the same feedback (B-SF). In addition, most users were able to build an accurate mental representation of the map during exploration (task 1), and active comparison of elements location during task 2 was not necessary in a majority of trials.

# 6 Study 2: Impact of feedback on performance and exploration strategies

We ran a second experiment with 12 users with VI, who have an expertise in two-handed exploration of raised-line diagrams compared to the participants of the previous study. Since feedback similarity did not raise differences in the first study, we focused on the location of the feedback (unilateral vs. bilateral) for two-handed exploration.

**6.1 Interaction techniques**

In order to shorten the experiment time in this study, we focused on three conditions only: Control, U-SF and B-DF. We made that choice according to the results of the first study. Indeed, there was no difference between U-SF and U-DF on one side, and B-SF and B-DF on the other side (see Figure 3). Secondly, as mentioned earlier, the Control and U-SF conditions are similar to exploring a digital map on a tablet, with one or two fingers respectively. We selected the Control condition because there is no feedback ambiguity, and because people with visual impairment already use a similar technique on regular phones and tablets. Hence, as stated in its name, it represents a good control condition. B-DF was selected because it is the best and preferred technique identified in the first study.

**6.2 Participants**

We recruited 12 people with VI (5 females) aged between 21 and 73 years old (M=52, SD=15). The level of visual impairment varied: 9 of them were totally blind, and 3 had residual light perceptions. All participants use VoiceOver daily, and three of them use a map application (i.e. Ariadne GPS) on their smartphone. Seven participants had already used a smartwatch before, and two participants own a smartwatch. None of them had hearing problems. All of them were used to exploring raised-line diagrams.
<table><caption>**Table 1.** Description of the visually impaired participants</caption><thead><tr><th>Subject</th><th>Gender</th><th>Age</th><th>Vision</th><th>Device owned</th><th>Map application</th><th>Raised-line Expertise (1-5)</th></tr></thead><tbody><tr><td>1</td><td>M</td><td>58</td><td>Light</td><td>Smartphone<br>Smartwatch</td><td>None</td><td>2</td></tr><tr><td>2</td><td>M</td><td>50</td><td>Blind</td><td>Smartphone<br>Smartwatch</td><td>Plans<br>Ariadne GPS</td><td>3</td></tr><tr><td>3</td><td>F</td><td>21</td><td>Blind</td><td>Smartphone</td><td>None</td><td>4</td></tr><tr><td>4</td><td>M</td><td>58</td><td>Light</td><td>Smartphone<br>Tablet</td><td>None</td><td>2</td></tr><tr><td>5</td><td>F</td><td>62</td><td>Blind</td><td>Smartphone</td><td>None</td><td>4</td></tr><tr><td>6</td><td>F</td><td>47</td><td>Blind</td><td>Smartphone<br>Tablet</td><td>Ariadne GPS<br>BlindSquare</td><td>3</td></tr><tr><td>7</td><td>M</td><td>27</td><td>Blind</td><td>Smartphone</td><td>Ariadne GPS</td><td>3</td></tr><tr><td>8</td><td>M</td><td>68</td><td>Blind</td><td>Smartphone</td><td>None</td><td>4</td></tr><tr><td>9</td><td>F</td><td>45</td><td>Blind</td><td>Smartphone</td><td>None</td><td>2</td></tr><tr><td>10</td><td>F</td><td>73</td><td>Light</td><td>Smartphone</td><td>None</td><td>4</td></tr><tr><td>11</td><td>M</td><td>61</td><td>Blind</td><td>Smartphone</td><td>None</td><td>3</td></tr><tr><td>12</td><td>M</td><td>55</td><td>Blind</td><td>Smartphone</td><td>None</td><td>2</td></tr></tbody></table>

## 6.3 Maps

To design the maps, we used the Voronoi algorithm to randomly generate 30 regions on each map. We generated 216 maps in order to use a different map in each trial (for each participant, condition, and task). In addition, in order to verify whether bilateral feedback improves performance on the comparison task too, we systematically changed the map between task 1 and task 2, which means that participants had to locate four new regions in task 2 before comparing them. Then, both discovery and memorization strategies should appear in task 2.

## 6.4 Results

In total, we collected 3 techniques × 2 tasks × 3 repetitions × 12 participants = 216 trials.

**Task 1: completion time and success rate.** Results show a clear difference between techniques concerning completion time to find the four regions (task 1). Participants completed the task faster with bilateral feedback (50 s, CI [42.9, 58.8]) than with Control (76 s (CI [66.7, 88.3])) or unilateral (93 s, CI [80.7, 107.2]), as illustrated in Fig. 4. This difference is quite important, since B-DF is 31.2% faster than Control and 42.9% than U-SF. Eleven trials that lasted longer than 2 minutes were considered as failure: 7 Control (19.4%) and 4 U-SF (11.1%).
<figure><img src="image_5.png" /><figcaption>Fig. 4. Mean time in s to complete Task 1.</figcaption></figure>

**Task 2: completion time and success rate.** On average, participants completed the comparison task, i.e. finding the four regions and answering the question, in 92.8 s (CI [69.7, 118.5]) with B-DF, in 110 s (CI [92.8, 141.3]) with Control and in 137 s (CI [113.1, 171]) with U-SF. The ratio analysis confirmed the difference between B-DF and U-SF: the ratio and the corresponding CIs was above 1 (meaning that the B-DF is faster than U-SF).

<figure><img src="image_6.png" /><figcaption>Fig. 5. Mean completion time in seconds (left), and ratio (right) for Task 2.</figcaption></figure>

The time needed to find the four regions for the first time was, on average, 83 s with B-DF, 94 s with Control and 122 s with U-SF. These times are higher than those observed in task 1 because participants mixed up discovery and memorization strategies from the beginning of task 2.

The percentage of correct answers was 77% (CI [54.5, 99.5]) with the Control condition, 50% (CI [27.5, 72.50]) with U-SF, and 77% (CI [57.25, 96.75]) with B-DF. Because the task had to be completed within 4 minutes, 2 trials were considered as failure, corresponding to 1 Control (0.9%) and 1 U-SF (0.9%).

**Strategies used.** During task 1, when exploring with the control technique, the most used strategy was the “Path scan” (83.3% of the trials), and then “Perimeter” (8.3%). When exploring with two hands and unilateral feedback (U-SF), the most used strategy was the “Path scan” (66.6% of the trials) then “Freeform” (16.6%; see Fig. 6 right). When exploring with two hands and bilateral feedback (B-DF), the most used strategy was the “Path scan” (74.9%) and then “Freeform” (11.1%). We also identified
when hands explore in symmetry (“Path scan”; Fig. 6 left). This strategy appeared
41.6% with unilateral feedback, and 72.2% with bilateral feedback.

Because users were exposed to a new map during task 2, discovery strategies were
also observed. The “Path scan” was used in 88.9% of the trials with the Control tech-
nique, 91.6% with U-SF, and 77.7% with B-DF.

<figure><img src="image_7.png" /><figcaption>**Fig. 6.** Example of strategies used (the underlying digital map is virtual. It is displayed for illustration purpose only). Left: “Path scan” strategy (P4 with B-DF, task 1, simultaneous hand movements). Right: “Freeform” (P2 with U-SF, task 2, sequential hand movements). Brown and blue line correspond to left and right hands movements respectively.</figcaption></figure>

In task 2, we more specifically focused on the memorization strategies (“Back-and-forth”, “Cyclic”, and “Point-of-reference”). “Back-and-forth” movements were observed in every trials, and the number of occurrences in each trial varied according to the technique being used: 5.3 (CI[4.1, 6.3]) for Control, 3.6 (CI[2.8, 5.4]) for U-SF, and 2.4 (CI[1.9, 3.8]) for B-DF (see Figure 7).

<figure><img src="image_8.png" /><figcaption>**Fig. 7.** Example of “Back-and-forth” strategies used (the underlying digital map is virtual. It is displayed for illustration purpose only). Left: P2 with U-SF, task 2. Right: P7 with B-DF, task 2. The blue line corresponds to right hand movements.</figcaption></figure>

Contrary to “Back-and-forth” strategy, “Cyclic” and “Point-of-reference” strategies
were observed in some trials only. The “Cyclic” strategy with the same hand (i.e. going
from region A, to B, to C, and back to A) over 3 regions was observed in 33.3% of the
Control trials, 38.8% of the U-SF trials, and 22.2% of the B-DF trials. “Cyclic” strategy
over 4 regions was observed in 30.5% of the Control trials, and only 5.5% of the U-SF
and B-DF trials (see Figure 8).
<figure><img src="image_9.png" /><figcaption>**Fig. 8.** Example of “Cyclic” strategies used (the underlying digital map is virtual. It is displayed for illustration purpose only). Left: P12 with U-SF, task 2. Right: P9 with B-DF, task 2. The blue line corresponds to right hand movements.</figcaption></figure>

The “Point-of-reference” strategy with the same hand (going from A to B, back to A, and then to C) was observed in 58.3% of the Control trials, 41.6% of the U-SF trials, and 33.3% of the B-DF trials (see Figure 9).

<figure><img src="image_10.png" /><figcaption>**Fig. 9.** Example of “Point-of-reference” strategies used (the underlying digital map is virtual. It is displayed for illustration purpose only). Left: P5 with U-SF, task 2. Right: P5 with B-DF, task 2. Brown and blue lines correspond to left and right hands movements respectively.</figcaption></figure>

Finally, we observed the number of occurrences of memorization strategies realized with two hands (i.e. placing one hand on one target region, as an anchor, and using the other hand to re-locate other target regions): 2.5 (CI[2.27, 3.06]) occurrences per trial for U-SF, and 2.8 (CI[2.27, 3.37]) occurrences for B-DF. These results show that participants combined the use of both hands during two-handed exploration to compare the relative location of two regions.

In the Annex of the paper, we have added many figures corresponding to the different strategies used during exploration.

**Simultaneous hand movements.** With unilateral feedback (U-SF), 37% of the exploration time was based on simultaneous hands movements. This percentage increased to 70% with bilateral feedback (B-DF), which highlights the impact of feedback on the use of the two hands. Even though both hands were involved with U-SF, users mainly did sequential movements. On the contrary, they effortlessly do simultaneous hand movements with B-DF.

**Subjective report.** *User preference.* 9 out of 12 participants ranked the bilateral technique as their favorite, 2 preferred the control condition (none of these two people use any map application on his/her smartphone or tablet), and 1 preferred unilateral feedback. Participants preferring bilateral feedback mentioned that using both hands in parallel, and hence hearing two concurrent voices, “does not raise any cognitive issue”
(P1, P3, P4, P7, P8, P9, P10, P11 and P12). Two of them reported that they are regularly "confronted to different audio stimuli at the same time in their daily lives" (P3, P4) (i.e. listening to street noises while talking to a person). However, two of them reported that bilateral feedback requires "more concentration and cognitive load" (P2, P5), but that using it for such a short task was not disturbing. All participants reported feeling the vibration. Eleven out of twelve participants reported that wearing two smartwatches for the bilateral technique was convenient as "it is easy to understand where the feedback comes from" (P1, P10). One participant said that "wearing two smartwatches is useless" (P6 - same participant that scored unilateral technique first).

NASA-TLX. We did not observe any major difference on the different dimensions of the NASA TLX questionnaire (mental demand, physical demand, temporal demand, effort and frustration) between the three techniques. Participants evaluated the performance of B-DF and U-SF (M=75 and M=74 respectively) better than control (M=63).

# 7 Discussion

## 7.1 Main findings

These two studies showed that bilateral feedback improves map exploration and more specifically the localization and comparison of specific elements on the map. We can consider [4] that there is a difference of raised-line exploration expertise between users with VI (study 2) and blindfolded users (study 1). Because the results show the same trends in these two studies, it seems that expertise does not have any effect on the usability of the bilateral techniques. A more surprising result is that exploring the map with two hands but triggering unilateral feedback was not faster than one-handed exploration. This is an important result showing than an efficient two-handed exploration technique for digital maps must provide bilateral feedback.

## 7.2 The effect of bilateral feedback on exploration strategies

We observed different exploration strategies depending on the technique being used. With the two-handed technique and bilateral feedback, users explored the map with their two hands, and each hand mainly explored one half of the map only. In task 2 (comparison of map elements), users also took advantage of using two hands: quite often, one hand was used as an anchor while the other hand was moving around. Consequently, they did less back and forth movements between elements. With these bi-manual strategies, based on adapted bilateral feedback, users were more efficient when exploring a map and when comparing the location of different regions. When the bilateral feedback was not provided (U-SF), users were less efficient and preferred using one hand only.
## 7.3 Recommendation for touch devices

It is important to note that two of our experimental conditions (Control and U-SF) may correspond to a situation where users with VI explore maps on a tablet with TTS activated (e.g. VoiceOver or TalkBack). In such a situation, they are free to explore the map with one or two fingers (corresponding to Control and U-SF respectively), but with ambiguous feedbacks. When it is not possible to use bilateral unambiguous feedback, our results show that it is more efficient to use one finger only.

Our results also show that bilateral feedback may improve tactile exploration, provided that both hands are identified during exploration. In order to implement bilateral feedback on touchscreens, it is mandatory to identify and track each finger unambiguously (or at least right and left hands if only one finger is used on each hand). Recent studies show that finger identification can be based on finger orientation [41], or on external hardware such as a wearable device, or a camera [20].

## 7.4 Other applications for bilateral feedback

A recent study [4] has shown that people with VI intuitively use two hands to explore different types of raised line graphics, such as common drawings, mathematical graphs or neighborhood maps. Bilateral feedback could be of interest to render these different graphics accessible on touch surfaces. For instance, a two-handed exploration of digital graphs could allow the user to explore the horizontal or vertical axes with one hand, and the graph itself with the other [34]. It could also be used on tagged drawings or photos (e.g. on Facebook) to explore tags with both hands, which is similar to our experimental task. Previous work [4] showed that the average of simultaneous exploration time for people with VI on raised-line diagrams is 74.6%. In the current study, we observed that simultaneous exploration using bilateral technique was 70%. These two results are comparable and show that people with VI can perform similar two-handed exploration on raised-line diagrams and digital maps.

# 8 Conclusion and future work

In this paper, we designed and evaluated four feedback techniques for two-handed exploration of digital interactive maps by users with visual impairments. These techniques resulted from a design space combining feedback location and similarity. The results, which were comparable for both blindfolded and users with visual impairments, show the advantages of two-handed vs. one-handed exploration, providing that bilateral feedback is available and unambiguous. Although we observed these results on graphics that represent maps, we are convinced that they can apply to other interactive accessible graphics (figurative drawings, graphs, etc.). Future work will focus on how to design affordable hand and finger tracking techniques that could provide multimodal bilateral feedback. Moreover, future work will investigate bimanual interactions on complex digital maps (i.e. with different data sets).
9 Acknowledgments

We thank all the users who participated to the studies. We also thank the special edu-
cation Center IJA, and the “Cherchons pour Voir” lab, both in Toulouse, FR. This work
was part of the AccessiMap project (research grant AccessiMap ANR-14-CE17-0018).

References

1. Arons, B. (1992). A review of the cocktail party effect. *Journal of the American Voice I/O Society, 12*(7), 35-50.

2. Monya Baker. 2015. Statisticians issue warning over misuse of P values. Nature 531, 7593 (March 2015), 151. DOI:http://dx.doi.org/10.1038/nature.2016.19503

3. Sandra Bardot, Marcos Serrano, and Christophe Jouffrais. 2016. From tactile to virtual: using a smartwatch to improve spatial map exploration for visually impaired users. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '16). ACM, New York, NY, USA, 100-111. DOI: https://doi.org/10.1145/2935334.2935342

4. Sandra Bardot, Marcos Serrano, Bernard Oriola, and Christophe Jouffrais. 2017. Identifying how Visually Impaired People Explore Raised-line Diagrams to Improve the Design of Touch Interfaces. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 550-555. DOI: https://doi.org/10.1145/3025453.3025582

5. Anke Brock, Samuel Lebaz, Bernard Oriola, Delphine Picard, Christophe Jouffrais, and Philippe Truillet. 2012. Kin'touch: understanding how visually impaired people explore tactile maps. In CHI '12 Extended Abstracts on Human Factors in Computing Systems (CHI EA '12). ACM, New York, NY, USA, 2471-2476. DOI: https://doi.org/10.1145/2212776.2223821

6. Anke M. Brock, Philippe Truillet, Bernard Oriola, Delphine Picard & Christophe Jouffrais (2015) Interactivity Improves Usability of Geographic Maps for Visually Impaired People, Human-Computer Interaction, 30:2, 156-194, DOI: 10.1080/07370024.2014.924412

7. Tusher Chakraborty, Taslim Arefin Khan, and A. B. M. Alim Al Islam. 2017. FLight: A Low-Cost Reading and Writing System for Economically Less-Privileged Visually-Impaired People Exploiting Ink-based Braille System. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 531-540. DOI: https://doi.org/10.1145/3025453.3025646

8. Carroll D., Chakraborty S., Lazar J. (2013) Designing Accessible Visualizations: The Case of Designing a Weather Map for Blind Users. In: Stephanidis C., Antona M. (eds) Universal Access in Human-Computer Interaction. Design Methods, Tools, and Interaction Techniques for eInclusion. UAHCI 2013. Lecture Notes in Computer Science, vol 8009. Springer, Berlin, Heidelberg. DOI: https://doi.org/10.1007/978-3-642-39188-0_47

9. Geoff Cumming. 2014. The new statistics: Why and how. Psychological Science 25, 1 (Jan. 2014), 7–29. DOI: http://dx.doi.org/10.1177/0956797613504966

10. Delogu, F., Palmiero, M., Federici, S., Plaisant, C., Zhao, H., & Belardinelli, O. (2010). Non-visual exploration of geographic maps: Does sonification help? Disability & Rehabilitation: Assistive Technology, 5(3), 164–174. doi:10.3109/17483100903100277

11. Pierre Dragicevic. 2016. Fair statistical communication in HCI. In Modern Statistical Methods for HCI, Judy Robertson and Maurits Kaptein (Eds.). Springer International Publishing,
Cham, Switzerland, Chapter 13, 291–330. DOI: http://dx.doi.org/10.1007/978-3-319-26633-6_13

12. Pierre Dragicevic, Fanny Chevalier, and Stephane Huot. 2014. Running an HCI experiment in multiple parallel universes. In Extended Abstracts on Human Factors in Computing Systems. ACM, New York, 607–618. DOI: http://dx.doi.org/10.1145/2559206.2578881

13. Ducasse J., Brock A.M., Jouffrais C. (2018) Accessible Interactive Maps for Visually Impaired Users. In: Pissaloux E., Velazquez R. (eds) Mobility of Visually Impaired People. Springer, Cham. DOI: https://doi.org/10.1007/978-3-319-54446-5_17

14. Fortune, S. 1987. A sweepline algorithm for Voronoi diagrams. Algorithmica 2, 1--4: 153--174.

15. Gaunet, F., Martinez, J. L., & Thinus-Blanc, C. (1997). Early-blind subjects' spatial representation of manipulatory space: exploratory strategies and reaction to change. *Perception*, 26(3), 345-366

16. Giudice, N. A., Palani, H. P., Brenner, E., & Kramer, K. M. (2012). Learning non-visual graphical information using a touch-based vibro-audio interface. In Proceedings of the 14th international ACM SIGACCESS conference on Computers and accessibility - ASSETS '12 (pp. 103–110). New York, New York, USA: ACM Press. https://doi.org/10.1145/2384916.2384935

17. Goncu, C., & Marriott, K. (2011). GRAVVITAS: Generic Multi-touch Presentation of Accessible Graphics. In P. Campos, N. Graham, J. Jorge, N. Nunes, P. Palanque, & M. Winckler (Eds.), Human-Computer Interaction – INTERACT 2011 (Vol. 6946, pp. 30–48). Berlin, Heidelberg: Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-23774-4

18. João Guerreiro and Daniel Gonçalves. 2014. Text-to-speeches: evaluating the perception of concurrent speech by blind people. In *Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility* (ASSETS '14). ACM, New York, NY, USA, 169-176. DOI: https://doi.org/10.1145/2661334.2661367

19. Tiago Guerreiro, Kyle Montague, João Guerreiro, Rafael Nunes, Hugo Nicolau, and Daniel J.V. Gonçalves. 2015. Blind People Interacting with Large Touch Surfaces: Strategies for One-handed and Two-handed Exploration. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces (ITS '15). ACM, New York, NY, USA, 25-34. DOI: https://doi.org/10.1145/2817721.2817743

20. Aakar Gupta and Ravin Balakrishnan. 2016. DualKey: Miniature Screen Text Entry via Finger Identification. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI '16, 59–70. https://doi.org/10.1145/2858036.2858052

21. Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, Peter A. Hancock and Najmedin Meshkati (eds.). Elsevier, 139--183. http://doi.org/10.1016/S0166-4115(08)62386-9

22. Hawley, M. L., Litovsky, R. Y., & Culling, J. F. (2004). The benefit of binaural hearing in a cocktail party: Effect of location and type of interferer. *The Journal of the Acoustical Society of America*, 115(2), 833-843.

23. Hill, E. W., Rieser, J. J., Hill, M. M., & Hill, M. (1993). How persons with visual impairments explore novel spaces: Strategies of good and poor performers. *Journal of visual impairment & blindness*.

24. M. T Horn. TopCode: Tangible Object Placement Codes. Retrieved from http://hci.cs.tufts.edu/topcodes/

25. Nikolaos Kaklanis, Konstantinos Votis, and Dimitrios Tzovaras. 2013. Open Touch/Sound Maps: A system to convey street data through haptic and auditory feedback. *Computers & Geosciences* 57: 59--67.
26. Shaun K. Kane, Meredith Ringel Morris, Annuska Z. Perkins, Daniel Wigdor, Richard E. Ladner, and Jacob O. Wobbrock. 2011. Access overlays: improving non-visual access to large touch screens for blind users. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11). ACM, New York, NY, USA, 273-282. DOI: https://doi.org/10.1145/2047196.2047232

27. Shaun K. Kane, Jacob O. Wobbrock, and Richard E. Ladner. 2011. Usable gestures for blind people: understanding preference and performance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 413-422. DOI: https://doi.org/10.1145/1978942.1979001

28. Lahav, O., & Mioduser, D. (2008). Haptic-feedback support for cognitive mapping of unknown spaces by people who are blind. *International Journal of Human-Computer Studies, 66*(1), 23-35.

29. Jack M. Loomis, Roberta L. Klatzky and Susan J. Lederman. 1991. Similarity of Tactual and Visual Picture Recognition with Limited Field of View. In Perception vol. 20 no. 2 167-177. DOI: 10.1068/p200167

30. David McGookin, Stephen Brewster, and WeiWei Jiang. 2008. Investigating touchscreen accessibility for people with visual impairments. In Proceedings of the 5th Nordic conference on Human-computer interaction: building bridges (NordiCHI '08). ACM, New York, NY, USA, 298-307. DOI: http://dx.doi.org/10.1145/1463160.1463193

31. Valerie S Morash, Allison E Connell Pensky, Steven T W Tseng, and Joshua A Miele. 2014. Effects of using multiple hands and fingers on haptic performance in individuals who are blind. Perception 43, 6: 569-588.

32. Moray, N. (1959). Attention in dichotic listening: Affective cues and the influence of instructions. *Quarterly Journal of Experimental Psychology*, *11*, 56-60

33. Benjamin Poppinga, Charlotte Magnusson, Martin Pielot, and Kirsten Rassmus-Gröhn. 2011. TouchOver map: audio-tactile exploration of interactive maps. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '11). ACM, New York, NY, USA, 545-550. DOI: https://doi.org/10.1145/2037373.2037458

34. Rameshsharma Ramloll and Stephen Brewster. 2002. A generic approach for augmenting tactile diagrams with spatial non-speech sounds. CHI '02 extended abstracts on Human factors in computing systems - CHI '02, ACM Press, 770. http://doi.org/10.1145/506443.506589

35. Andrew Sears and Vicki Hanson. 2011. Representing users in accessibility research. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 2235-2238. DOI: https://doi.org/10.1145/1978942.1979268

36. Roy Shilkrot, Jochen Huber, Wong Meng Ee, Pattie Maes, and Suranga Chandima Nanayakkara. 2015. FingerReader: A Wearable Device to Explore Printed Text on the Go. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2363-2372. DOI: https://doi.org/10.1145/2702123.2702421

37. Mathieu Simonnet and Eamonn Ryall. 2013. Blind sailors' spatial representation using an on-board force feedback arm: two case studies. Adv. in Hum.-Comp. Int. 2013, Article 10 (January 2013), 1 pages. DOI: http://dx.doi.org/10.1155/2013/163718

38. M. Simonnet and S. Vieilledent, "Accuracy and Coordination of Spatial Frames of Reference during the Exploration of Virtual Maps: Interest for Orientation and Mobility of Blind People?," Adv. Human-Computer Interact., vol. 19, 2012.
39. Jing Su, Alyssa Rosenzweig, Ashvin Goel, Eyal de Lara, and Khai N. Truong. 2010. Timbremap: enabling the visually-impaired to use maps on touch-enabled devices. In Proceedings of the 12th international conference on Human computer interaction with mobile devices and services (MobileHCI '10). ACM, New York, NY, USA, 17-26. DOI: https://doi.org/10.1145/1851600.1851606

40. Gary R. VandenBos (Ed.). 2009. Publication Manual of the American Psychological Association (6th ed.). American Psychological Association, Washington, DC. http://www.apastyle.org/manual/

41. Feng Wang, Xiang Cao, Xiangshi Ren, and Pourang Irani. 2009. Detecting and leveraging finger orientation for interaction with direct-touch surfaces. In Proceedings of the 22nd annual ACM symposium on User interface software and technology (UIST '09). ACM, New York, NY, USA, 23-32. DOI: https://doi.org/10.1145/1622176.1622182

42. Wall, S. & Brewster, S. (2011). Providing external memory aids in haptic visualisations for blind computer users. International Journal on Disability and Human Development, 4(4), pp. 331-338. Retrieved 19 Sep. 2018, from doi:10.1515/IJDHD.2005.4.4.331

43. M. W. A. Wijntjes, T. van Lienen, I. M. Verstijnen, and A. M. L. Kappers, “Look what I have felt: unidentified haptic line drawings are identified after sketching,” Acta Psychol. (Amst)., vol. 128, no. 2, pp. 255–263, Jun. 2008.