# Can perception be extended to a “feel of north”? Tests of automaticity with the NaviEar

Christoph Witzel, Annika Lübbert, J. Kevin O’regan, Sylvain Hanneton,
Frank Schumann

► To cite this version:

Christoph Witzel, Annika Lübbert, J. Kevin O’regan, Sylvain Hanneton, Frank Schumann. Can perception be extended to a “feel of north”? Tests of automaticity with the NaviEar. Adaptive Behavior, 2022, pp.105971232211302. 10.1177/10597123221130235. hal-03846036

HAL Id: hal-03846036

https://hal.science/hal-03846036

Submitted on 27 Apr 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Can perception be extended to a “feel of north”? Tests of automaticity with the NaviEar

Adaptive Behavior
2022, Vol. 0(0) 1–26
© The Author(s) 2022

<figure><img src="image_2.png" /></figure>

Article reuse guidelines:
sagepub.com/journals-permissions
DOI: 10.1177/10597123221130235
journals.sagepub.com/home/adb

<figure><img src="image_3.png" /></figure>

Christoph Witzel<fnref n="1" />, Annika Lübbert<fnref n="2" />, J. Kevin O'Regan<fnref n="3" />, Sylvain Hanneton<fnref n="3" /> and Frank Schumann<fnref n="4" />,<fnref n="5" />

## Abstract

This study investigated the potential for the development of novel perceptual experiences through sustained training with a sensory augmentation device. We developed (1) a new geomagnetic sensory augmentation device, the NaviEar, and (2) a battery of tests for automaticity in the use of the device. The NaviEar translates head direction toward north into continuous sound according to a “wind coding” principle. To facilitate automatization of use, its design is informed by considerations of the embodiment of spatial orientation and multi-sensory integration, and it uses a sensory coding scheme derived from means for auditory perception of wind direction that is common in sailing because it is easy to understand and use. The test battery assesses different effects of automaticity (interference, rigidity of responses, and dynamic integration) assuming that automaticity is a necessary criterion to show the emergence of perceptual feel, that is, an augmented experience with perceptual phenomenal quality. We measured performance in simple training tasks, administered the tests for automaticity, and assessed subjective reports through a questionnaire. Results suggest that the NaviEar is easy and comfortable to use and has a potential for applications in real-world situations. Despite high usability, however, a 5-day training with the NaviEar did not reach levels of automaticity that are indicative of perceptual feel. We propose that the test battery for automaticity may be used as a benchmark test for iterative research on perceptual experiences in sensory augmentation and sensory substitution.

## Keywords

Automaticity, learning, perceptual feel, sensorimotor theory, sensory augmentation, sensory substitution

Handling Editor: Marek McGann

## 1. Introduction

Is it possible to extend perceptual experience, or “feel,” through an artificial device that provides novel sensory information that is not naturally available to the senses? A positive answer would both pave the way for the compensation of perceptual deficiencies and inspire the design of extensions to our existing sensory apparatus.

The potential for experiential expansion of the sensory apparatus is a fundamental prediction of the sensorimotor theory of perceptual consciousness (O'Regan, 2011; O'Regan & Noë, 2001). Sensorimotor theory seeks to explain the distinctive phenomenal character of how perception is subjectively experienced, termed its perceptual feel. Perceptual feel denotes the “what it is like” for an observer when she experiences a percept, for example, as visual or auditory, or as green or red, or loud or soft. The perceptual feel of an experience is distinct from conscious

cognitive access or evaluation of the experience but refers to the phenomenal aspect of how the experience distinctively feels (O'Regan, 2011). For example, reading a direction from a compass typically involves an effortful cognitive

## Corresponding author:

Christoph Witzel, School of psychology, University of Southampton, Building 44, Highfield Campus, Southampton 35394, UK.

Email: cwitzel@daad-alumni.de

<footnote n="1">¹School of Psychology, University of Southampton, Southampton, UK</footnote>

<footnote n="2">²Department of Neuro- and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany</footnote>

<footnote n="3">³Laboratoire Psychologie de la Perception, Université Paris Descartes, Paris, France</footnote>

<footnote n="4">⁴Sorbonne Université, INSERM, CNRS, Institut de la Vision, 17 rue Moreau, F-75012 Paris, France</footnote>

<footnote n="5">⁵Laboratoire des systèmes perceptifs, Département d'études cognitives, école normale supérieure, PSL University, CNRS, Paris, France</footnote>
evaluation of one's orientation (i.e., an interpretation of the symbolic meaning of the compass needle or digits). In contrast, hearing a sound involves an automatic (spatial) perceptual feel for the direction of the source emitting the sound.

Sensorimotor theory explains perceptual feel as constituted in a perceiver's implicit mastery of sensorimotor contingencies, the lawful ways in which sensory signals, such as the stimulations of the photoreceptors, change when the observer interacts with their environment. Similar to ecological theories of perception, such mastery of sensorimotor contingencies is based on the active use of sensorimotor invariants. However, unlike Gibsonian affordances (Gibson, 1977; for review, see Greeno, 1994), sensorimotor theory does not utilize sensorimotor invariants as sources of information for behavioral action (e.g., Favela et al., 2018; Lobo et al., 2014; Travieso et al., 2015), but as constitutive for the phenomenal character with which perceptual content is subjectively experienced (O'Regan and Noë, 2001, p. 1019).

The assumption that mastery of sensorimotor contingencies can be acquired via the observer's continuous interaction with the environment has been a basis for sensory rehabilitation and for empirical tests of the theory itself (e.g., Auvray, et al., 2007; Auvray & Myin, 2009; Auvray, Philipona, et al., 2007; Bermejo et al., 2015; Blackmore, 2001; Bompas & O'Regan, 2006a; 2006b; Froese et al., 2012; Kärcher et al., 2012; König et al., 2016; Lenay & Steiner, 2010; Lenay & Stewart, 2012; Nagel et al., 2005; O'Regan & Noë, 2001, p. 1020). Exposition to new sensorimotor contingencies should lead the perceiver to experience a novel perceptual feel, provided that the user masters the novel sensorimotor contingencies, either spontaneously, or through learning (Hurley & Noë, 2003; Kaspar et al., 2014; König et al., 2016; Myin & Degenaar, 2014; Nagel et al., 2005; Noë, 2004; O'Regan, 2011). Thus, a central prediction of sensorimotor theory is the potential to experientially expand the sensory apparatus.

This prediction appears to be supported by a large body of research on sensory substitution that suggests that it is possible to re-create perception and perceptual feel of one modality, say vision, by using technology that translates the sensory signals of that modality into another modality, for example, touch (Bach-y-Rita et al., 1969) or audition (Meijer, 1992). Seminal studies in the field have argued that we can learn to “see” with the ears (Meijer, 2015), tongue (Bach-y-Rita et al., 1998; Kupers & Ptito, 2004), or skin (Bach-y-Rita et al., 1969; White et al., 1970) using such artificial sensors. Several studies observed learning effects on how observers use such devices following several weeks of training (Bach-y-Rita et al., 1969; Bermejo et al., 2015), and in some cases, even after only a few hours (Bach-y-Rita et al., 1998; Durette et al., 2008; Favela et al., 2018; Lobo et al., 2014). These observations have led researchers in sensory substitution to claim that sufficient training with a

well-functioning sensory substitution device can provide a kind of perception that is comparable to the perceptual feel of the source modality (Auvray, et al., 2007; Bach-y-Rita & Kercel, 2003; Capelle et al., 1998; Haigh et al., 2013; Poirier, De Volder, Tranduy, et al., 2007; Proulx, 2010; Segond et al., 2013; Ward & Meijer, 2010; Ward & Wright, 2014).

Similarly, beyond substitution, sensory augmentation research suggests that a perceiver may also develop novel kinds of perceptual feel from artificial sensorimotor contingencies provided by technical sensors that convey information not available to the natural apparatus, such as the direction of geomagnetic north obtained by a magnetic compass (e.g., Kaspar et al., 2014; König et al., 2016; Nagel et al., 2005; Schumann & O'Regan, 2017).

## 1.1. Automaticity as a prerequisite of perceptual feel

It is undeniable that sensory substitution and sensory augmentation devices equip their users with new skills (see, e.g., Auvray & Myin, 2009). However, the acquisition of skilled use of a device does not necessarily imply the development of a genuine perceptual feel (Arnold et al., 2017; Auvray & Harris, 2014; Block, 2003; Brown et al., 2011; Deroy & Auvray, 2012). Evidence for the effects of sensory substitution or augmentation consists mainly of performance in very specific discrimination tasks with small sets of stimuli (e.g., Auvray et al., 2007; Bach-y-Rita et al., 1969; Buchs et al., 2021; Chebat et al., 2015; Diaz et al., 2012; Favela et al., 2018; Goeke et al., 2016; Haigh et al., 2013; Lobo et al., 2014; Proulx et al., 2015; Proulx et al., 2008; Travieso et al., 2015). Yet observed performance in this kind of discrimination may be explained by explicit cognitive interpretation rather than by the development of a perceptual feel (Deroy & Auvray, 2012; see also Goeke et al., 2016; Schumann & O'Regan, 2017). Observers may learn by heart that properties specific to the substitution signal are associated with the distinctive features of a small number of stimuli in the substituted modality, for example, that a certain vibrotactile pattern corresponds with a certain visual stimulus shape. Then, observers may explicitly retrieve this associative knowledge to identify the stimulus based on “reading” the substitution signal, similar to reading and interpreting the characters of written text (Deroy & Auvray, 2012). Neither such stimulus knowledge nor retrieval from symbolic memory necessarily reflect the experience of a perceptual feel of the substituted modality, such as the visual feel when seeing the stimulus’ shape (for a recent discussion, see Cohen, 2018; Corns, 2018; Dokic, 2018; Macpherson, 2018; Noordhof, 2018; Renier, 2018; Smith, 2018; Spence, 2018).

Other kinds of evidence for the development of perceptual feel from substitution have been obtained in subjective reports (Guarniero, 1974; Segond et al., 2005;
Vazquez-Alvarez et al., 2011; Ward & Meijer, 2010). However, such reports are difficult to assess objectively because of their idiosyncratic and subjective nature (Marcel, 2003; Spence, 2018). This indeed reflects the intrinsic challenge of research on perceptual feel: Its essential aspects are subjective and thus seem to evade objective assessment.

Here we argue that while a direct objective assessment of the content of perceptual feel is methodologically challenging, perceptual feel has a core feature that can be assessed operationally: Automaticity. Take for instance vision. The perceptual interpretation of cues that produce, for example, the perceptual feel of motion, texture, depth, color, or gestalt forms occurs automatically (e.g., Goldstein & Brockmole, 2017; Snowden et al., 2006). Automaticity can also occur with non-perceptual cognitive processes, such as in semantic priming (Hutchison et al., 2013) and reading (e.g., Roembke et al., 2021). So, automaticity is not sufficient to demonstrate perceptual feel. However, under the assumption that genuine perceptual feel is highly automatic, evidence for automaticity is a necessary condition to show the potential existence of perceptual feel. If observers acquire a perceptual feel from training with a sensory substitution device, their response to the substitution signal of the device must be highly automatic. Conversely, if users do not show automaticity, then their interpretation of the substitution signal was not a perceptual feel. In this way, automaticity allows us to distinguish candidate cases of perceptual feel from explicit cognitive

<figure><img src="image_4.png" /></figure>

**Figure 1.** Rationale behind our approach to perceptual feel. The learning process must start with the acquisition of skills to master the device as a tool. We assess participants' mastery of the NaviEar through data from the training tasks. A stronger learning success is automatization, in which the participants internalized completely the functioning of the device and directly react to the device without any cognitive effort. Automaticity implies a high level of learning and tool mastery. We test automatization through our test battery. Finally, a perceptual feel develops, that is, a distinctive subjective qualitative character of not mere using but of perceiving with the augmentation device. We used a questionnaire to get a (limited) idea of participants' subjective impressions. While due to its subjective nature perceptual feel is difficult to measure objectively, it necessarily implies that users react directly and automatically. Thus, we propose that tests of automaticity can single out cases where a genuine perceptual feel may have developed and rule out where it has not.

evaluation that involves cognitive effort. Figure 1 illustrates the logic of our approach.

Tests of automaticity can make some headway toward a more operational study of perceptual feel by excluding cases where perceptual feel was absent. The criteria that characterize automaticity of responding (Ashby & Crossley, 2012; Moors, 2015; Palmeri, 2003) can be applied as an objective criterion to investigate the development of perceptual feel. Automatic responses, first, have the property that the match between stimulation and response is rigid and cannot be changed by a simple decision (*rigidity* and *absence of cognitive control*). The rigidity of automatic responses has originally been shown by the seminal experiments of Shiffrin and Schneider (Shiffrin & Schneider, 1977). The resistance of perceptual feel to cognitive control is vividly demonstrated by mandatory fusion (Hillis et al., 2002) and experiments with inverting glasses (e.g., Degenaar, 2014; Lillicrap et al., 2013). Second, automaticity requires little attentional capacity (criterion of *efficiency*) and is *unconscious*. While perceptual feel may involve attention and consciousness, the interpretation of the sensory signal itself is unconscious. Examples are the unconscious inferences involved in the color perception of #theDress (for review, see Witzel & Gegenfurtner, 2018) and the independence of visual-haptic cue combination from modality-specific attention (Helbig & Ernst, 2008). Third, automaticity is *fast* compared to cognitive inferences. The high automaticity of perceptual processes is illustrated by the extreme speed of scene recognition, which may occur in less than 20 ms (Potter et al., 2014). Fourth, automatic performance involves integrating single elements into a coherent whole, such as in chunking (Chase & Simon, 1973) and the automatic perception of musical sequences (Richler et al., 2011). Finally, automatized performance leads to automatic interference, such as in the Stroop effect (Augustinova & Ferrand, 2014; Stroop, 1935). An example of automatic interference specifically in perception are memory color effects (for review, see Witzel & Gegenfurtner, 2018).

The present study investigated whether the perceptual interpretation of an augmented signal can reach levels of automaticity comparable to natural (i.e., non-augmented) perceptual modalities, such as vision or audition. This approach extends the demonstrations of subcognitive integration of augmentation signals (König et al., 2016; Nagel et al., 2005) to test for automaticity as a precondition for demonstrating the emergence of perceptual feel. For this purpose, we introduced a new sensory augmentation device, called *NaviEar*, and developed a battery with different tests of automaticity.

## 1.2. The NaviEar

Geomagnetic augmentation signals the direction of geomagnetic north. This approach has been introduced to
investigate learning effects in sensory augmentation. The feelSpace belt translates allocentric direction of the torso toward north into vibrations delivered on the waist (Goeke et al., 2016; Kärcher et al., 2012; Kaspar et al., 2014; König et al., 2016; Nagel et al., 2005). Following sensorimotor theory, the rationale is that mastery of sensorimotor contingencies is constitutive of perceptual feel, and hence learning to master these new sensorimotor contingencies about geomagnetic allocentric orientation should ultimately lead to a “feel of north” (and east, south, west, etc.), that is, an immediate impression of being globally oriented when engaging with these artificial sensorimotor contingencies. Results, however, have been mixed (Kaspar et al., 2014; König et al., 2016; Nagel et al., 2005).

To further develop the paradigm of geomagnetic augmentation, here we pick up the suggestion that effective augmentation devices may be informed by the science(s) of sensory integration (Cuturi et al., 2016; Gori et al., 2016). From the perspective of embodied theories of perception, such as sensorimotor theory, signaling orientation to north contingent to self-movement should lead to a perceptual feel of north not available by the natural modalities. However, considering the physiology of human embodiment, signaling the direction of north contingent to the orientation of the torso might not be the most effective way of designing an artificially augmented embodiment to endow users with a feel for their global allocentric orientation. Signaling the direction of north contingent to the orientation of the head may provide a closer link to human orientation. Head orientation aligns a magnetic directional signal with the reference frame of our natural external senses—vision, audition, and smell. From a sensory processing perspective, such implicit alignment may reduce the need for challenging reference frame transformations between these natural senses and the artificial north signal. Further, movements of the head occur much more frequently and are more fine-grained than movements of the body as a whole (Einhauser et al., 2007; Schumann et al., 2008). From a learning perspective, the head thus provides a much richer and more frequent movement source for enacting novel sensorimotor contingencies, which should aid their active exploration and learning. In addition, previous studies have shown that head movements play a key role in navigation by ear (Heller et al., 2009; Vazquez-Alvarez et al., 2011). From an ecological psychology perspective, these observations may highlight an important role of head movements in probing directional information that may further aid active exploration and learning of magnetic-directional sensorimotor contingencies. Hence, displaying magnetic allocentric information contingent to the orientation of the head rather than of the body might improve both the integration of the novel allocentric signal into the existing modalities, as well as observers’ exploration and training behavior. Indeed, in rodents, it has been shown that blind rats can use

head-centered cues from an implanted geomagnetic compass for navigation (Norimoto & Ikegaya, 2015). And in humans, a self-report (Barry, 2010) described signs of automatized interpretation and perceptual experience of cardinal directions with a “compass hat.”

Also, the tactile modality, while frequently used, may pose limits to the integration of an artificial directional signal. The tactile modality has been advocated for practical reasons (e.g., Lenay et al., 2003). However, the tactile system naturally receives signals that have a proximal origin and are experienced as such. Hence, there may be a tactile proximity bias that may work toward an egocentric and against an allocentric perceptual interpretation of tactile signals. Further, the large onset asynchronies of affordable eccentric tactile actuators, such as those used in most sensory substitution studies (e.g., Cassinelli et al., 2006; Froese et al., 2012; Kärcher et al., 2012), severely limit the design of refined haptic stimulation profiles suitable to the tactile system. These onset asynchronies also impose a substantial lag between motor action and resulting sensory stimulation, typically much larger than the signal update rate in Virtual Reality applications, and orders of magnitude beyond the lag within the natural exteroceptive receptors. With high lag, a tactile directional augmentation signal may arrive only outside the temporal integration windows of co-occurrence based multisensory integration processes (Parise & Ernst, 2016), challenging the multi-modal integration of artificial information presented via (slow) tactile signals.

For these reasons, a head-centered, auditory sensory augmentation device seemed preferable to promote automatic, sensorimotor integration. This idea is supported by our previous study in the laboratory using head-centred auditory augmentation (Schumann & O'Regan, 2017). We had investigated whether training with an auditory signal conveying head-centered cardinal directions leads to automatic interference with perceived orientation under highly controlled conditions. Participants were seated in a dark room on a motorized rotation chair to precisely control their orientation and prevent interference from visual orientation cues. Results provided evidence that training with a head-centered auditory signal leads to automatic interference with perceived self-rotation. These observations contrast those with a waist-based tactile system (Goeke et al., 2016).

A drawback of such a highly controlled approach is that sensorimotor contingencies are very much limited, for example, to the rotation of the chair. In contrast, people move freely and engage in meaningful activities, such as commuting to work or doing a walk, in everyday life. Following sensorimotor theory, we considered that embedding the augmented signal into meaningful activities may be key to sensorimotor learning because they provide richer experience of sensorimotor contingencies due to freer, more complex movements. They may also improve learning through intrinsic motivation to navigate
and self-orient in real life. In addition, we were inspired by the striking effects of acquired automaticity in a field setting demonstrated by riding a bike after reversing its handlebars (Sandlin, 2015). While those effects concerned sensorimotor control, here we were aiming at such effects of acquired automatization on the perceptual interpretation of an augmented signal.

Hence, we developed the NaviEar as a novel geomagnetic sensory augmentation device that can be worn during everyday life activities. The auditory directional signal provided by the NaviEar has been designed analogously to the use of the ears to determine wind direction, a simple and intuitive technique commonly used in sailing (e.g., Isler & Isler, 2006). This novel magnetic-auditory augmentation signal should thus be aligned with the reference frame of the natural exteroceptive senses, presented to a modality that naturally deals with distant sources, updated within the temporal integration windows of multisensory integration, and further easy and intuitive to understand. Thus, we expect users to learn the artificial relationship between head orientation to north and the auditory representation thereof quickly, improving the conditions for automatization and development of a perceptual feel of north when using the device during everyday-life activities outside the lab.

## 1.3. Tests for automaticity

Figure 1 illustrates the three parts of our experimental approach to test for automaticity: Training to use the NaviEar, tests of automaticity, and a post-experimental questionnaire.

To train participants with the NaviEar, we developed a 5-day-training program to be completed as part of everyday life activities. For users to efficiently achieve mastery of a device, it is important to engage them in relevant tasks (Bertram & Stafford, 2016; Maidenbaum, Abboud, et al., 2014). Allocentric orientation is less important in an urban than in a natural environment, as people can rely on street names and particular buildings to orient themselves. For this reason, we developed a set of training tasks specifically tailored to involve information accessible only through the NaviEar. Changes in performance on these tasks across the training period also allow us to assess general effects of training—which corresponds to evidence for “weak integration” of an augmentation signal according to Nagel et al. (2005).

To test for automaticity, we created a test battery to measure achievements of automaticity after training in the field. The battery included four test tasks that participants completed before (pre-test) and after (post-test) training. Those tests assessed involuntary automatic interference effects, cognitive flexibility, and the integration of the NaviEar signal in movement and orientation.

First, we tested whether responses to the NaviEar become robust to cognitive control in a way that indicates automaticity. To this end, we assessed participants' ability to adapt to changes in the coding function of the NaviEar, that is, the relationship between the orientation of the head and the sound signal (*rotated-signal test*).

Second, an automatic interpretation of the signal should involve recognizing complex patterns defined by the sensorimotor contingencies in dynamic sequences (chunking, pattern recognition). To test whether participants can recognize complex patterns in sequences of the NaviEar signal, we developed a *distortion-detection test*. Here, participants walked over different paths that did or did not contain a section where the orientation-to-sound relationship was distorted and had to indicate whether they noticed an abnormality in the behavior of the NaviEar signal.

Third, in the absence of a stable external visual reference, human observers are unable to walk in a perfectly straight line but instead deviate to either side. This phenomenon is called veering (Guth & LaDuke, 1994). Veering itself happens automatically, as suggested by the fact that observers are unaware of their veering and cannot prevent themselves from veering to accomplish a straight walk. If users interpret the NaviEar signal automatically, the unbiased veridical auditory-directional information should be integrated with vestibular-proprioceptive sensory signals. This should increase the precision of orientation and reduce veering (cf. Kärcher et al., 2012). For this reason, we tested whether veering reduces after training with the NaviEar (*veering test*).

Finally, if using the NaviEar automatically triggers a feeling of orientation, providing an incorrect NaviEar signal should interfere with task completion. We tested whether modulations of the NaviEar signal involuntarily affect participants' orientations when walking a series of paths (*interference test*).

The post-experimental questionnaire had two purposes. One part of the questions solicited participants' first-person evaluation about the comfort and usability of the NaviEar. Similar to previous approaches (Auvray et al., 2005; Guarniero, 1974; Nagel et al., 2005; Segond et al., 2005; Vazquez-Alvarez et al., 2011; Ward & Meijer, 2010). Another set of questions targeted effects of NaviEar usage on subjective experience. These questions complemented the tests of automaticity: If the necessary condition of automaticity is met, indications of subjective experience may contribute to disambiguate whether the observed case of automaticity may be related to perceptual feel or is a non-perceptual kind of automatic cognition.

# 2. Method

## 2.1. Participants

Recruiting participants that would complete two experimental sessions per week as part of their everyday life activities was a
particular challenge of this study. Ten healthy adults (8 women) aged 22–37 years (M = 30.60, SD = 5.72) participated in this study. This sample size seemed acceptable in comparison with previous related studies (n = 12 in Faugloire & Lejeune, 2014; n = 11 in Froese et al., 2012; n = 1 in Kärcher et al., 2012; n = 9 in Kaspar et al., 2014; n = 9 in König et al., 2016; n = 4 Nagel et al., 2005). One participant was the author CW (who was not the experimenter, see Procedure). The others received compensation of €120 for the (at least) 15 h of training tasks and automaticity tests they completed. An overview of participants and the training and tests they completed is provided in Table 1. This study has been approved by the Comité d'Éthique de la Recherche en Santé (CERES) of the Université Paris Descartes (Nr 2015/50).

## 2.2. Apparatus and materials

The NaviEar consists of three parts (see Figure 2): a head-mountable orientation sensor (YEI technology, 3-space Bluetooth sensor (YOST Labs, 1998-2015)), bone-conduction headphones (Aftershokz Sportz M3 AS450), and an Android phone with low latency in orientation to sound mapping (LG G3 16GB). The orientation sensor consisted of three types of sensors: a gyroscope, an accelerometer, and a compass. The raw output of these sensors was processed through a Kalman filter to provide the final orientation reading. The sensor's recording rate was 100 Hz. Nominal orientation accuracy was on average 1 deg and orientation resolution <0.08 deg (YOST Labs, 1998-2015). Before each use, the sensor was zeroed in on north by comparison with a manual geomagnetic compass. We implemented the NaviEar orientation-to-sound coding as well as the training tasks in a custom-made Android App

<table><caption>**Table 1.** Participants. The first initial of the participant ID (PP) indicates the gender. Participant cw is the first author (male). “Sess” and days refer to sessions of training and the number of days those sessions involved.</caption><thead><tr><th>PP</th><th>Age</th><th>Sess</th><th>Days</th><th>Rotat</th><th>Distort</th><th>Veer</th><th>Interf</th></tr></thead><tbody><tr><td>f1</td><td>36</td><td>8</td><td>5</td><td>Post</td><td>Both</td><td>Both</td><td>Post</td></tr><tr><td>f2</td><td>23</td><td>7</td><td>4</td><td>Both</td><td>Both</td><td>Both</td><td>Both</td></tr><tr><td>f3<sup>a</sup></td><td>37</td><td>8</td><td>5</td><td>Both</td><td>Both</td><td>Both</td><td>Both</td></tr><tr><td>f4</td><td>35</td><td>10</td><td>6</td><td>Both</td><td>Both</td><td>Both</td><td>Both</td></tr><tr><td>f5</td><td>22</td><td>6</td><td>4</td><td>Both</td><td>Both</td><td>Both</td><td>Both</td></tr><tr><td>f6</td><td>28</td><td>9</td><td>5</td><td>Pre</td><td>Both</td><td>Both</td><td>Both</td></tr><tr><td>f7</td><td>25</td><td>8</td><td>5</td><td>Both</td><td>Both</td><td>Both</td><td>Both</td></tr><tr><td>f8</td><td>31</td><td>8</td><td>5</td><td>Both</td><td>Both</td><td>Both</td><td>Both</td></tr><tr><td>cw</td><td>36</td><td>8</td><td>5</td><td>Both</td><td>Both</td><td>Both</td><td>Both</td></tr><tr><td>m2</td><td>33</td><td>9</td><td>5</td><td>Both</td><td>Both</td><td>Both</td><td>Both</td></tr></tbody></table>

(for source code, see https://github.com/christophWit/ NaviEar). Measured all-round latency of the final system, including the external orientation sensor, was 80–120 ms.

## 2.3. Signal

The sound signal of the NaviEar is designed in analogy to the way wind produces sound in the ears (cf. Figure 3(a)): When oriented toward the wind, the wind produces sound in both ears. In the NaviEar, the direction of the “wind” is determined to be north: When the head is oriented toward north, the NaviEar signal is played to both sides at full volume. When the head is turned clockwise away from north toward east, the signal’s volume on the right side decreases to arrive at complete silence when facing east. At the same time, the left ear continues to face north; the sound on the left is therefore still played at full volume. Turning

<figure><img src="image_5.png" /><figcaption>**Figure 2.** NaviEar Hardware. (a) Individual components of the NaviEar: Android phone, bone-conduction headphones, and orientation sensor with head-mount. (b) One of the authors (AL) wearing the NaviEar for illustrative purposes.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>**Figure 3.** NaviEar coding. (a) The “wind coding” principle: The more the ear is turned toward the wind, the louder the sound played to that side; when in positions where the wind blows directly into the ear, the volume in the respective ear is at maximum (see text for details). (b) The coding function that translates cardinal directions into sound. The x-axis corresponds to allocentric orientation in degrees, the y-axis to the (relative) volume of the bubble sound in the left (red curve) and right (blue curve) ear. Note that the functions are symmetrical, implying that the same change in the signal occurs in either direction away from north (east vs. west), just inverted with respect to the left and right ear.</figcaption></figure>

<footnote><sup>a</sup>This participant (f3) was trained with a 90-deg-rotated signal after the first rotated-signal test (see Results).</footnote>
the head further until south is reached decreases the volume on the left side, with the signal on the right remaining switched off, to end in complete silence on both sides when facing south. The same principle applies when turning counterclockwise from north toward west, with the change of the signal on the two sides now being reversed compared to the eastward turn.

The sound simulated the sound of “bubbles,” which was pleasant to hear and stood out against typical noise in an everyday environment. Users adjusted the volume of the phone to a level that was comfortable while enabling them to complete the training tasks in the respective environments. As a result, the minimum intensity was silence, and maximum intensity of the bubble sound depended on user preferences and noise level in the environment. To improve the detection of volume differences between ears, the bubble sound on the left side had a slightly higher pitch than the one played on the right (the sounds are available in the GitHub repository).

The modulation of the volume was related by sigmoid functions to allocentric orientation (cf. Figure 3(b)). This was done to account for the non-linear perception of sound intensity (e.g., Stevens, 1957).

## 2.4. Procedure

Figure 4(a) illustrates the overall time course of the study. During this time, participants engaged with the training tasks, completed the tests of automaticity before (pre) and after the training (post), and answered a questionnaire in the form of an interview at the end of the study. Experimental sessions were run by author AL.

### 2.4.1. Training procedure.
Participants completed training tasks while wearing the NaviEar during everyday life navigation activities. These navigation activities mainly involved walking through Paris, for example, on their way to work or during a walk.

Figure S1 in the supplementary material illustrates how the different tasks appeared on the smart phone. We developed five different training tasks to encourage participants to learn the NaviEar signal (Figure 4(b)). Tasks 1 to 3 involved signal mapping between the following cue-to-response combinations: Word-to-Pointing (task 1), Sound-to-Pointing (task 2), and Sound-to-Word (task 3). In task 1 (Word-to-Pointing), participants were instructed to point with their head to the direction indicated to them as a verbal label, such as “NE” (north-east). In task 2 (Sound-to-Pointing) and 3 (Sound-to-Word), they were instead presented with a probe NaviEar sound representing a certain direction and asked to orient toward it with their head (task 2) or press the button that corresponds to that direction (task 3).

<figure><img src="image_7.png" /><figcaption>**Figure 4.** Procedure. (a) Overall procedure. TRAIN = training session, PRE/POST = pre- and post-training test session, QA = questionnaire. Approximate duration is given. (b) Training tasks. Off/on = without and with NaviEar signal; delay = delay without signal, cue-on = auditory cue until response. (c) Tests for automaticity: “none/weak/strong” = without any distortion, intact NaviEar signal/weak distortion/strong distortion; “off/unbiased/biased” = no NaviEar signal, unbiased NaviEar signal/biased NaviEar signal during test completion; half of the biased trials were biased to the left and half to the right (i.e., 4 each in veering and 5 each in interference). * blindfolded, ears plugged, step-size controlled.</figcaption></figure>

Tasks 1 (Word-to-Pointing) and 2 (Sound-to-Pointing) involved two sub-versions. In the first sub-version, the NaviEar signal was turned off when participants started the trial, and they had to point in the cued direction without NaviEar (*off version*). In the second sub-version, the NaviEar signal stayed on (*on version*) while participants pointed in the cued direction so that participants could use the NaviEar signal for their pointing. The sub-versions allow for examining whether improvement in performance in those training tasks is independent of wearing the NaviEar or specific to the use of the NaviEar. A trial with the second sub-version always followed a trial with the first sub-version, implying that tasks 1 and 2 were always presented as double trials.

Task 3 (Sound-to-Word) involved three sub-versions. The first sub-version required memorizing the auditory cue: The auditory cue was presented, followed by a silence of 2.5 s before participants could give their response (*delay version*). The second subtype aimed to measure performance without memory. For this reason, a response could be given directly after cue presentation, and the auditory cue stayed on until response (*cue-on version*). In the third subtype, the NaviEar signal went on after cue presentation (*on version*). Task 3 was always presented in triple trials with the three subtypes presented one after the other (delay, cue-on, and on versions).

In the Word-to-Pointing (1) and Sound-to-Word (3) tasks, 8 directions were cued, the 4 cardinal (north, east, south, and west) and the 4 intermediate directions (NE, SE, SW, and NW). The order of presentation of these directions was randomized and counterbalanced across sub-versions
and trials. In the Sound-to-Pointing task (2), the cued direction was sampled randomly from the continuum of 360°. In all sub-versions of all three tasks, response times (time from cue presentation) and errors (the angular difference from the cue) were automatically recorded by the app. After each trial of these training tasks, feedback about the error was given by displaying the cued direction and responded direction on the mobile phone screen.

In addition, participants completed *post-trial ratings* after each trial of those tasks. Due to the situated setting, we could not control participants' familiarity with and the noise level in the environment. The post-trial ratings were meant to examine these factors in our analyses and also to get an ongoing self-report of orientation throughout the training. Participants were asked to indicate on a scale from one to five (1) their familiarity with the local environment ("how often you have been here"), (2) their orientation ("how confidently you orient (N, E, S, W)"), and (3) the loudness of noise in the environment ("how loud it is around you"). At the end of each trial, the ratings set in the previous trial were displayed and participants were asked to adjust the ratings if the situation had changed (cf. Figure S1).

In a fourth task, participants were asked to estimate differences between angles corresponding to presented sounds (*Angle-Difference*). This task was more difficult than the others and required a higher mental effort and engagement, which we considered to be beneficial to learning (e.g., Dinsmore & Alexander, 2012; Lee et al., 1994; Royet et al., 2004). Participants would hear two NaviEar probe sounds in quick succession and indicate the angular distance between them using a slide whose displacement was visualized on a circle. The two auditory probes were determined randomly so that the angles they represented had a minimum difference of 20°. This minimum difference made sure that participants could hear the onset of the second probe sound even in a noisy street environment. We recorded accuracy and response times as in the signal mapping tasks, but we did not provide feedback in this task.

In a fifth task (*map drawing*), participants had to sketch the next intersection of streets they would come across with a pencil on a sheet of paper. To draw a map, participants needed to check precisely the orientation of the streets while wearing the NaviEar. Our hope was that the need for precision would naturally lead participants to integrate the NaviEar signal in their estimation of orientation. There was no guarantee that this would work; but we thought it would be good to have a diverse set of tasks to increase the likelihood that users internalize the NaviEar signal. This task was only used for training, and no data was recorded.

Trials of these tasks were presented in random order within one session. The only exception was the Angle-Difference task. It involved more complicated instructions and required more concentration than the other tasks. So, we thought it

good to present it as one block at the end of each session. Each session involved all 5 types of tasks, with an overall of 74 trials that took between 45 and 60 min to be completed.

We aimed at a sufficient amount of training to reach ceiling effects in performance. Eight sessions of 45 min to 1 hour was more training than in previous studies that involved learning an orientation signal (Faugloire & Lejeune, 2014; Kärcher et al., 2012; Schumann & O'Regan, 2017). So, participants were told to complete at least 8 training sessions across 5 days (green slots in Figure 4(a)). On days without test sessions, participants were asked to complete at least two training sessions per day; on days with test sessions (first and last day), they completed at least one training session before the test session (Figure 4(a)). However, we did not have full control over the participants' completion of the training tasks because the experiment was conducted in the field as part of their everyday life activities. As a result, the number of training sessions varied across participants, spread over a period of four to six days (see Table 1 for details, and Results for additional explanations).

### 2.4.2. Tests for automaticity.
We used four different tests to measure the degree of automaticity participants achieved after training with the NaviEar (Figure 4(c)). Each test session (an entire pre- or post-test for an individual participant) took between two and four hours to complete. All testing took place in a park with a large open space in the north-east of Paris.

#### 2.4.2.1. Rotated signal.
In this test, we asked participants to complete a training session with a 90° or 270° shift in the coding implying that the sound “comes from” west or east, not north. Training tasks 2 to 3 (signal mapping) should become more difficult as the association between NaviEar sound and corresponding cardinal direction gets increasingly automatic.

#### 2.4.2.2. Distortion detection.
We developed the distortion-detection task to assess participants' familiarity with dynamic patterns in the NaviEar signal. Participants walked freely over eight different paths (four paths in both directions, see Figure 5 for an example). The signal either behaved normally (control) or included a “distortion window” placed at one of two possible locations along the path (reddish triangles in Figure 5). Within a distortion window, the signal would either first decelerate (i.e., changing more slowly than normal) and then accelerate (distortion 1) or vice versa (distortion 2). Acceleration and deceleration compensated each other so that the contingency of NaviEar signal with orientation was kept undistorted outside the distortion area. High and low distortion intensities made the distortion more easy or hard to detect. Participants judged whether the signal was
<figure><img src="image_8.png" /><figcaption>**Figure 5.** Examples of distortion-detection paths. Panels a and b show two of four different kinds of paths used in the distortion-detection task. Mirrored versions of each path were also used. The reddish triangles illustrate “distortion windows,” that is, the parts of the path in which the NaviEar signal could be biased. For the other paths, see Figure S3 of the Supplementary Material.</figcaption></figure>

distorted at any point during the path (“yes/no”). If so, participants indicated where along the trajectory the distortion occurred. Participants completed 24 trials that included 8 control (no distortion), 8 high (easy), and 8 low (difficult) distortion trials (except for f2 who did only 4 easy and 4 difficult distortions). Paths, distortion types (distortion 1 or 2), and windows were selected at random. We varied the orientation of the paths across subjects, so that, within a participant, the path (and its mirror) would stay identical, but it would be laid out in a different cardinal direction for other participants.

#### 2.4.2.3. Veering.
In the veering task, participants were instructed to walk 18 steps (approximately 9 m) straight ahead toward a visual target (cf. Figure 6). To quantify veering, we measured both angular deviation $α$ in degrees (difference between start and end angles given by the sensor) and distance $Δ$ in centimeters from the straight line (measured with a tape measure).

In one session, participants completed the veering task eight times without (off version) and eight times with the NaviEar signal (on version). Half of the on-trials with NaviEar were biased to test for interference, as further explained below (see Interference). Under each condition, participants completed the task once toward each intercardinal direction (NE, SE, SW, and NW), resulting in overall 16 trials per session in random order. Participants were blindfolded, their ears were plugged and covered, and step-size was controlled at 50 cm by a sling when completing a trial of the task (by making steps that stretched the sling, participants knew that they were making 50 cm steps, as required for the task).

#### 2.4.2.4. Interference.
If the NaviEar signal becomes an integral part of the user's sense of orientation, it should automatically be used in situations requiring users to orient. To test this aspect of automaticity, we implemented an “interference test,” during which we manipulated the NaviEar signal and observed whether users' orientation showed a respective bias. Just as for the veering test,

<figure><img src="image_9.png" /><figcaption>**Figure 6.** Example of veering path. The solid black line represents the straight reference line. The red line illustrates a veering path. It is approximately 9 m long because participants made 18 steps of 50 cm. Gray lines highlight distances. We measured: $Δ$ = distance between target and straight path in cm; $α$ = angle between target and actual end orientation in degree.</figcaption></figure>

participants were blindfolded, ear-plugged, ear-covered, and their step-size was controlled. We furthermore told participants to ignore the NaviEar signal throughout the test, as it might misinform them about their orientation.

Participants' task was to walk a path shown to them on a piece of paper before each task. For example, they were asked to walk four steps forward, take a 90° right turn, walk another four steps forward, take another right turn and again walk four steps forward (see Figure 7 for an illustration). The NaviEar signal was always on, from the beginning to the end of each trial. There were control conditions in which the NaviEar signal was unbiased (no bias) and the same as during training, and there were interference conditions, in which the NaviEar signal was biased. In interference conditions, the NaviEar signal could either be biased to the left or to the right, implying that the signal changed more slowly (as a function of angle) as participants turned left, and faster as they turned right (leftward bias), or vice versa (rightward bias).

There were overall 11 paths. One of these paths was the same as the veering path, that is, a straight line, but with biased signal. The other ten paths were five paths and their mirror versions. Each path involved different numbers of steps and sequences of 90° turns in the same direction. For each of these five paths, there was one version that involved turns to the left, and a mirror version with turns to the right (Figure 7). Table S1 in the supplementary material describes all 11 paths. The NaviEar App recorded the end orientation, and we measured participants' distance from the starting point using a tape measure.

There were 12 kinds of trials in the interference condition: Apart from the straight line, each of the 5 paths was once presented with a leftward and once with a rightward bias. There was an unbiased control condition for each of those 10 biased trials. The starting direction of a path was one of the 4 inter-cardinal directions. Turning direction (left or right), bias direction (leftward, rightward, or no bias), and
<figure><img src="image_10.png" /><figcaption>**Figure 7.** Example of interference path. In this example, participants had to walk 4 steps forward, turn and walk another 4 steps, and turn a second time and walk another 4 steps. Panels a and b illustrate the path with turns toward the right and toward the left, respectively. The green and red lines illustrate the path that corresponds to a leftward biased and a rightward biased NaviEar signal, respectively. Gray lines = distance Δ between target and actual end position in cm; α = angle between target and actual end orientation in degree. See Table S1 for other paths.</figcaption></figure>

start direction were randomly determined and counter-
balanced across paths. The other two kinds of interference
trials were left and right biased veering paths (straight lines).
They were measured as part of the veering task. There were
four unbiased and four biased trials (see above). Two of the
biased trials involved a leftward and two a rightward bias.

2.4.3. *Post-experimental questionnaire.* After the last test session, participants came to the laboratory to be paid and complete a questionnaire of 20 items in total (see Table S2 of the supplementary material). This was done the day of or the day after the experiment. These questions solicited participants’ subjective appraisal of how useful and comfortable they found the NaviEar, how they experienced the NaviEar, how well they could orient, and how much they paid attention to cardinal directions when orienting in everyday life. Fourteen of the 20 questions included a quantitative response (*numerical item*) consisting of a rating on a 1 to 5 scale.

**3. Results**

Data is available at Zenodo (Witzel, Lübbert, O'Regan,
Hanneton, & Schumann, 2022). As shown in Table 1, two
participants completed less than 8 training sessions (f5 and
f2), and two participants missed one of the test sessions
(f1 and f6). Participant f2 skipped one training session for
unknown reasons. Participant f6 completed unrotated in-
stead of rotated-signal tasks in the second test session by
accident (hence 9 training sessions for f6). The rotated-
signal and interference data of participant 1 is missing for
unknown reasons. The two missing training sessions of
f5 had to be removed due to an error in handling the app. For

the same reason, participant f3 was exposed to and trained
with the 90-deg-rotated signal after the first rotated-signal
test. Conditions were recoded accordingly (i.e., the 90-deg-
rotated signal was considered as training after the pre-test).
The **supplementary Figure S4** illustrates the distribution of
performance in the training tasks. Despite those differences
in training procedure, f3 and f5 yielded performance similar
to the other participants, and no participant seemed to be an
outlier. We will further test for systematic effects of the
different numbers of training sessions in each main analysis
of training and test tasks below.

First, we analyzed the development of performance in
the training tasks across days to assess the success of
learning and the mastery of the device. Then, we tested
effects of automaticity in each test task. In several of the
below analyses, we applied similar tests to two measures
(e.g., response times and errors). This requires correction for
multiple testing. To account for this, we assess our pre-
dictions with two-tailed instead of one-tailed statistics even
though our hypotheses are clearly directed. This implies that
the *p*-value is twice as high as for one-tailed tests, which
effectively corresponds to a Bonferroni correction for
2 tests. This approach is taken because it provides additional
information for the Discussion. Cohen's *d* is reported as a
measure of effect size in the t-tests following below.

3.1. *Training tasks*

If participants learned successfully to use the NaviEar and
to complete the tasks, performance should increase with
training; that is, response times and errors should decrease
across training sessions. For each participant, response
times were aggregated by median across trials. Errors were
calculated as the difference between the target direction and
the response by the participant. In the Angle-Difference
task, an error is defined as the deviation (absolute differ-
ence) of a user's estimate from the real difference between
the angles of the test sounds according to the NaviEar
coding function. There was no interesting difference be-
tween the results for the different sub-versions. For this
reason and to avoid clutter, we averaged response times and
errors across subversions. For details on each subversion of
the training tasks, consider **supplementary Table S3-4** and
**Figure S5**.

Figure 8 illustrates the development of response times
(red curves) and errors (blue curves) over training days. For
each participant, we calculated correlations between the
number of training sessions and response times and errors,
respectively, to determine whether they have a negative
(decreasing) trend over the days of training. For this, we
used Spearman correlations to account for nonlinearities.
We determined the regression slopes for the rank-ordered
data (i.e., the basis of Spearman correlations) and tested
whether slopes were below zero with a t-test. According to
<figure><img src="image_11.png" /><figcaption>**Figure 8.** Performance in training tasks over days. Red colors indicate response times; blue indicates average errors. Curves show results from the training tasks; symbols indicate results from the rotated-signal test (averaged across task subversions and participants). The shaded area and the error bars indicate standard errors of mean. In all panels, the x-axis refers to the number of training days. “Last” is the last session before the post-training rotated-signal test on the same day as that test. The red y-axis on the left represents response time in seconds, and the blue y-axis on the right is the difference between response and target angle in degree (error). Table S3 provides numerical details on averages and standard errors.</figcaption></figure>

these tests, response times and errors in all four training
tasks decreased with sessions of training, min. $d = -0.87$,
$t(9) = -2.7$, all $p < .03$. The only exception from this were
the errors in the Word-to-Pointing task (blue curve in
Figure 8(a)), the slopes of which did not differ from zero,
$d = -0.29$, $t(9) = -0.9$, $p = .38$. Analyses with non-
parametric sign tests instead of t-tests and with t-test for
Fisher-transformed Pearson coefficients provide very sim-
ilar results (see Table S4). These results show that partic-
ipants became faster with training in all tasks and more
precise in three of four tasks.

We examined whether the different numbers of training
sessions across participants (cf. Table 1) was related to the
learning effects, that is, the improvement in terms of de-
creasing response times and errors. For this purpose, we
calculated correlations across the 10 participants between
the number of training sessions and the response time and
error slopes in each training task. The more negative the
Spearman slope, the stronger the improvement across
sessions. So, we expected a negative correlation between
the amount of training and the slopes. Table S5 in the
supplementary material provides detailed statistics. Errors
in the Sound-to-Word tasks yielded a positive correlation
coefficient close to significance, *r* (8) = .60, *p* = .06. This

tendency contradicts the prediction. None of the other seven
correlations was significant (*ps* > .15). So, there was no
evidence that more training sessions led to higher levels of
improvement.

Post-trial ratings were aimed to control for external
factors that might affect the performance in the training
tasks. Familiarity with the environment and orientation
might increase the performance in the three training tasks
involving self-orientation (Word-to-Pointing, Sound-to-
Pointing, and Sound-to-Word) because participants might
know cardinal directions even without the NaviEar. In
contrast, loud noise would decrease performance, if it
prevented participants to hear the NaviEar signal. For each
participant, we calculated slopes of Spearman correlations
between subjective ratings and performance across trials. As
above, we tested across participants whether slopes differed
from zero.

Ratings of orientation were negatively correlated with
errors in the Word-to-Pointing, *d* = −1.3, *t* (9) = −4.0, *p* =
.003, and the Sound-to-Word task, *d* = −0.92, *t* (9) = −2.9,
*p* = .02. The negative Spearman slopes suggest that par-
ticipants felt more oriented when their responses were more
accurate (and vice versa). There were no other significant
Spearman slopes between subjective ratings and perfor-
mance. There were also no significant slopes for correla-
tions between subjective orientation and familiarity across
trials (all *p* > .11), which would relate the effects of ori-
entation to the familiarity with the environment.

Loudness ratings were positively correlated with errors
in the Sound-to-Pointing, *d* = 1.1, *t* (9) = 3.6, *p* = .006 and
the Sound-to-Word tasks, *d* = 1.6, *t* (9) = 5.1, *p* < .001. The
positive correlations indicate that participants made more
errors when they reported louder environmental noise in the
tasks that require listening to the NaviEar signal. All three
types of post-trial ratings were rather flat over training
sessions (cf. Figure S7), and there were no significant
Spearman slopes between sessions and post-trial ratings for
the three training tasks (all *p* > .15).

3.2. Rotated-signal test

With the rotated-signal test, we wanted to test if the flex-
ibility of participants in interpreting the NaviEar signal
decreases across training due to automatization. If this is the
case, rotating the spatial orientation relative to the auditory
signal should strongly reduce the performance (increase
response times and errors) after training compared to
the performance in the respective training tasks with the
original coding of the training sessions. The change of the
code should not affect their performance as strongly before
as after the training because participants would not need to
counteract any automatic response in the first session before
any training.
In order to test this idea, we contrasted the performance (response times and errors) in the rotated-signal tests (circles in Figure 8) to the performance in the corresponding training sessions (curves in Figure 8). For this, we calculated a “rotation contrast,” which is the difference between the performance in the rotated-signal task and the performance in the training tasks of the same day of training (day 1 for the pre-training test, last day for the post-test). A positive rotation contrast indicates that performance was higher (response times and errors lower) in the training than in the rotated-signal task, and vice versa for negative rotation contrasts.

If performance in the rotated-signal test was not affected by the change of the signal, the performance should follow the overall trend of performance in the training tasks (curves in Figure 8). In this case, performance should be slightly better in the rotated-signal test than in the training task because the rotated-signal test is measured after the corresponding training task. As a result, the rotation contrast should tend to be slightly below zero. However, in the case of automatization during training, performance in the rotated-signal test should become relatively bad after training. Hence, the rotation contrast should be positive after training (post).

Figure 9 illustrates the rotation contrast for the two tasks that involved a response to the auditory NaviEar signal, namely, Sound-to-Pointing and Sound-to-Word. Again, we pooled the different sub-versions of these tasks to avoid clutter (for detailed results on each subversion, see supplementary Table S6). To test whether performance in the rotated-signal test differed from performance in the training task, we tested whether rotation contrasts differed from zero through a paired two-tailed t-test across participants. One participant (f1) missed the pre-training, and another participant (f6) the post-training rotated-signal test, reducing the number to nine participants per test.

In the pre-training tests (dark bars in Figure 9), none of the rotation contrasts differed from zero, all *p* > .11 (top part of Table S6). In the post-training tests (light bars in Figure 9), both tasks yielded average rotation contrasts above zero, especially in the Sound-to-Word task (light bars in Figure 9(b)). Higher response times and errors reflect the difficulty of changing from the original to the rotated NaviEar signal. However, only rotation contrasts of response times in the Sound-to-Word task were significantly above zero, *d* = 0.94, *t* (8) = 2.8, *p* = .02, but neither those for response times in the Sound-to-Pointing task, *d* = 0.25, *t* (8) = 0.7, *p* = .48, nor those for errors in both tasks, both *p* > .32.

If sustained training led to automatization, rotation contrasts should be larger in the post- than in the pre-test. In all tasks, average rotation contrasts for response times and errors were higher in post- than in pre-training tests. We applied a paired two-tailed t-test across participants to

<figure><img src="image_12.png" /><figcaption>**Figure 9.** Rotated-signal test. The y-axis shows the difference between performance in the rotated-signal test and performance in the corresponding training task. The bars and the y-axis on the left side correspond to contrasts of response times (RT) in seconds and those on the right to contrasts of errors (ER) in degrees. Error bars here and in subsequent figures correspond to standard errors of mean. * *p* < 0.05.</figcaption></figure>

compare rotation contrasts in pre- and post-training tests. These comparisons could only include the eight participants with data in both, pre- and post-training tests. Only the difference between pre- and post-training rotation contrasts of response times in the Sound-to-Word task was significant (*d* = 1.12, *t* (7) = 3.2, *p* = .02), but neither those for response times in the Sound-to-Pointing task, *d* = 0.58, *t* (7) = 1.6, *p* = .14, nor those for errors in both tasks (Sound-to-Pointing: *d* = 0.14, *t* (7) = 0.4, *p* = .70; Sound-to-Word: *d* = 0.21, *t* (7) = .6, *p* = .57). After applying a Bonferroni correction for testing similar predictions in two tasks (alpha = .25), the results for response times in the Sound-to-Word task are still just so significant. Hence, these results provide weak evidence for automatization.

The extent of automatization depends on the success of the training sessions, and different participants might have engaged to different degrees with the training. Figure S8 in the Supplementary Material illustrates the pre-/post-comparisons for single participants. In the Sound-to-Pointing task, six of the eight participants produced higher rotation contrasts of response times in post- than in pre-training tests in the two tasks (cf. Figure S8(a)). This was the case for 7 of 8 in the Sound-to-Word task (cf. Figure S8(c)). To test for a relationship between training and rotated-signal tests, we calculated correlations between the Spearman-slopes in the (non-rotated) training tasks (Figure 8) and the rotation contrasts in the post-training tests (light bars in Figure 9(b)). If stronger training effects led to higher automatization, post-training contrasts should be higher when slopes are more negative, implying a negative correlation. However, only the correlation for errors in the Sound-to-Word task was significant, *r* (7) = -.70, *p* = .04 (Table S7 (left side) provides detailed statistics). So, there is some evidence for a relationship between training effects and rotated-signal effects.
We also calculated correlations between rotated-signal effects and the number of training sessions across participants. See Table S7 (right side) for detailed statistics. There was no significant correlation with rotated-signal effects on response times and errors in both, the rotated Sound-to-Pointing and Sound-to-Word task, *ps* > .38.

### 3.3. Distortion-detection test

The distortion-detection test assessed the degree to which participants integrated and coordinated the NaviEar signal with their orientation. This integration and coordination should allow them to detect the distortions in the NaviEar signal in the respective trials. Most importantly, if participants internalized the coupling between orientation and NaviEar signal through training, they should better detect distorted signals after than before training.

Figure 10 shows the proportions of correct answers before (pre) and after (post) the 5-day training. We used one-tailed t-tests to test whether participants’ accuracy was significantly higher than the chance level of 50% (red dotted line) and whether accuracy differed significantly between pre-training and post-training tests. Accuracy was significantly higher than chance level before, *d* = 0.73, *t* (9) = 2.3, *p* = .02, and after, *d* = 1.37, *t* (9) = 4.3, *p* < .001, training sessions. Accuracy was 17.9% higher for easy (strong distortion) than for difficult (weak distortion) trials, *d* = -0.93, *t*(9) = -2.9, *p* = .02. However, overall performance before (55%) and after training (62%) was not very high. These results show that it was very difficult for participants to detect smooth distortions in the signal. Accuracy after training is significantly higher than before training, *d* = 0.63, *t* (9) = 2.0, *p* = .04. This result implies that participants are slightly better in detecting distortions after than before training. This suggests that participants might have learned the coupling between sound and orientation across training. However, the effect of training seems to be rather weak.

There was neither a significant correlation with Spearman slopes for response times and errors in the training tasks, *ps* > .25, nor with the number of training sessions, *r* (8) = -0.16, *p* = .67. The effects in the distortion-detection task did not depend on amount and success of training (for details, see Table S8).

### 3.4. Veering test

If participants integrate the NaviEar in their orientation behavior, the NaviEar should reduce veering. Moreover, veering should further reduce with training if training produces the internalization and automatization of the NaviEar signal. Veering might reduce in both conditions (with and without signal) across sessions due to general learning of how to accomplish the task. However, if there is specific learning of the NaviEar signal, the

<figure><img src="image_13.png" /><figcaption>**Figure 10.** Distortion-detection test. The height of the bars indicates the frequency of correctly identifying whether the NaviEar signal was distorted or not. The dotted red line indicates chance level. \*p* < .05, \*\*p* < .01.</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>**Figure 11.** Veering test. The height of the bars indicates the deviation from a straight line in the veering test in terms of distance in centimeters (a) and in terms of angular differences (b). Pre-Post = Difference between pre- and post-test. \*p* < .05.</figcaption></figure>

reduction of veering should be stronger with than without the NaviEar signal.

Figure 11 illustrates the amount of veering, measured as distances in cm from the straight line (panel a) and angular deviations in degree from the orientation of the straight line (panel b). We calculated a repeated measures analysis of variance with the factors condition (no NaviEar and with NaviEar) and session (pre and post) for distances and orientations, separately. For both distances and orientations trials with NaviEar (dark bars) yielded significantly less veering than without NaviEar (light bars), *F* (1,9) = 9.66, *p* = .013, *η*<sub>p</sub><sup>2</sup> = 0.518, and *F* (1,9) = 12.80, *p* = .006, *η*<sub>p</sub><sup>2</sup> = 0.587. These results are in line with the predictions: They support the idea that the NaviEar generally reduces veering and improves orientation.
Concerning the effect of training, veering in post-training sessions (center groups of bars) did not significantly differ from veering in pre-training sessions (left groups of bars) (distances in panel a: $F(1,9) = 4.13, p = .07, \eta_p^2 = 0.315$; orientation in panel b: $F(1,9) = 0.19, p = .67, \eta_p^2 = 0.021$). Moreover, there was no interaction in the predicted direction, according to which the effects of learning would be higher with than without NaviEar, $F(1,9) = 2.13, p = .18, \eta_p^2 = 0.191$ and $F(1,9) = 0.0004, p = .99, \eta_p^2 = 0$. Instead, average deviations went to the opposite direction, indicating more learning without than with NaviEar (cf. rightmost group of bars).

Table S9 in the supplementary material provides details on paired, two-tailed t-tests. They showed that veering was significantly lower with than without the NaviEar in both the pre- ($d = 0.74, t(9) = 2.3, p = .04$) and the post-training session, $d = 0.96, t(9) = 3.1, p = .01$, when measured in terms of orientation (left and center groups of bars in panel b). This, however, was only the case for the pre- ($d = .98, t(9) = 3.1, p = .01$), but not for the post-training session, $d = 0.38, t(9) = 1.2, p = .26$, when veering is determined in terms of distances (left and center groups of bars in panel a). These results support the idea that participants showed less veering with than without NaviEar. However, contrary to our prediction the benefit of using the NaviEar did not increase with training.

We examined the effect of training on veering through correlations between training slopes (of each training task) and veering deviation and orientation with or without the NaviEar. We expected that stronger learning in training tasks (more negative slopes) should be correlated with higher improvement in veering after training (more negative post-pre difference), resulting in positive correlations. Table S10 provides all correlations. When measured in terms of orientation, there was a positive correlation between slopes for errors in the Sound-to-Pointing task and veering effects with the NaviEar on, $r(8) = .64, p = .045$. Slopes for errors in the Sound-to-Word task yielded a similar tendency that just missed significance, $r(8) = .63, p = .0502$. There were also significant positive correlations between the numbers of training sessions per participant (cf. Table 1) and their veering effects in the condition with NaviEar for both measures, distance ($r(8) = .78, p = .007$) and orientation ($r(8) = .69, p = .03$). There were similar tendencies in the veering condition without NaviEar, but they just missed significance ($r(8) = .62, p = .06$; $r(8) = .59, p = .08$). These correlations suggest that participants with higher levels of training and learning in those training tasks tended to improve their veering after training more than did participants with less training and learning. However, it is important to note the inconsistency of those correlations across tasks and measures.

### 3.5. Interference test

If the interpretation of the NaviEar signal becomes strongly automatized with training it should interfere with the user's orientation and bias the user to orient in the direction in which the signal is biased (see Figure 7 for illustration). Such interference should occur after training (post-test) when automatization is high, but not or much less before training (pre-test) when automatization is low.

The straight lines and 90-deg turns of each of the 11 paths define a target location and orientation of the participant (arrows in Figure 7). The location has a specific distance from the starting point (horizontal dotted line in Figure 7). A bias would result in a deviation from the target orientation (α in Figure 7), and a difference in the distance from the starting point (other dotted lines in Figure 7). Instead of the theoretical target locations and orientations, we considered trials with an unbiased signal as baseline to account for potential systematic deviations from the target orientation that are not due to the biases in the signal (e.g., when a participant has the tendency to veer to the left). We contrasted measurements with the biased signal to that baseline without signal. We determined the sign of the difference (in angle and distance) so that it was positive when it followed the bias of the signal (like α in Figure 7) and negative when it went in the opposite direction. For each participant, we averaged these deviations across all 11 paths. There was no data for the pre-training session of f1 (cf. Table 1).

Figure 12 illustrates the differences between biased and unbiased trials averaged across participants. We calculated two-tailed paired t-tests to test whether deviations in biased trials differed from zero and whether deviations differed between pre- and post-training sessions. For details, see lower part of Table S9. Neither distances nor orientations differed significantly from zero in both, the pre- and the post-training sessions, min. $d = -0.32, t(9) = -1.0, p = .34$,

<figure><img src="image_15.png" /><figcaption>**Figure 12.** Interference test. Bars indicate the deviation from the paths in the interference test in terms of distance in centimeters (a) and in terms of angles (b). Positive values along the y-axis represent deviation from the paths in the same direction as the bias in the signal, and negative values indicate deviation in the direction opposite to the bias in the signal.</figcaption></figure>
and there was no significant difference between pre- and post-training sessions, both *p* > .79. These results do not provide any support for our predictions and hence the idea that the NaviEar signal automatically interferes with navigation.

If interference effects were related to training, we ex-
pected stronger interference effects (difference between bias
after and before training) when training slopes were more
negative (negative correlation). See Table S11 for details.
There were no significant correlations. However, slopes for
errors in the Sound-to-Pointing (r (7) = -.59, p = .096) and
Sound-to-Word (r (7) = -.61, p = .08) tasks yielded
marginally, but non-significant negative correlations with
interference in orientation. There was no correlation be-
tween the number of training sessions and inference effects
measured as distance (r (7) = .16, p = .68) or orientation
(r (7) = -.36, p = .34).

3.6. Post-experimental questionnaire

Figure 13 illustrates the ratings (numerical items) of the post-experimental questionnaire. Supplementary Table S12 reports the qualitative answers and comments.

To evaluate ratings, we tested with a two-tailed t-test
whether ratings differed from 3, the center rating between
1 and 5. Users rated the overall utility of the NaviEar with on
average 3.6, which missed significance (utility in Figure 13,
d = 0.60, t (9) = 1.9, p = .09). Participants felt that the
NaviEar signal provided reliable information about cardinal
directions, as indicated by a rating of 4.2 of 5 (reliab in
Figure 13, d = 2.42, t (9) = 7.7, p < .001). The comfort rating
for the NaviEar in general was 3.6, but the difference from

<figure><img src="image_16.png" /></figure>

Figure 13. Results of post-experimental questionnaire. Panel a illustrates average rating responses to simple questions. Panel b shows responses to comparative questions. The questions about knowing directions (dir know) and getting lost (get lost) ask participants to compare the situation with and without NaviEar. The questions about cognitive effort (cog effort) and signal integration (sig integr) ask participants to compare the situation before and after training. *** p < .001; **p < .01; * p < .05 (two-tailed).

3 did not reach significance (*comfort*, *d* = 0.58, *t* (9) = 1.8, *p* = .10). Almost all participants responded that the bubble sound of the NaviEar was very comfortable as shown by the average rating of 4.5 of 5 (*snd comf*, *d* = 2.12, *t* (9) = 6.7; *p* < .001). As the main drawback in the utility and comfort of the NaviEar, users indicated the effort of charging, mounting, and connecting the three separate items of the NaviEar (mobile phone, headset, and sensor).

We also asked users to rate (1) whether they always knew
the direction they were heading when wearing and when not
wearing the NaviEar (dir know) and (2) how easily they got
lost with and without the NaviEar (get lost). Paired two-
tailed t-tests showed that users felt they knew better where
they were heading (average rating of 3.5 with vs. of
1.4 without NaviEar; *d* = 3.03, *t* (9) = 9.6, *p* < .001) and that
they got less often lost when using the NaviEar (*M* = 2.2 vs.
*M* = 3.2; *d* = −0.94, *t* (9) = −3.0, *p* = .02). These results
support the idea that the NaviEar is useful for orientation
according to subjective evaluation. However, participants
also responded they usually do not think in terms of cardinal
directions when they navigate through the city (*nesw rel,
d* = −0.80, *t* (9) = −2.5, *p* = .03). They also did not feel they
do better in remembering their path through training with
the NaviEar (*pth integr*, *d* = −1.32, *t* (9) = −4.2, *p* = .002).

Concerning the efficiency criterion of automaticity, we
also asked participants to rate (1) how much effort or
thinking was required to use the NaviEar and to do the
training tasks (*cog effort*), and (2) how integrated the Na-
viEar signal was into normal hearing without requiring
focusing on either the NaviEar signal or usual environ-
mental sound cues (*sig integr*). They had to give these
ratings by comparing retrospectively between their im-
pressions before and after they completed the training tasks.
Users felt they needed significantly less cognitive effort to
interpret the NaviEar after than before the training
(*d* = −3.51, *t* (8) = −10.5, *p* < .001; one answer missing).
According to their ratings, the NaviEar signal was almost
twice as well integrated after (*M* = 3.4 of 5) than before (*M* =
1.7) the training, *d* = 2.12, *t* (9) = 6.7, *p* < .001. In their
comments, some participants anticipated that they would
become even more automatic in interpreting the signal when
given more time for training.

Concerning changes of perceptual feel due to training,
several participants reported sudden insights about their
orientation (“aha-effects”). For example, they realized that
streets and places familiar to them were actually oriented in
rather different ways than they had thought. Most inter-
estingly, half of the participants (i.e., 5 out of 10) experi-
enced “aftereffects.” Some of them felt as if they still heard
the NaviEar signal (i.e., the bubble sound changing as a
function of head orientation); others reported an immediate
and accurate awareness of cardinal directions during the
minutes after taking off the device.
# 4. Discussion

In sum, Results showed a clear increase in performance (lower response times and errors) across sessions in all four training tasks (Figure 8), except for errors in Word-to-Pointing (blue curve in Figure 8(a)). In contrast, evidence for automaticity was weak and contradictory: Results in the rotated-signal (Figure 9) and distortion-detection tasks (Figure 10) went in the predicted direction, but were very small, missed significance (Figure 9), or were just above significance (response times in Figure 9(b); Figure 10). In contrast, results from veering (Figure 11) and interference tasks (Figure 12) did not provide any evidence of automatic effects of the NaviEar on orientation during navigation. According to the post-experimental questionnaire, participants found the NaviEar useable and comfortable (Figure 13(a)), and they had the impression their use of the NaviEar improved across training (Figure 13(b)).

## 4.1. Usability and usefulness of the NaviEar

The increase in performance across training sessions (Figure 8) indicates that participants successfully learned how to handle and interpret the NaviEar. The only exception was the errors in the Word-to-Pointing task (blue curve in Figure 8(a)). When comparing the on- and off-condition of the Word-to-Pointing task (Figure S5(b)), the presence of the NaviEar-signal increased accuracy, but no learning took place across sessions. This observation indicates that the NaviEar signal could be cognitively interpreted up from the start, and the cognitive interpretation seemed not to benefit from additional learning.

The second day of training seemed to produce a local increase of errors in the Word-to-Pointing, Sound-to-Pointing and maybe the Angle-Difference task (blue curve at day 2 in Figure 8(a), (b) and (d)). This dip in performance coincides with a decrease of subjective orientation in the post-trial ratings (Figure S7(b)). It may be explained by participants' confusion when the NaviEar signal was changed back after the rotated-signal test. In all four tasks, response times most steeply reduced after the first training session (cf. day 1 and day 2 in Figure 8). This fast improvement suggests that little training was necessary to reach a high level of behavioral mastery of the NaviEar.

The negative correlations between errors and subjective orientation in the post-trial ratings indicate that participants' feeling of orientation improved thanks to the use of the NaviEar. Familiarity, as measured through subjective ratings, did not explain those correlations with orientation. However, an increased subjective impression of orientation might also result from the feedback from the tasks, rather than being the result of mastering the augmented signal.

In addition to the results from the training tasks, participants showed less veering with than without NaviEar already in the pre-training session (Figure 11). Performance in the distortion-detection test was already above chance in the pre-training session, even though only slightly (Figure 10). These results further support the idea that participants were able to understand the NaviEar signal and to use it for orientation without sustained training.

Figure S6 in the supplementary material allows for comparing the average results in our training task with those in previous, related studies. These comparisons only give a rough reference for the evaluation of the NaviEar because all those studies used different tasks and devices. Average orientation error in our training tasks in the first session were generally equal or lower than those in our previous study (10.7 to 47.8° in Schumann & O'Regan, 2017). They were also equal or lower than the landmark-pointing performance by a blind person (~22 to 48° in Kärcher et al., 2012) and thresholds for discriminating self-orientation in a rotation chair (~32 to 40° in Goeke et al., 2016). In contrast, the orientation errors (8.9–14.6°) in Faugloire and Lejeune (2014) were smaller than errors in our first session, and they were close to errors in the Sound-to-Word and smaller than errors in Word-to-Point and Sound-to-Point tasks in the last session. These observations are rather surprising because the studies of Schumann and O'Regan (2017), Goeke et al. (2016), and Faugloire and Lejeune (2014) were conducted at a fixed location (no movement apart from rotation) in the laboratory. The higher control of interfering factors in the laboratory was expected to produce more accurate and precise measurements. That our measurements under field conditions produced performance better or close to measurements under those controlled conditions highlights the precision of our measurements and an apt use of the NaviEar.

Together, these observations suggest that the NaviEar is easy to interpret and use for orientation. This conclusion is further supported by the positive subjective ratings in the post-experimental questionnaire: Apart from practical issues about mounting and connecting the gear, the ratings indicate that the NaviEar is generally easy and comfortable to use.

The ease and comfort of using the NaviEar suggest that this device might be applicable in real-life contexts, in which allocentric orientation plays an important role, such as in navigation at sea or in landscapes with few points of reference. Beyond that, the NaviEar could be adapted for other practical applications. In particular, an option could be implemented that turns the NaviEar signal into an auditory beacon by tuning it to a target location instead of being fixed to the north. The NaviEar could then serve as an intuitive and highly practical orientation aid when commuting through a city or finding one's way back to a hotel when visiting a city as a tourist (e.g., Clemenson et al., 2021;
Pielot & Boll, 2010). Moreover, it could be of use for the navigation of firefighters in emergencies. A device that constantly updates their position relative to the way out and/or relative to other firefighters could make the difference between life and death (e.g., Harris, 2013). Finally, it might be interesting to investigate the usefulness of the NaviEar as a prosthesis for navigation in blind people (cf. Johnson & Higgins, 2006; Kärcher et al., 2012; Kupers & Ptito, 2004; Maidenbaum, Hanassy, et al., 2014; Poirier et al., 2007). However, since visually impaired users have different skills and needs than sighted users, the potential use of the device with visually impaired users requires more specific needs assessments (Elli et al., 2014; Gori et al., 2016; Pasqualotto et al., 2013).

Many other navigation devices exist for practical applications, including torso-based (e.g., Cosgun et al., 2014; Elliott et al., 2010; Faugloire & Lejeune, 2014; Johnson & Higgins, 2006; Kärcher et al., 2012; Nagel et al., 2005; Pielot & Boll, 2010; Stoll et al., 2015), hand-held (e.g., Froese et al., 2012; Maidenbaum et al., 2014), and head-mounted (e.g., Kerdegari et al., 2016; Kojima et al., 2009; Sohl-Dickstein et al., 2015) devices. Unlike the NaviEar, most of these devices provide tactile rather than auditory stimulation (exceptions are, e.g., Maidenbaum et al., 2014; Sohl-Dickstein et al., 2015; Stoll et al., 2015). The devices also differ in whether they provide information about orientation (Cosgun et al., 2014; Elliott et al., 2010; Faugloire & Lejeune, 2014; Kärcher et al., 2012; Kerdegari et al., 2016; König et al., 2016; Nagel et al., 2005; Pielot & Boll, 2010) or obstacles in the person's environment (Froese et al., 2012; Johnson & Higgins, 2006; Maidenbaum et al., 2014; Sohl-Dickstein et al., 2015). Comparative studies would be necessary to determine which device is most useful in which particular application context.

## 4.2. Automaticity

We investigated the level of automaticity achieved after training as a necessary precondition for the development of perceptual feel. Overall, the observed patterns of results are small, and ambiguous with respect to automatization.

On the one hand, a few results might reflect automaticity. Response times in the Sound-to-Word tasks slightly decreased when changing the coding in the rotated-signal test (Figure 9(b)). Average errors (Figure 9(b), right) and performance in the Sound-to-Pointing task (Figure 9(a), left) went in the same direction but were not significant. Participants were slightly better in the distortion-detection task after than before training (Figure 10). In addition, the responses to the post-experimental questionnaire suggested that there may be some effects on the participants' subjective impression that may possibly reflect automatization (Figure 13, see also Table S12).

On the other hand, other results do not support the idea that the NaviEar became automatic and internalized through training. There was no evidence for interference effects (Figure 12). Although the NaviEar improved veering, this effect did not increase with training, contrary to what we predicted in case of increasing automatization through training (Figure 11). Instead, this pattern of results may stem from participants using the NaviEar like an auditory compass, that is, by cognitively translating the signal into information about orientation (Deroy & Auvray, 2012).

The weak effects in the rotated-signal task could be the result of rapid adjustment based on cognitive interpretation. If users can easily interpret the NaviEar signal without intensive learning, they might be able to almost directly adapt to the rotation of the signal. This would explain why performance in the rotated-signal test was close to the development of performance with the unrotated signal across sessions, at least in the Word-to-Pointing and Sound-to-Pointing tasks (Figure 8(a) and (b)). In addition, subjective reports from the post-experimental questionnaire need to be treated with caution, as argued in the Introduction. This is particularly true because the pre-post questions required several days of hindsight.

At the same time, a lack of significant effects in the rotated-signal and the interference test may be due to the small number of observers and a lack of statistical power. Although, the sample size in our study was comparable to some related previous studies (Cassinelli et al., 2006; Faugloire & Lejeune, 2014; Froese et al., 2012; Kärcher et al., 2012; Kaspar et al., 2014; König et al., 2016; Nagel et al., 2005), it was lower than in our laboratory study, which did observe interference effects (n = 23 and 15 in Schumann & O'Regan, 2017). It is possible that more evidence for automaticity could be obtained if we had the opportunity to measure a larger sample of participants.

However, if a genuine perceptual feel of north had developed through NaviEar training, we would expect strong effects of automatization: Participants should detect distortions with high reliability (~90%), make more errors in all rotated-signal than in the respective training tasks, and show strong interference after training. Even if the few observed patterns were definitely due to automatization, the magnitude of these effects are in no way close to genuine perceptual feel, which occurs in a few milliseconds (cf. Introduction, e.g., Potter et al., 2014). Hence, our findings rather indicate that participants did not learn and internalize the sensorimotor contingencies conveyed by the NaviEar in a way that would allow for perceptual feel according to sensorimotor theory. In this respect, these results are similar to those obtained with the feelSpace belt that did not find evidence for automatic (or subcognitive) use, either (Kaspar et al., 2014; König et al., 2016; Nagel et al., 2005).

Previous studies did not specifically investigate automaticity but provide data that is potentially interesting
in the quest for automaticity. For example, several studies
found that skilled use of sensory augmentation devices
does not interfere with other tasks that required the user's
cognitive resources (Elliott et al., 2010; Pielot & Boll,
2010; Van Erp & Werkhoven, 2006). While those ob-
servations support the practical advantage of the re-
spective devices in the particular context under
investigation, the tests in those studies were not specific
for automatic processing.

The observation that the interpretation of the vOICe
signal might become insensitive to attentional load from a
distractor task (Stiles & Shimojo, 2015) provides evidence
for automatic processing. Similarly, our study in the lab
showed automatic interference of the augmented signal
(hearSpace) with perceptual orientation (Schumann &
O'Regan, 2017). In contrast to our present study, the first
of those studies (Stiles & Shimojo, 2015) used stimulus
images chosen to have pre-existing cross-modal mappings,
and the hearSpace signal (Schumann & O'Regan, 2017)
mimics the natural low-level spatial contingencies of distal
sounds. Thus, it is not clear to which degree the observed
automaticity reflects learning or preexisting mechanisms. In
addition, the above two studies were conducted in the
laboratory. It is not clear how their results transfer to the
more complex situations and activities in the field.

The lack of automaticity in our study may be because of
insufficient or unsuccessful training. In contrast to the
experiment in the laboratory (Schumann & O'Regan, 2017),
field experiments and training depend on the participants
activities and their context, which prevents a control of all
potentially interfering factors (cf., e.g., Kärcher et al., 2012;
Kaspar et al., 2014; Nagel et al., 2005). Post-trial subjective
ratings indicated that loudness of environmental noise
reduced performance in training tasks that required lis-
tening to the NaviEar signal (Sound-to-Pointing and
Sound-to-Word). This interference with training may have
reduced learning the sensorimotor contingencies. Al-
though familiarity was not related to training performance,
we cannot fully exclude that some participants might have
used other kinds of information that would undermine
training success. For example, participants could have
used the position of the sun (in relation to time) to orient
themselves in the Word-to-Pointing and Sound-to-
Pointing task without listening to the NaviEar. Never-
theless, the observation that participants performed much
better with than without the NaviEar in those tasks (Figure
S5) suggests that they did engage with the NaviEar signal,
no matter the potential effects of environmental noise and
other factors.

Furthermore, participants had completed unequal
amounts of training sessions (between 6 and 10, cf. Table 1),
making it difficult to compare learning success across in-
dividual participants. Generally, the amount of training
necessary to reach learning effects can strongly differ across

individuals (e.g., Dale et al., 2021; Katz et al., 2021; Traut
et al., 2021; Wen et al., 2021). So, a fixed amount of training
alone would not guarantee that participants underwent
sufficient training to reach automaticity. The question is
whether the training was above the minimum needed to
expect automatization in all participants.

Unfortunately, we cannot know a priori how much
training is needed individually or on average. The training
period of 6–10 sessions might have been too short to acquire
automatization, especially in the comparatively complex
context of the field study (Ashby & Crossley, 2012; Bertram
& Stafford, 2016). Correlations between measures of au-
tomaticity, numbers of sessions, and learning success
provided some indication that amount of training (number
of sessions) and learning (slopes) might be related to per-
formance in the veering and the interference test. A definite
conclusion cannot be drawn here because results are in-
consistent across tasks and measures, and the statistical
power is low due to the small number of participants. So, we
cannot exclude that the lack of automaticity effects is due to
insufficient learning. We might obtain much higher levels of
automaticity when users train for weeks, months, or even
years (e.g., Sandlin, 2015; Ward & Meijer, 2010).

However, our device conveyed relatively simple signals
that are easy to interpret. This idea is supported by the high
usability of the NaviEar, as discussed in the previous
section. Devices that require the interpretation of more
complex signals, for example, when translating visual
images into sound (e.g., Bologna et al., 2009) or touch (e.g.,
Bach-y-Rita et al., 1969), likely require much higher
amounts of training to make any potential automatization
possible. At the same time, it is also possible that, in contrast
to sensory augmentation, automatization in sensory sub-
stitution, for example, of vision by sound, could benefit
from predisposed brain structures that may be recruited by
the substituting signal (e.g., Poirier, De Volder, & Scheiber,
2007).

Another factor that could have hampered automatization
in our study is that the geomagnetic allocentric information
transmitted by the NaviEar might be of minor importance
for orientation in an urban environment. We tried to rein-
force participants’ engagement with this novel information
through the training tasks. However, it would be beneficial
to further improve the training tasks to be more meaningful
in the context of everyday life activity (Bertram & Stafford,
2016; Lenay et al., 2003).

4.3. Test battery for automaticity

Although, we focused on the implications of sensorimotor
theory, the observation that perception is immediate and
automatic does not depend on specific theories about the
nature of perceptual experience. The assumption that per-
ception is automatic does neither imply that all automatic
responses are perceptual, nor that the development of
perceptual experience first transitions through a stage of
non-perceptual automaticity, as suggested by our Figure 1.
For example, this assumption is in line with representational
(Marr, 2010 [1982]), Bayesian (Geisler, 2011; Hillis et al.,
2002), ecological (Greeno, 1994), empirical (Purves et al.,
2001, 2015), and predictive coding (Friston, 2018; Seth,
2014; Vance & Stokes, 2017) accounts of perception.
Hence, the idea of testing for automaticity as a necessary
criterion for changes in perceptual experience is not limited
to a specific theoretical framework and may be used as a
more general, methodological approach.

As already noted in the introduction, to evaluate how
participants use the respective device, most studies in
sensory substitution and augmentation rely exclusively on
simple performance measures (e.g., Bologna et al., 2009;
Brown et al., 2011; Durette et al., 2008; Proulx et al.,
2008), subjective reports (e.g., Guarniero, 1974; Kaspar
et al., 2014; Ward & Meijer, 2010), or both (e.g., Auvray
et al., 2007; Bach-y-Rita et al., 1969; Bermejo et al., 2015;
Haigh et al., 2013; Nagel et al., 2005; Segond et al., 2005).
However, these measures do not allow for firm conclusions
about the emergence of perceptual feel through substitu-
tion or augmentation devices. Any intensive training leads
to some improvement in performance. Such improvements
do not necessarily imply automaticity or perceptual feel in
the use of the devices. This may also be visible in our
results. While first-person subjective reports are important
for evaluating perceptual feel, such reports strongly de-
pend on how users interpret the testing situation and the
experimenter's expectations. Nevertheless, many studies
have at least implicitly suggested that their measures
produced a genuine perceptual feel or changes in per-
ceptual awareness (Auvray et al., 2007; Bach-y-Rita et al.,
1969; Bach-y-Rita et al., 1998; Bach-y-Rita & Kercel,
2003; Bermejo et al., 2015; Capelle et al., 1998; Durette
et al., 2008; Haigh et al., 2013; Kaspar et al., 2014; König
et al., 2016; Kupers & Ptito, 2004; Meijer, 1992; 2015;
Nagel et al., 2005; Poirier et al., 2007; Proulx, 2010;
Segond et al., 2013; Ward & Meijer, 2010; Ward & Wright,
2014; White et al., 1970). Given the insufficiencies of the
measures used, claims and conclusions about changes in
perceptual feel or awareness are premature and unjustified
(Auvray & Harris, 2014; Block, 2003; Brown et al., 2011;
Deroy & Auvray, 2012).

We suggest that studies should show high levels of
automaticity as a necessary—but not sufficient—criterion
for perceptual feel. As mentioned above (section 4.2),
several studies have measured some aspects of automaticity
in the use of sensory augmentation and sensory substitution
devices (Elliott et al., 2010; Kaspar et al., 2014; König et al.,
2016; Nagel et al., 2005; Pielot & Boll, 2010; Schumann &
O'Regan, 2017; Stiles & Shimojo, 2015; Van Erp &
Werkhoven, 2006). However, to our knowledge, the test

battery we developed here is the first that systematically
tests not one but a range of criteria for automatization in the
use of sensory augmentation and substitution devices.

Our test battery could serve as a model for investigating
automaticity with sensory augmentation and sensory sub-
stitution devices. For example, following our work an
ongoing study (Negen et al., 2021) implemented tests of
automaticity to investigate auditory cues to distance, similar
to echolocation.

However, the individual tests of the battery we pro-
posed need improvement in their technical im-
plementation. For example, a technical improvement
would be to conduct the tests of the battery in a more
controlled setting rather than in a park or to implement
them in a virtual environment (e.g., Negen et al., 2021) to
better control confounding factors and measurement
noise. Moreover, it would be important to optimize and
benchmark the sensitivity of the tests of automaticity we
propose. The single tests in this battery might also need
some adjustments to match the specificities of other kinds
of sensory augmentation and substitution.

The test battery can also be extended with additional
tests. Taking inspiration from the studies mentioned above
(Elliott et al., 2010; König et al., 2016; Stiles & Shimojo,
2015), an interesting additional test could involve a dis-
tractor task to assess efficiency and cognitive load as a
further criterion of automaticity. If the interpretation of the
augmenting signal becomes automatized, it should not
produce distraction due to dividing attention (e.g., Wickens,
2021). Negen et al. (2021) propose a test for “redundant
signal effects” to test whether the augmenting signal (e.g.,
the auditory cues to distance, or here the NaviEar signal)
becomes integrated in the augmented signal (e.g., visual
cues about distance or orientation). Such an effect implies
that participants are faster to respond to the combination of
the augmented and augmenting signals than to each of these
signals alone.

Another way to extend the test battery could include
tests of distal attribution (Auvray et al., 2005; Hartcher-
O'Brien & Auvray, 2014; Siegle & Warren, 2010; Visell,
2008). In the case of the NaviEar, a perceptual feel of
north might be considered a case of distal attribution in
that the proximal auditory signal in the ear would be
attributed to the distal cardinal directions in the envi-
ronment. However, known measures of distal attribution
in sensory substitution rely on subjective verbal reports
(for review see, e.g., Hartcher-O'Brien & Auvray, 2014),
hence eluding our aim of establishing objective tests of
automaticity. Nevertheless, paradigms for measuring
perceptual constancy may well serve as objective tests for
distal attribution in some sensory substitution or aug-
mentation devices. For example, it has been proposed to test
for size constancy in sensory substitution (Renier et al., 2005,
2006). Another example is color constancy, where observers
perceive the color of a surface, that is, the distal stimulus, as
constant across illumination changes even though the
proximal stimulus on the retina changes with illuminations
(for review, see, e.g., Witzel & Gegenfurtner, 2018). Using
established paradigms of color constancy, one could test
whether the interpretation of a substitution signal will reflect
the proximal or distal color stimulus after sustained training
with color-substitution devices (for review, see Hamilton-
Fletcher & Ward, 2013).

**5. Conclusions**

The present study investigated the development of auto-
maticity as a criterion for demonstrating the emergence of a
distinctive perceptual feel in sensory augmentation. Per-
ceptual feel is a key goal for sensory rehabilitation and
empirical testing of sensorimotor theory; however, it is
challenging to objectively assess. We proposed that auto-
maticity should be considered as a necessary criterion to test
for the emergence of a new perceptual feel as it is more
amenable to objective verification than first-person intro-
spective methods. To test for automaticity, we developed (1)
a new augmentation device, the NaviEar, as a physiologi-
cally and neurophysiologically informed improvement of
previous approaches to geomagnetic augmentation. The
NaviEar translates the allocentric orientation of the head
toward the direction of north into a continuous sound signal.
Further, we developed (2) a battery of tests that assess the
level of automaticity in the use of the device. We draw three
conclusions: First, our findings indicate that the NaviEar is
easy and comfortable to use. High usability suggests that it
could be interesting to adapt and apply the NaviEar to ev-
eryday life situations and professional contexts. Second,
despite high usability, training with the NaviEar over 5 days
did not lead to the level of automaticity expected if users
developed a new perceptual feel of geomagnetic allocentric
orientation or a sense of north. These results illustrate that it
may be non-trivial to meet the prerequisites of artificial
perceptual feel even with a physiologically and neurophysi-
ologically informed device design. Third, we propose that
tests of automaticity can serve as an objective criterion for
iterative research that may uncover the conditions for the
emergence of experiential perceptual feel from sensory
augmentation signals.

Acknowledgments

We thank James Alvarez for coding the NaviEar app, and Gu-
glielmo Montone, Lionel Granjon, Denis Pellerin, and Christian
Graff for helpful discussion.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with re-
spect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by ERC Advanced Grant “FEEL” (H2020 European Research Council), number 323674 to J. Kevin O’Regan and in part by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - number 395696283 to Frank Schumann and by FrontCog grant ANR-17-EURE-0017.

ORCID iD

Christoph Witzel https://orcid.org/0000-0001-9944-2420

**Supplemental Material**

Supplemental material for this article is available online.

References

Arnold, G., Pesnot-Lerousseau, J., & Auvray, M. (2017). Individual differences in sensory substitution. *Multisensory Research*, *30*(6), 579–600. doi:https://doi.org/10.1163/22134808-00002561.

Ashby, F. G., & Crossley, M. J. (2012). Automaticity and multiple
memory systems. *Wiley Interdisciplinary Reviews: Cognitive
Science*, *3*(3), 363–376. doi:https://doi.org/10.1002/wcs.1172.

Augustinova, M., & Ferrand, L. (2014). Automaticity of word reading: Evidence from the semantic stroop paradigm. *Current Directions in Psychological Science*, *23*(5), 343–348. doi:https://doi.org/10.1177/0963721414540169.

Auvray, M., Hanneton, S., Lenay, C., & O’Regan, J. K. (2005). There is something out there: Distal attribution in sensory substitution, twenty years later. *Journal of Integr Neurosci*, *4*(4), 505–521. doi:https://doi.org/10.1142/s0219635205001002.

Auvray, M., Hanneton, S., & O'Regan, J. K. (2007a). Learning to perceive with a visuo-auditory substitution system: Localisation and object recognition with 'the vOICe'. *Perception*, *36*(3), 416–430. doi:https://doi.org/10.1068/p5631.

Auvray, M., & Harris, L. R. (2014). The state of the art of sensory substitution. *Multisensory Research*, *27*(5-6), 265–269. doi: https://doi.org/10.1163/22134808-00002464.

Auvray, M., & Myin, E. (2009). Perception with compensatory devices: From sensory substitution to sensorimotor extension. *Cognitive Science*, *33*(6), 1036–1058. doi:https://doi.org/10.1111/j.1551-6709.2009.01040.x.

Auvray, M., Philipona, D. L., O'Regan, J. K., & Spence, C. (2007b). The perception of space and form recognition in a simulated environment: The case of minimalist sensory-substitution devices. *Perception*, *36*(12), 1736–1751. doi: https://doi.org/10.1068/p5800.

Bach-y-Rita, P., Collins, C. C., Saunders, F. A., White, B., & Scadden, L. (1969). Vision substitution by tactile image projection. *Nature*, *221*(5184), 963–964. doi:https://doi.org/10.1038/221963a0.
Bach-y-Rita, P., Kaczmarek, K. A., Tyler, M. E., & Garcia-Lara, J. (1998). Form perception with a 49-point electrotactile stimulus array on the tongue: A technical note. *Journal of Rehabilitation Research and Development*, 35(4), 427–430. doi: http://www.ncbi.nlm.nih.gov/pubmed/10220221.

Bach-y-Rita, P., & Kercel, S. W. (2003). Sensory substitution and the human-machine interface. *Trends in Cognitive Sciences*, 7(12), 541–546. doi: https://doi.org/10.1016/j.tics.2003.10.013.

Barry, S. R. (2010). Can you acquire a sense of direction? With a compass hat, I developed a sense of direction. *Psychology Today, Blog: Eyes on the Brain*. https://www.psychologytoday.com/blog/eyes-the-brain/201009/can-you-acquire-sense-direction

Bermejo, F., Di Paolo, E. A., Hug, M. X., & Arias, C. (2015). Sensorimotor strategies for recognizing geometrical shapes: A comparative study with different sensory substitution devices. *Frontiers in Psychology, 6*, 679. doi: https://doi.org/10.3389/fpsyg.2015.00679.

Bertram, C., & Stafford, T. (2016). Improving training for sensory augmentation using the science of expertise. *Neuroscience and Biobehavioral Reviews*, 68, 234–244. doi: https://doi.org/10.1016/j.neubiorev.2016.05.026.

Blackmore, S. J. (2001). Three experiments to test the sensorimotor theory of vision. *The Behavioral and Brain Sciences*, 24(5), 977–977. doi: https://doi.org/10.1017/S0140525X01260117.

Block, N. (2003). Tactile sensation via spatial perception. *Trends in Cognitive Sciences*, 7(7), 285–286. doi: https://doi.org/10.1016/S1364-6613(03)00132-3

Bologna, G., Deville, B., & Pun, T. (2009). On the use of the auditory pathway to represent image scenes in real-time. *Neurocomputing*, 72(4-6), 839–849. doi: https://doi.org/10.1016/j.neucom.2008.06.020.

Bompas, A., & O'Regan, J. K. (2006a). Evidence for a role of action in colour perception. *Perception*, 35(1), 65–78. doi: https://doi.org/10.1068/p5356.

Bompas, A., & O'Regan, J. K. (2006b). More evidence for sensorimotor adaptation in color perception. *Journal of Vision*, 6(2), 145–153. doi: https://doi.org/10.1167/6.2.5. [pii].

Brown, D., Macpherson, T., & Ward, J. (2011). Seeing with sound? Exploring different characteristics of a visual-to-auditory sensory substitution device. *Perception*, 40(9), 1120–1135. doi: https://doi.org/10.1068/p6952.

Buchs, G., Haimler, B., Kerem, M., Maidenbaum, S., Braun, L., & Amedi, A. (2021). A self-training program for sensory substitution devices. *Plos One*, 16(4), Aeticle e0250281. doi: https://doi.org/10.1371/journal.pone.0250281.

Capelle, C., Trullemans, C., Arno, P., & Veraart, C. (1998). A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution. *IEEE Transactions on Biomedical Engineering*, 45(10), 1279–1293. doi: https://doi.org/10.1109/10.720206.

Cassinelli, A., Reynolds, C., & Ishikawa, M. (2006). Augmenting spatial awareness with haptic radar. Paper presented by the

10th IEEE international symposium on wearable computers, Montreux, Switzerland

Chase, W. G., & Simon, H. A. (1973). Perception in chess. *Cognitive Psychology*, 4(1), 55–81. doi: https://doi.org/10.1016/0010-0285(73)90004-2.

Chebat, D.-R., Maidenbaum, S., & Amedi, A. (2015). Navigation using sensory substitution in real and virtual mazes. *Plos One*, 10(6), Article e0126307. doi: https://doi.org/10.1371/journal.pone.0126307.

Clemenson, G. D., Maselli, A., Fiannaca, A. J., Miller, A., & Gonzalez-Franco, M. (2021). Rethinking GPS navigation: creating cognitive maps through auditory clues. *Sci Rep.* 11(1), 7764. doi: https://doi.org/10.1038/s41598-021-87148-4.

Cohen, J. (2018). Sensory Substitution and Perceptual Emergence. In *Sensory Substitution and Augmentation*. British Academy. doi: https://doi.org/10.5871/bacad/9780197266441.003.0013

Corns, J. (2018). Disambiguating the perceptual assumption. In *Sensory substitution and augmentation*. London, England: British Academy. doi: https://doi.org/10.5871/bacad/9780197266441.003.0004.

Cosgun, A., Sisbot, E. A., & Christensen, H. I. (2014, 23-26 Feb. 2014). Evaluation of rotational and directional vibration patterns on a tactile belt for guiding visually impaired people. Paper presented by the 2014 IEEE Haptics Symposium (HAPTICS), Daejeon, Korea

Cuturi, L. F., Aggius-Vella, E., Campus, C., Parmiggiani, A., & Gori, M. (2016). From science to technology: Orientation and mobility in blind children and adults. *Neuroscience and Biobehavioral Reviews*, 71, 240–251. doi: https://doi.org/10.1016/j.neubiorev.2016.08.019.

Dale, G., Cochrane, A., & Green, C. S. (2021). Individual difference predictors of learning and generalization in perceptual learning. *Attention, Perception, & Psychophysics*, 83(5), 2241–2255. doi: https://doi.org/10.3758/s13414-021-02268-3.

Degenaar, J. (2014). Through the inverting glass: First-person observations on spatial vision and imagery [journal article]. *Phenomenology and the Cognitive Sciences*, 13(2), 373–393. doi: https://doi.org/10.1007/s11097-013-9305-3.

Deroy, O., & Auvray, M. (2012). Reading the world through the skin and ears: A new Perspective on sensory substitution. *Frontiers in Psychology*, 3, 457. doi: https://doi.org/10.3389/fpsyg.2012.00457.

Díaz, A., Barrientos, A., Jacobs, D. M., & Travieso, D. (2012). Action-contingent vibrotactile flow facilitates the detection of ground level obstacles with a partly virtual sensory substitution device. *Human Movement Science*, 31(6), 1571–1584. doi: https://doi.org/10.1016/j.humov.2012.05.006.

Dinsmore, D. L., & Alexander, P. A. (2012). A critical discussion of deep and surface processing: What it means, how it is measured, the role of context, and model specification. *Educational Psychology Review*, 24(4), 499–567. doi: https://doi.org/10.1007/s10648-012-9198-7.

Dokic, J. (2018). The role of noetic feelings in sensory substitution. In *Sensory substitution and augmentation*. London, England:
British Academy. doi:https://doi.org/10.5871/bacad/9780197266441.003.0017.

Durette, B., Louveton, N., Alleysson, D., & Hérault, J. (2008). Visuo-auditory sensory substitution for mobility assistance: Testing TheVIBE. Paper presented at the Workshop on Computer Vision Applications for the Visually Impaired, Marseille, France.

Einhauser, W., Schumann, F., Bardins, S., Bartl, K., Boning, G., Schneider, E., & Konig, P. (2007). Human eye-head co-ordination in natural exploration. *Network-Computation in Neural Systems, 18*(3), 267–297. doi:https://doi.org/10.1080/09548980701671094.

Elli, G. V., Benetti, S., & Collignon, O. (2014). Is there a future for sensory substitution outside academic laboratories? *Multi-sensory Research, 27*(5–6), 271–291. doi:https://doi.org/10.1163/22134808-00002460.

Elliott, L. R., Erp, J. v., Redden, E. S., & Duistermaat, M. (2010). Field-based validation of a tactile navigation device. *IEEE Transactions on Haptics, 3*(2), 78–87. doi:https://doi.org/10.1109/TOH.2010.3.

Faugloire, E., & Lejeune, L. (2014). Evaluation of heading performance with vibrotactile guidance: The benefits of information-movement coupling compared with spatial language. *Journal of Experimental Psychology: Applied, 20*(4), 397–410. doi:https://doi.org/10.1037/xap0000032.

Favela, L. H., Riley, M. A., Shockley, K., & Chemero, A. (2018). Perceptually equivalent judgments made visually and via haptic sensory-substitution devices. *Ecological Psychology, 30*(4), 326–345. doi:https://doi.org/10.1080/10407413.2018.1473712.

Friston, K. (2018). Does predictive coding have a future? *Nature Neuroscience, 21*(8), 1019–1021. doi:https://doi.org/10.1038/s41593-018-0200-7.

Froese, T., McGann, M., Bigge, W., Spiers, A., & Seth, A. K. (2012). The enactive torch: A new tool for the science of perception. *IEEE Transactions on Haptics, 5*(4), 365–375. doi:https://doi.org/10.1109/TOH.2011.57.

Geisler, W. S. (2011). Contributions of ideal observer theory to vision research. *Vision Research, 51*(7), 771–781. doi:https://doi.org/10.1016/j.visres.2010.09.027.

Gibson, J. J. (1977). The theory of affordances. In R. Shaw & J. Bransford (Eds.), *Perceiving, acting, and knowing: Toward an ecological psychology* (Vol. 1, pp. 67–82). Mahwah, NJ: Lawrence Erlbaum.

Goeke, C. M., Planera, S., Finger, H., & König, P. (2016). Bayesian alternation during tactile augmentation [Original Research]. *Frontiers in Behavioral Neuroscience, 10*(187). doi:https://doi.org/10.3389/fnbeh.2016.00187.

Goldstein, E. B., & Brockmole, J. R. (2017). *Sensation and perception* (10 ed.). Boston, MA: Boston Cengage learning.

Gori, M., Cappagli, G., Tonelli, A., Baud-Bovy, G., & Finocchietti, S. (2016). Devices for visually impaired people: High

technological devices with low user acceptance and no adaptability for children. *Neuroscience and Biobehavioral Reviews, 69*, 79–88. doi:https://doi.org/10.1016/j.neubiorev.2016.06.043.

Greeno, J. G. (1994). Gibson's affordances. *Psychological Review, 101*, 336–342. doi:https://doi.org/10.1037/0033-295X.101.2.336

Guarniero, G. (1974). Experience of tactile vision. *Perception, 3*(1), 101–104. doi:https://doi.org/10.1068/p030101.

Guth, D. A., & LaDuke, R. (1994). The veering tendency of blind pedestrians: An analysis of the problem and literature review. *Journal of Visual Impairment and Blindness, 88*, 391–400.

Haigh, A., Brown, D. J., Meijer, P. B. L., & Proulx, M. J. (2013). How well do you see what you hear? The acuity of visual-to-auditory sensory substitution. *Frontiers in Psychology, 4*, 330. doi:https://doi.org/10.3389/fpsyg.2013.00330.

Hamilton-Fletcher, G., & Ward, J. (2013). Representing colour through hearing and touch in sensory substitution devices. *Multisensory Research, 26*(6), 503–532. doi:https://doi.org/10.1163/22134808-00002434.

Harris, M. (2013). The way through the flames. *Spectrum, IEEE, 50*(9), 30-35. doi:https://doi.org/10.1109/MSPEC.2013.6587186.

Hartcher-O'Brien, J., & Auvray, M. (2014). The process of distal attribution illuminated through studies of sensory substitution. *Multisensory Research, 27*(5-6), 421-441. doi:https://doi.org/10.1163/22134808-00002456.

Helbig, H. B., & Ernst, M. O. (2008). Visual-haptic cue weighting is independent of modality-specific attention. *Journal of Vision, 8*(1), 21. doi:https://doi.org/10.1167/8.1.21.

Heller, F., Knott, T., Weiss, M., & Borchers, J. (2009). Multi-user interaction in virtual audio spaces. Paper presented at the CHI '09 Extended Abstracts on Human Factors in Computing Systems, Boston, MA.

Hillis, J. M., Ernst, M. O., Banks, M. S., & Landy, M. S. (2002). Combining sensory information: Mandatory fusion within, but not between, senses. *Science, 298*(5598), 1627–1630. doi: https://doi.org/10.1126/science.1075396.

Hurley, S., & Noë, A. (2003). Neural plasticity and consciousness. *Biology and Philosophy, 18*(1), 131–168. doi:https://doi.org/10.1023/A:1023308401356.

Hutchison, K. A., Balota, D. A., Neely, J. H., Cortese, M. J., Cohen-Shikora, E. R., Tse, C.-S., Yap, M. J., Bengson, J. J., Niemeyer, D., & Buchanan, E. (2013). The semantic priming project. *Behavior Research Methods, 45*(4), 1099–1114. doi: https://doi.org/10.3758/s13428-012-0304-z.

Isler, J. J., & Isler, P. (2006). *Sailing for dummies* (2 ed.). Hoboken, NJ: Wiley.

Johnson, L. A., & Higgins, C. M. (2006). A navigation aid for the blind using tactile-visual sensory substitution. *IEEE Annual Conferences of the Engineering in Medicine and Biology Society, 1*, 6289–6292. doi:https://doi.org/10.1109/IEMBS.2006.259473.

Kärcher, S. M., Fenzlaff, S., Hartmann, D., Nagel, S. K., & König, P. (2012). Sensory augmentation for the blind. *Frontiers in
*Human Neuroscience*, **6**, 37. doi:https://doi.org/10.3389/fnhum.2012.00037.

Kaspar, K., König, S., Schwandt, J., & König, P. (2014). The experience of new sensorimotor contingencies by sensory augmentation. *Consciousness and Cognition*, **28**, 47–63. doi: https://doi.org/10.1016/j.concog.2014.06.006.

Katz, B., Jones, M. R., Shah, P., Buschkuehl, M., & Jaeggi, S. M. (2021). Individual differences in cognitive training research. In T. Strobach & J. Karbach (Eds.), *Cognitive training: An overview of features and applications* (pp. 107–123). Berlin, Germany: Springer International Publishing. doi:https://doi.org/10.1007/978-3-030-39292-5_8.

Kerdegari, H., Kim, Y., & Prescott, T. J. (2016). Head-mounted sensory augmentation device: Designing a tactile language. *IEEE Transactions on Haptics*, **9**(3), 376–386. doi:https://doi.org/10.1109/TOH.2016.2554111.

Kojima, Y., Hashimoto, Y., Fukushima, S., & Kajimoto, H. (2009). Pull-navi: A novel tactile navigation interface by pulling the ears. SIGGRAPH '09 ACM SIGGRAPH 2009 Emerging Technologies Article No. 19.

König, S. U., Schumann, F., Keyser, J., Goeke, C., Krause, C., Wache, S., & König, P. (2016). Learning new sensorimotor contingencies: Effects of long-term use of sensory augmentation on the brain and conscious perception. *Plos One*, **11**(12), Article e0166647. doi:https://doi.org/10.1371/journal.pone.0166647.

Kupers, R., & Ptito, M. (2004). “Seeing” through the tongue: Cross-modal plasticity in the congenitally blind. *International Congress Series*, **1270**, 79–84. doi:https://doi.org/10.1016/j.ics.2004.04.053.

Lee, T. D., Swinnen, S. P., & Serrien, D. J. (1994). Cognitive effort and motor learning. *Quest*, **46**(3), 328–344. doi:https://doi.org/10.1080/00336297.1994.10484130.

Lenay, C., Gapenne, O., Hanneton, S., Marque, C., & Genouëlle, C. (2003). Sensory substitution - limits and perspectives. In Y. Hatwell, A. Streri, & E. Gentaz (Eds.), *Touching for Knowing: Cognitive psychology of haptic manual perception* (pp. 275–292). Amsterdam, Netherlands: John Benjamin Publishing Company. doi:https://doi.org/10.1075/aicr.53.22len.

Lenay, C., & Steiner, P. (2010). Beyond the internalism/externalism debate: The constitution of the space of perception. *Consciousness and Cognition*, **19**(4), 938–952. doi: https://doi.org/10.1016/j.concog.2010.06.011.

Lenay, C., & Stewart, J. (2012). Minimalist approach to perceptual interactions [Methods]. *Frontiers in Human Neuroscience*, **6**, doi:https://doi.org/10.3389/fnhum.2012.00098.

Lillicrap, T. P., Moreno-Briseño, P., Diaz, R., Tweed, D. B., Troje, N. F., & Fernandez-Ruiz, J. (2013). Adapting to inversion of the visual field: A new twist on an old problem. *Experimental Brain Research*, **228**(3), 327–339. doi:https://doi.org/10.1007/s00221-013-3565-6.

Lobo, L., Travieso, D., Barrientos, A., & Jacobs, D. M. (2014). Stepping on obstacles with a sensory substitution device on

the lower leg: Practice without vision is more beneficial than practice with vision. *Plos One*, **9**(6), Article e98801. doi: https://doi.org/10.1371/journal.pone.0098801.

Macpherson, F. (2018). Sensory substitution and augmentation: An introduction. In *Sensory substitution and augmentation*. London, England: British Academy. doi:https://doi.org/10.5871/bacad/9780197266441.003.0001.

Maidenbaum, S., Abboud, S., & Amedi, A. (2014a). Sensory substitution: Closing the gap between basic research and widespread practical visual rehabilitation. *Neuroscience and Biobehavioral Reviews*, **41**, 3–15. doi:https://doi.org/10.1016/j.neubiorev.2013.11.007.

Maidenbaum, S., Hanassy, S., Abboud, S., Buchs, G., Chebat, D. R., Levy-Tzedek, S., & Amedi, A. (2014b). The "EyeCane", a new electronic travel aid for the blind: Technology, behavior & swift learning. *Restorative Neurology and Neuroscience*, **32**(6), 813–824. doi:https://doi.org/10.3233/RNN-130351.

Marcel, A. (2003). Introspective report: Trust, self-knowledge and science. *Journal of Consciousness Studies*, **10**(9–10), 167–186. http://www.ingentaconnect.com/content/imp/jcs/2003/00000010/F0020009/art00013

Marr, D. (2010 [1982]). *Vision: A computational investigation into the human representation and processing of visual information*. Cambridge, MA: MIT Press.

Meijer, P. B. L. (1992). An experimental system for auditory image representations. *IEEE Transactions on Biological Engineering*, **39**(2), 112–121. doi:https://doi.org/10.1109/10.121642.

Meijer, P. B. L. (2015). *Seeing with your ears! the vOICe*. Retrieved Nov. 08 from https://www.seeingwithsound.com/

Moors, A. (2015). Automaticity: Componential, causal, and mechanistic explanations. *Annual Review of Psychology*. doi: https://doi.org/10.1146/annurev-psych-122414-033550.

Myin, E., & Degenaar, J. (2014). Enactive vision. In L. Shapiro (Ed.), *Routledge handbook of embodied cognition* (pp. 90–98). London, England: Routledge.

Nagel, S. K., Carl, C., Kringe, T., Martin, R., & König, P. (2005). Beyond sensory substitution--learning the sixth sense. *Journal of Neural Engineering*, **2**(4), R13–R26. doi:https://doi.org/10.1088/1741-2560/2/4/R02.

Negen, J., Bird, L.-A., Slater, H., Thaler, L., & Nardini, M. (2021). A new sensory skill shows automaticity and integration features in multisensory interactions. *bioRxiv*, 2021. doi: https://doi.org/10.1101/2021.01.05.425430.

Noë, A. (2004). *Action in perception*. Cambridge, MA: MIT Press.

Noordhof, P. (2018). Sensory substitution and the challenge of acclimatization. In *Sensory substitution and augmentation*. London, England: British Academy. doi:https://doi.org/10.5871/bacad/9780197266441.003.0005.

Norimoto, H., & Ikegaya, Y. (2015). Visual cortical prosthesis with a geomagnetic compass restores spatial navigation in blind rats. *Current Biology*, **25**(8), 1091–1095. doi:https://doi.org/10.1016/j.cub.2015.02.063.
O'Regan, J. K. (2011). *Why red doesn't sound like a bell - Understanding the feel of consciousness*. Oxford, UK: Oxford University Press.

O'Regan, J. K., & Noë, A. (2001). A sensorimotor account of vision and visual consciousness. *Behavioural and Brain Sciences*, 24(5), 939-973; discussion 973-1031. https://doi.org/10.1017/s0140525x01000115

Palmeri, T. J. (2003). Automaticity. In L. Nadel (Ed.), *Encyclopedia of cognitive science* (Vol. 1, pp. 290–301). Berlin, Germany: Nature Publishing Group.

Parise, C. V., & Ernst, M. O. (2016). Correlation detection as a general mechanism for multisensory integration. *Nature Communications*, 7(1), 11543. doi:https://doi.org/10.1038/ncomms11543.

Pasqualotto, A., Spiller, M. J., Jansari, A. S., & Proulx, M. J. (2013). Visual experience facilitates allocentric spatial representation. *Behavioural Brain Research*, 236(1), 175-179. doi:https://doi.org/10.1016/j.bbr.2012.08.042.

Pielot, M., & Boll, S. (2010). Tactile Wayfinder: Comparison of tactile waypoint navigation with commercial pedestrian navigation systems. In P. Floréen, A. Krüger, & M. Spasojevic (Eds.), *Pervasive computing* (Vol. 6030, pp. 76-93). Berlin, Germany: Springer Berlin Heidelberg. doi:https://doi.org/10.1007/978-3-642-12654-3_5.

Poirier, C., De Volder, A., Tranduy, D., & Scheiber, C. (2007a). Pattern recognition using a device substituting audition for vision in blindfolded sighted subjects. *Neuropsychologia*, 45(5), 1108-1121. doi:https://doi.org/10.1016/j.neuropsychologia.2006.09.018.

Poirier, C., De Volder, A. G., & Scheiber, C. (2007b). What neuroimaging tells us about sensory substitution. *Neuroscience Biobehavioral Review*, 31(7), 1064-1070. doi:https://doi.org/10.1016/j.neubiorev.2007.05.010.

Potter, M. C., Wyble, B., Hagmann, C. E., & McCourt, E. S. (2014). Detecting meaning in RSVP at 13 ms per picture. *Attention Perception & Psychophysics*, 76(2), 270-279. doi: https://doi.org/10.3758/s13414-013-0605-z.

Proulx, M. J. (2010). Synthetic synaesthesia and sensory substitution. *Consciousness and Cognition*, 19(1), 501-503. doi: https://doi.org/10.1016/j.concog.2009.12.005.

Proulx, M. J., Gwinnutt, J., Dell'Erba, S., Levy-Tzedek, S., de Sousa, A. A., & Brown, D. J. (2015). Other ways of seeing: From behavior to neural mechanisms in the online "visual" control of action with sensory substitution. *Restorative Neurology and Neuroscience*, 34(1), 29-44. doi:https://doi.org/10.3233/RNN-150541.

Proulx, M. J., Stoerig, P., Ludowig, E., & Knoll, I. (2008). Seeing 'where' through the ears: Effects of learning-by-doing and long-term sensory deprivation on localization based on image-to-sound substitution. *Plos One*, 3(3), Article e1840. doi:https://doi.org/10.1371/journal.pone.0001840.

Purves, D., Lotto, R. B., Williams, S. M., Nundy, S., & Yang, Z. (2001). Why we see things the way we do: Evidence for a wholly empirical strategy of vision. *Philosophical

Transactions of the Royal Society B: Biological Sciences, 356(1407), 285-297. doi:https://doi.org/10.1098/rstb.2000.0772.

Purves, D., Morgenstern, Y., & Wojtach, W. T. (2015). Perception and reality: Why a wholly empirical paradigm is needed to understand vision [Hypothesis and Theory]. *Frontiers in Systems Neuroscience*, 9. doi:https://doi.org/10.3389/fnsys.2015.00156.

Renier, L. (2018). Sensory substitution: From sensations to phenomenology. In *Sensory substitution and augmentation*. London, England: British Academy. doi:https://doi.org/10.5871/bacad/9780197266441.003.0002.

Renier, L., Bruyer, R., & De Volder, A. G. (2006). Vertical-horizontal illusion present for sighted but not early blind humans using auditory substitution of vision. *Percept Psychophys*, 68(4), 535-542. https://doi.org/10.3758/bf03208756

Renier, L., Laloyaux, C., Collignon, O., Tranduy, D., Vanlierde, A., Bruyer, R., & De Volder, A. G. (2005). The Ponzo illusion with auditory substitution of vision in sighted and early-blind subjects. *Perception*, 34(7), 857-867. doi:https://doi.org/10.1068/p5219

Richler, J. J., Wong, Y. K., & Gauthier, I. (2011). Perceptual expertise as a shift from strategic interference to automatic holistic processing. *Current Directions in Psychological Science*, 20(2), 129-134. doi:https://doi.org/10.1177/0963721411402472.

Roembke, T. C., Hazeltine, E., Reed, D. K., & McMurray, B. (2021). Automaticity as an independent trait in predicting reading outcomes in middle-school. *Developmental Psychology*, 57(3), 361-375. doi:https://doi.org/10.1037/dev0001153.

Royet, J.-P., Koenig, O., Paugam-Moisy, H., Puzenat, D., & Chasse, J.-L. (2004). Levels-of-processing effects on a task of olfactory naming. *Perceptual and Motor Skills*, 98(1), 197-213. doi:https://doi.org/10.2466/pms.98.1.197-213.

Sandlin, D. (2015). The backwards brain bicycle: Un-doing understanding [Video]. https://www.youtube.com/watch?v=MFzDaBzBILO

Schumann, F., Einhauser-Treyer, W., Vockeroth, J., Bartl, K., Schneider, E., & König, P. (2008). Salient features in gaze-aligned recordings of human visual input during free exploration of natural environments. *Journal of Vision*, 8(14), 1211-1217. doi:https://doi.org/10.1167/8.14.12.

Schumann, F., & O'Regan, J. K. (2017). Sensory augmentation: Integration of an auditory compass signal into human perception of space. *Scientific Reports*, 7, 42197. doi:https://doi.org/10.1038/srep42197.

Segond, H., Weiss, D., Kawalec, M., & Sampaio, E. (2013). Perceiving space and optical cues via a visuo-tactile sensory substitution system: A methodological approach for training of blind subjects for navigation. *Perception*, 42(5), 508-528. doi:https://doi.org/10.1068/p6339
Segond, H., Weiss, D., & Sampaio, E. (2005). Human spatial navigation via a visuo-tactile sensory substitution system. *Perception*, 34(10), 1231–1249. https://doi.org/10.1068/p3409

Seth, A. K. (2014). A predictive processing theory of sensorimotor contingencies: Explaining the puzzle of perceptual presence and its absence in synesthesia. *Cognitive Neuroscience*, 5(2), 97–118. doi:https://doi.org/10.1080/17588928.2013.877880.

Shiffrin, R. M., & Schneider, W. (1977). Controlled and automatic human information processing: II. Perceptual learning, automatic attending, and a general theory. *Psychological Review*, 84, 127–190.

Siegle, J. H., & Warren, W. H. (2010). Distal attribution and distance perception in sensory substitution. *Perception*, 39(2), 208–223. doi:https://doi.org/10.1068/p6366.

Smith, B. C. (2018). Sensory substitution and the transparency of visual experience: Comments on laurent renier. In *Sensory substitution and augmentation*. London, England: British Academy. doi:https://doi.org/10.5871/bacad/9780197266441.003.0003.

Snowden, R., Thompson, P., & Troscianko, T. S. (2006). *Basic Vision - an introduction to visual perception*. Oxford, UK: Oxford University Press.

Sohl-Dickstein, J., Teng, S., Gaub, B. M., Rodgers, C. C., Li, C., DeWeese, M. R., & Harper, N. S. (2015). A device for human ultrasonic echolocation. *IEEE Transactions on Biomedical Engineering*.

Spence, C. (2018). Sensory substitution: Unfulfilled promises and fundamental limitations. In *Sensory substitution and augmentation*. London, England: British Academy. doi:https://doi.org/10.5871/bacad/9780197266441.003.0015.

Stevens, S. S. (1957). On the psychophysical law. *Psychological Review*, 64(3), 153–181. https://doi.org/10.1037/h0046162

Stiles, N. R. B., & Shimojo, S. (2015). Auditory sensory substitution is intuitive and automatic with texture stimuli. *Scientific Reports*, 5, 15628. doi:https://doi.org/10.1038/srep15628.

Stoll, C., Palluel-Germain, R., Fristot, V., Pellerin, D., Alleysson, D., & Graff, C. (2015). Navigating from a depth image converted into sound. *Applied Bionics and Biomechanics*, 2015, 9, Article 543492. doi:https://doi.org/10.1155/2015/543492.

Stroop, J. R. (1935). Studies of interference in serial verbal reactions. *Journal of Experimental Psychology*, 18, 643–662.

Traut, H. J., Guild, R. M., & Munakata, Y. (2021). Why does cognitive training yield inconsistent benefits? A meta-analysis of individual differences in baseline cognitive abilities and training outcomes [original Research]. *Frontiers in Psychology*, 12. doi:https://doi.org/10.3389/fpsyg.2021.662139.

Travieso, D., Gómez-Jordana, L., Díaz, A., Lobo, L., & Jacobs, D. M. (2015). Body-scaled affordances in sensory

substitution. *Consciousness and Cognition*, 38, 130–138. doi: https://doi.org/10.1016/j.concog.2015.10.009.

Vance, J., & Stokes, D. (2017). Noise, uncertainty, and interest: Predictive coding and cognitive penetration. *Consciousness and Cognition*, 47, 86–98. doi:https://doi.org/10.1016/j.concog.2016.06.007.

Van Erp, J. B. F., & Werkhoven, P. J. (2006). Validation of principles for tactile navigation displays 50th annual meeting of the human factors and ergonomics society meeting, San Francisco, CA. Santa Monica.

Vazquez-Alvarez, Y., Oakley, I., & Brewster, S. A. (2011). Auditory display design for exploration in mobile audio-augmented reality [journal article]. *Personal and Ubiquitous Computing*, 16(8), 987-999. doi:https://doi.org/10.1007/s00779-011-0459-0.

Visell, Y. (2008). Tactile sensory substitution: Models for enaction in HCI. *Interacting with Computers*, 21(1-2), 38–53. doi: https://doi.org/10.1016/j.intcom.2008.08.004.

Ward, J., & Meijer, P. B. L. (2010). Visual experiences in the blind induced by an auditory sensory substitution device. *Consciousness and Cognition*, 19(1), 492–500. doi:https://doi.org/10.1016/j.concog.2009.10.006.

Ward, J., & Wright, T. (2014). Sensory substitution as an artificially acquired synaesthesia. *Neuroscience and Biobehavioral Reviews*, 41, 26–35. doi:https://doi.org/10.1016/j.neubiorev.2012.07.007.

Wen, W., Ishii, H., Ohata, R., Yamashita, A., Asama, H., & Imamizu, H. (2021). Perception and control: Individual difference in the sense of agency is associated with learnability in sensorimotor adaptation. *Scientific Reports*, 11(1), 20542. doi:https://doi.org/10.1038/s41598-021-99969-4.

White, B., Saunders, F., Scadden, L., Bach-Y-Rita, P., & Collins, C. (1970). Seeing with the skin. *Perception & Psychophysics*, 7(1), 23–27. doi:https://doi.org/10.3758/BF03210126.

Wickens, C. (2021). Attention: Theory, principles, models and applications. *International Journal of Human-Computer Interaction*, 37(5), 403–417. doi:https://doi.org/10.1080/10447318.2021.1874741.

Witzel, C., & Gegenfurtner, K. R. (2018). Color Perception: Objects, constancy, and categories. *Annual Review of Vision Science*, 4(1), 475–499. doi:https://doi.org/10.1146/annurev-vision-091517-034231.

Witzel, C., Lübbert, A., J. Kevin, O. R., Hanneton, S., & Schumann, F. (2022). Data for “Can perception be extended to a “feel of north”? Tests of automaticity with the NaviEar”. doi:https://doi.org/10.5281/zenodo.7185537.

YOST Labs. (1998-2015). YEI 3-space Bluetooth Retrieved from https://www.yostlabs.com/productdisplay/3-space-bluetooth#overview
## About the Authors

<figure><img src="image_17.png" /></figure>

**Christoph Witzel** obtained his PhD at the Justus-Liebig-University in Gießen, Germany, on the role of language and experience in colour perception. He is more broadly interested in how culture and experience shape automatic assumptions to form our subjective reality. At the time of publication, Christoph is lecturer in psychology at the University of Southampton. See lab website.

<figure><img src="image_18.png" /></figure>

**Annika Lübbert** is about to complete her PhD at the department of neurophysiology, UKE, Hamburg. Her work investigates social cognition as an embodied and enacted phenomenon, using qualitative and quantitative methods to generate multi-channel observations of interactive behaviour and experience. In collaborative initiatives with fellow researchers, Annika further works to change the environments in which science takes place - see her work on *Playful Academics* and *Mindful Researchers*.

<figure><img src="image_19.png" /></figure>

**Kevin O'Regan** is emeritus ex-director of the Laboratoire Psychologie de la Perception, CNRS, Université Paris Descartes. After working on eye movements in reading he became interested in visual stability and discovered the phenomenon of change blindness. His current work concerns the sensorimotor approach to phenomenal consciousness and its applications to child development and robotics. See http://whatfeelingislike.net and http://kevin-oregan.net.

<figure><img src="image_20.png" /></figure>

**Sylvain Hanneton** studied at the Ecole Normale Supérieure (Paris) from 1989 to 1993 and received a Ph.D. in Biomathematics from Pierre et Marie Curie University in 1998. Since 1999, he is Assistant Professor at the University of Paris Cité (Department of Sport and Human Movement Sciences). He is also deputy director of the Sport Sciences & Health Institute of Paris (EA 3625, University of Paris Cité). His research interests include learning processes in motor control and perception, and modelling of the neural control of movements.

<figure><img src="image_21.png" /></figure>

**Frank Schumann** is currently a postdoctoral researcher working on sensory augmentation at the Laboratoire des Systémes Perceptifs at École Normale Supérieure and the Institut de la Vision at Sorbonne Université in Paris, France. He obtained his doctorate in Cognitive Science at the University of Osnabrück in Germany within the sensorimotor approach to consciousness, including work investigating *how sensorimotor contingencies for compass direction influence the brain and conscious perception*. He was a post doc in two ERC projects looking at consciousness within the sensorimotor and the predictive processing perspectives, 'FEEL' at the Université Paris Descartes, and 'XSPECT' at the University of Edinburgh. With J. Kevin O'Regan, he developed *contingency-mimetics as a novel approach to sensory augmentation*. His research interests further include *improvisation as a foundation of human cognition*, *embodiment as a foundation of mindfulness and human flourishing*, and how to improve the culture of academic practice.