# Looking into Task-Specific Activation Using a Prosthesis Substituting Vision with Audition

Authors: Paula Plaza; Isabel Cuevas; Cécile Grandin; Anne G. De Volder; Laurent Renier
Journal: ISRN Rehabilitation
 (2012)

---

## Abstract

A visual-to-auditory sensory substitution device initially developed for the blind is known to allow visual-like perception through sequential exploratory strategies. Here we used functional magnetic resonance imaging (fMRI) to test whether processing the location versus the orientation of simple (elementary) “visual” stimuli encoded into sounds using the device modulates the brain activity within the dorsal visual stream in the absence of sequential exploration of these stimuli. Location and orientation detection with the device induced a similar recruitment of frontoparietal brain areas in blindfolded sighted subjects as the corresponding tasks using the same stimuli in the same subjects in vision. We observed a similar preference of the right superior parietal lobule for spatial localization over orientation processing in both sensory modalities. This provides evidence that the parietal cortex activation during the use of the prosthesis is task related and further indicates the multisensory recruitment of the dorsal visual pathway in spatial processing.


---

## Body

## 1. Introduction

When deprived of its natural input, the “visual” cortex of early-blind subjects is widely activated during a variety of perceptual and cognitive tasks and its cross-modal recruitment may account for the improved perceptual abilities of these subjects in the auditory and tactile modalities. Sensory substitution prostheses translating vision into touch or vision into audition were developed initially for the blind since these devices could take advantage of the cross-modal plasticity changes induced by blindness. It should be noted, however, that cross-modal interactions are increasingly being recognized as playing a vital role in normal perception too. The visual cortex is functionally divided into a ventral/identification and a dorsal/localization stream [1–4]. Interestingly, using a sensory substitution device (SSD) often activates brain areas within both the dorsal and the ventral visual pathways, not only in the blind but also in normally sighted subjects [5–11]. Despite the increasing number of studies on sensory substitution, little is known about the specific versus nonspecific nature of these brain activations as regards to the task performed with the SSD. One may hypothesize that the brain activations elicited during the use of an SSD are partly driven by mechanisms specific to sensory substitution (i.e., the conversion of sounds or tactile sensations into “visual” representations) but also depend on the nature of the task performed with the SSD (e.g., stimulus identification or localization) [12, 13]. It is difficult to dissociate the brain activations related to the sensory substitution process per se from those specific to the task performed with the device since, to date, very few studies have directly compared the brain activation patterns in the same individuals who performed different tasks on identical stimuli with an SSD [6]. Furthermore, spatial and nonspatial aspects are most often jointly engaged when using an SSD. For instance, using an SSD often requires stimulus exploration strategies and a sequential perception and memorization of each single constituting element of a more complex object (due to the relatively narrow perceptual field of the SSD and its low resolution). All these aspects of SSD use involve working memory and spatial components. In the present study, we used a visual-to-auditory SSD to test whether processing the location versus the orientation of identical “visual” stimuli encoded into sounds modulated the brain activity within the dorsal visual stream in proficient SSD users. In vision, while orientation sensitivity has been reported in both dorsal (e.g., [14] for inferior parietal sulcus; [15] for parieto-occipital junction; [16] for the superior parietal lobule) and ventral streams (e.g., [17–19] for the lateral occipital complex; see also [20]), the localization is properly a spatial task that ought to activate the dorsal stream only. In addition, to reduce the importance of the spatial and working memory components inherent to the use of the SSD and to minimize the brain activation relating to decoding the SSD signal during the tasks, we used very simple stimuli: a horizontal and a vertical bar that were presented separately at the middle top or middle bottom of the SSD perceptual field. Such simple stimuli did not require any exploration strategies or self-generated exploratory movements from the subject, and they could be apprehended globally and almost instantly.


## 2. Materials and Methods

### 2.1. Subjects

Sixteen right-handed male volunteers, without any history of neurological, psychiatric, or sensory impairment participated in the present study (mean age:22.4±1.74 years). Fourteen of these volunteers learnt to use the SSD while the two remaining subjects stayed naive regarding the SSD coding scheme and learnt instead the arbitrary association between each SSD sound and its corresponding visual representation (i.e., a vertical and a horizontal bar placed at different locations in the SSD perceptual field). In addition, two of the fourteen subjects were involved in a separate control experiment that took place after the sensory substitution experiment and that used a “visual” version of the stimuli. All volunteers were blindfolded during all the conditions involving the SSD sounds, and they never saw visually the stimuli before the end of all the experimental conditions involving the SSD. The participants gave their written informed consent prior to the experiment, the protocol of which was approved by the Biomedical Ethics Committee of the School of Medicine of the Université catholique de Louvain.


### 2.2. Sensory Substitution Device

As SSD we used the so-called prosthesis substituting vision with audition (PSVA). Initially developed for the blind, it translates real-time monochrome images from a miniature head-mounted video camera (frame rate: 12.5 Hertz) into sounds using a pixel-frequency relationship [22]. With head movements, visual frames are grabbed in real time and then transformed into a set of corresponding complex sounds enabling recognition [23]. A demonstration is available at http://www.perceptionweb.com/misc.cgi?id=p2607/. Briefly, the camera image is pixelated according to a simplified dual resolution model of the human retina (see Figure 1). This artificial retina consists of a square matrix of 8 × 8 large pixels with the four central ones replaced by 8 × 8 smaller pixels representing the fovea, which has four times the resolution of the periphery. A single sinusoidal tone is assigned to each pixel of the artificial retina with frequencies increasing from left to right and from bottom to top; frequencies range between 50 and 12,526 Hz. The grayscale level of each pixel modulates the amplitude of its corresponding sine wave. The final auditory output of the SSD is the real-time weighted sum of all 124 sine waves that the subject hears through headphones.Figure 1
Schematic representation of the sensory substitution device (SSD) used in the behavioral part of the experiment, that is, the training sessions. A head-mounted video camera (attached to black covered goggles) allows online translation of visual patterns into sounds that are transmitted to the subject through headphones (demonstrations of the sounds produced by a pixel or a bar moving on the artificial retina of the SSD are currently available at thePerceptionWebsite (http://www.perceptionweb.com/perc0899/arno.html)).


### 2.3. Conditions and Stimuli

There were three conditions in the sensory substitution experiment: localization, orientation detection, and an SSD control condition. The control condition aimed at controlling the brain activation related to auditory processing, general attention aspects, and working memory mechanisms that were present in all sensory substitution conditions. Two subjects naive to the SSD code also underwent the same three conditions. The experiment in naive subjects aimed at testing whether an acoustic association elicited similar brain activation as the one observed when subjects processed the information once actually “decoded” according to the SSD code. The three conditions were also adapted to the visual modality for a control experiment in two subjects in vision. The purpose was to provide a reference regarding the neural substrate of stimulus localization and orientation detection in vision and with the same stimuli.The stimuli consisted in a single bar (formed by 3 continuous pixels at the SSD fovea scale) presented either horizontally or vertically at the middle top or the middle bottom of the SSD perceptual field (localization and orientation conditions) or three isolated dots placed randomly in the fovea of the SSD perceptual field (control condition) (see Figure2). It is worth noting that the spatial information was coded using arbitrary differences in frequencies in the 8 × 8 fovea and did not involve any auditory cues that convey naturally information about stimulus location or spatiality, like left/right stereo panning. Therefore, the access to the spatial information could only be done using the SSD code (and not intuitive auditory cues). Stimuli were provided in pairs and presented sequentially one by one with an interstimulus interval of one second. For the SSD experiment, the vertical and horizontal bars were presented successively at the predetermined locations in the SSD perceptual field (without any movement) and the corresponding sounds were then recorded to be delivered later on in the fMRI magnet using headphones (SDS device, the fMRI.pl group, http://www.fmri.pl/). It is worth noting that the sound corresponding to the same horizontal bar (and the same vertical bar) was different if presented at a different location in the SSD perceptual field and the sound corresponding to the horizontal bar was also different from the one of the vertical bar when presented in the same location in the SSD perceptual field (see Figure 3). In the additional visual experiment, the same “visual” stimuli as those encoded into sounds and used in the SSD conditions were backprojected using an MRI compatible projector placed at the rear of the magnet and viewed through a tilted mirror mounted on the head coil (Silent Vision System, Avotec, Inc., http://www.avotec.org). Stimuli were flashed one by one at the middle top or the middle bottom of the screen (experimental conditions), or three isolated square dots were displayed randomly in the half centre of the screen (control condition), with the same time course as in the SSD experiment. A fixation point was displayed at the centre of the screen during inter-stimulus intervals and resting periods. The subjects were instructed to keep their eyes on the fixation point whenever present and to avoid head or eye movements as much as possible throughout the experiment. Although the use of very simple stimuli may not be optimal to probe strong brain activation within the visual cortex, stimuli of a similar level of complexity were successfully used in previous studies on spatial processing in vision [3].Figure 2
Examples of stimuli. The stimuli used in the experiment, that is, a vertical or a horizontal bar, were presented at the middle-upper or middle-lower part of the artificial fovea of the SSD. Examples of pairs of stimuli that were sequentially presented are shown here. The control stimuli were three isolated pixels presented pseudorandomly in the artificial fovea of the SSD.Figure 3
fMRI paradigm (see Section2). We show here the temporal course of a representative run with the representation of some stimuli used in the conditions with the SSD. L: localization task, O: orientation detection task and C: control task. Visual representations and corresponding spectrograms of representative stimuli are displayed in the lower part of the figure in order to illustrate the image-to-sound conversion. On the left, one pair for the experimental condition (localization or orientation), where the sounds correspond to a horizontal bar followed by a vertical bar presented both in the middle-lower part of the artificial retina). On the right, one pair for the control condition, where the sounds correspond to three isolated pixels presented in the SSD artificial retina. The spectrograms below each stimulus are the spectral frequency display (top) and the waveform display (bottom) of 400 msec extracts of the corresponding sounds. In each display, the x-axis (horizontal ruler) measures time (expressed in hms, in a time window of 900 msec, for display purposes). In the spectral frequency display, the y-axis (vertical ruler) measures frequency (expressed in Hz), low-amplitude audio frequencies are in dark color and high-amplitude frequencies are in bright color. In the waveform display, the y-axis (vertical ruler) measures amplitude (the loudness of the audio signal on a decibel scale that ranges from [−Infinity] to zero dBFS).


### 2.4. Task and Procedure

We used a one-back comparison task in all conditions (see Figures3 and 4). During the experimental conditions (localization and orientation detection), the subjects had to determine whether the second element of each pair was the same or different regarding either its location or its orientation. During the SSD control condition, subjects had to determine whether each dot combination was the same or different as compared to the previous one. It is worth noting that unlike the localization and orientation conditions the control condition did not strictly require from the subject to decode the sensory substitution signal, but could rather be done based on a simple acoustic comparison of the pair of sounds. The spectral acoustic complexity of each stimulus was similar across the three conditions (see Figure 3).Figure 4
Illustration of an experimental (localization or orientation detection) and a control block for the visual modality. During an experimental block (left), the subjects had to determine whether the second element of each pair was the same or different regarding either its location or its orientation (in a one-back comparison task). During a control block (on the right), subjects had to determine whether each dot combination was the same or different as compared to the previous one.


### 2.5. Training Sessions

#### 2.5.1. Learning the SSD Code

All subjects were trained to use the SSD in five 1-hour sessions with the protocol adapted from Arno and collaborators [23] and successfully used by Renier and collaborators [24–26]. Subjects were trained to recognize simple and complex 2D figures, formed by vertical, horizontal, and oblique lines. One additional 1-hour session was devoted to the familiarization to the task and stimuli that would be used in the fMRI experiment. In order to ensure a good performance level in the noisy environment of fMRI, a score of 85% of correct answers was required at the end of this training period. Subjects who would not reach this criterion at the end of the last training session or familiarization session would have been discarded from the experiment, which did not happen in the present study. All subjects were blindfolded during all conditions involving the sensory substitution device and never saw visually the stimuli before the end of the experimental conditions involving the SSD.


#### 2.5.2. Learning the Acoustic Association

During three 1-hour sessions, the two subjects naive to the SSD code learnt to associate the sound of each stimulus coded by means of the SSD to its corresponding visual representation (the vertical and the horizontal bars presented at the top or the bottom of the SSD perceptual field). Each sound was presented several times, and subjects had to determine whether the sound corresponded to a horizontal or to a vertical bar or was placed in the upper or in the lower part of the SSD perceptual field. After each trial, the correct response was provided verbally to the subjects. We did not present the stimuli visually to prevent inducing visual memories that would have potentially influenced the brain activations during the subsequent fMRI experiment. At the end of the second training session, the two subjects were able to perform the task with >80% of accuracy. During the third session, subjects were familiarized to the fMRI task (one-back comparison). It is worth noting that these subjects received the same instructions as the other subjects for the fMRI experiment; they had to determine whether the second stimulus of each pair was the same or different than the preceding one regarding either its location or its orientation.


### 2.6. fMRI Paradigm

A block design paradigm was used with conditions alternating with baseline. Each condition was announced verbally at the end of each preceding resting period (i.e., “localization,” “orientation,” “control”). One pair of stimuli was presented in each block. Each stimulus was presented for four seconds with an inter-stimulus interval of one second. Subjects provided their response by pressing the button of one of two response pads, one held in each hand. In order to control any potential effect of the lateralization of brain activation related to the responding hand, we counterbalanced the response buttons across the subjects in the experimental conditions (e.g., right for “same” and left for “different”).


### 2.7. 3D-MRI and fMRI Acquisition

Structural brain imaging was obtained in all subjects in the bicommissural (AC-PC) orientation [21] on a 3 Tesla MRI unit (Achieva, Philips Medical systems) using a 3D fast T1-weighted gradient echo sequence with an inversion prepulse (Turbo field echo (TFE), TR [repetition time] = 9 ms, TE [echo time] = 4.6 ms, flip angle = 8 degree, 150 slices, 1 mm thickness, in plane resolution = 0.81 × 0.95 mm). The field of view was 220 × 197 mm, and the SENSE factor (parallel imaging) was 1.5. We used an 8-channel phased array head coil. Foam pads restrained the head, and eyes were blindfolded throughout the study, except in the visual control experiment.Blood-oxygen-level-dependent (BOLD) fMRI data were acquired using a 2D simple shot T2*-weighted gradient echo-planar imaging (EPI) sequence (TR = 3000 ms, TE = 32 ms) with 44 axial slices (slice thickness = 2.6 mm) in the AC-PC orientation. The matrix was 112 × 112 × 44, and the field of view was 220 mm. In plane resolution was 2.12 mm. The fMRI paradigms consisted in 4 runs of 21 alternating epochs of experimental or control conditions and rest (9 s per active epoch, 3 brain volume repetitions, alternating with 12 s [4 repetitions] resting periods). Each condition was assessed 7 times in a separate run, in counterbalanced order. The warning (the name of the task) was provided at the end of the preceding resting period, and stimulus onset was synchronized with the acquisition of the first slice of each epoch. Subject responses were recorded by key pressing at the end of each epoch, when the SSD sound or images were automatically turned off.


### 2.8. Data Analysis

Data were processed and analyzed using Statistical Parametric Mapping (SPM 2, The Wellcome Department of Imaging Neuroscience, London, UK,http://www.fil.ion.ucl.ac.uk/spm/), implemented in Matlab (MathWorks Inc., Sherborn, MA, USA). The first four volumes of each run were discarded to allow for T1 equilibration. The individual structural (TFE) brain volume was coregistered to the first remaining fMRI volume of the corresponding participant. The 3D structural volume was then spatially normalized into the referential defined by the MRI template supplied by the Montreal Neurological Institute (MNI) based on the atlas of Talairach and Tournoux [21] implemented in SPM. The fMRI data were spatially realigned using a least squares approach to estimate a six-parameter rigid body transformation for each scan (trilinear interpolation [27]) and further spatially normalized in one single step using the normalization parameters derived from the 3D TFE normalization. This procedure resulted in normalized fMRI scans with a cubic voxel size (2 × 2 × 2 mm) for individual and group analysis. Next, a spatial smoothing with a Gaussian kernel of 6 mm (full width at half maximum, FWHM) was applied in order to reduce the residual anatomical and functional variability between participants.Condition-related changes in regional brain activity were estimated for each participant by a general linear model (GLM) in which the responses evoked by each condition of interest were modeled by a standard hemodynamic response function. The contrasts of interest were first computed at the individual level to identify the cerebral regions significantly activated by each condition. To analyze the data from the sensory substitution experiment, significant brain activation patterns in the group were then examined at the whole-brain level in a random-effects analysis (RFX) using one-samplet-tests, with the statistical threshold set at P<0.05, corrected for false discovery rate (FDR) and extending to at least 10 contiguous voxels. Alternatively, a threshold of P<0.001 (uncorrected for multiple comparisons with a voxels cluster size extending to at least 10 contiguous voxels) was used in the condition comparisons depending on the strictness of the contrasts performed. Given the small size of the sample used in the visual and auditory control experiments, significant brain activation patterns averaged from the two tested subjects were estimated in a fixed-effects analysis (FFX), with the same statistical thresholds as in RFX analysis.


## 2.1. Subjects

Sixteen right-handed male volunteers, without any history of neurological, psychiatric, or sensory impairment participated in the present study (mean age:22.4±1.74 years). Fourteen of these volunteers learnt to use the SSD while the two remaining subjects stayed naive regarding the SSD coding scheme and learnt instead the arbitrary association between each SSD sound and its corresponding visual representation (i.e., a vertical and a horizontal bar placed at different locations in the SSD perceptual field). In addition, two of the fourteen subjects were involved in a separate control experiment that took place after the sensory substitution experiment and that used a “visual” version of the stimuli. All volunteers were blindfolded during all the conditions involving the SSD sounds, and they never saw visually the stimuli before the end of all the experimental conditions involving the SSD. The participants gave their written informed consent prior to the experiment, the protocol of which was approved by the Biomedical Ethics Committee of the School of Medicine of the Université catholique de Louvain.


## 2.2. Sensory Substitution Device

As SSD we used the so-called prosthesis substituting vision with audition (PSVA). Initially developed for the blind, it translates real-time monochrome images from a miniature head-mounted video camera (frame rate: 12.5 Hertz) into sounds using a pixel-frequency relationship [22]. With head movements, visual frames are grabbed in real time and then transformed into a set of corresponding complex sounds enabling recognition [23]. A demonstration is available at http://www.perceptionweb.com/misc.cgi?id=p2607/. Briefly, the camera image is pixelated according to a simplified dual resolution model of the human retina (see Figure 1). This artificial retina consists of a square matrix of 8 × 8 large pixels with the four central ones replaced by 8 × 8 smaller pixels representing the fovea, which has four times the resolution of the periphery. A single sinusoidal tone is assigned to each pixel of the artificial retina with frequencies increasing from left to right and from bottom to top; frequencies range between 50 and 12,526 Hz. The grayscale level of each pixel modulates the amplitude of its corresponding sine wave. The final auditory output of the SSD is the real-time weighted sum of all 124 sine waves that the subject hears through headphones.Figure 1
Schematic representation of the sensory substitution device (SSD) used in the behavioral part of the experiment, that is, the training sessions. A head-mounted video camera (attached to black covered goggles) allows online translation of visual patterns into sounds that are transmitted to the subject through headphones (demonstrations of the sounds produced by a pixel or a bar moving on the artificial retina of the SSD are currently available at thePerceptionWebsite (http://www.perceptionweb.com/perc0899/arno.html)).


## 2.3. Conditions and Stimuli

There were three conditions in the sensory substitution experiment: localization, orientation detection, and an SSD control condition. The control condition aimed at controlling the brain activation related to auditory processing, general attention aspects, and working memory mechanisms that were present in all sensory substitution conditions. Two subjects naive to the SSD code also underwent the same three conditions. The experiment in naive subjects aimed at testing whether an acoustic association elicited similar brain activation as the one observed when subjects processed the information once actually “decoded” according to the SSD code. The three conditions were also adapted to the visual modality for a control experiment in two subjects in vision. The purpose was to provide a reference regarding the neural substrate of stimulus localization and orientation detection in vision and with the same stimuli.The stimuli consisted in a single bar (formed by 3 continuous pixels at the SSD fovea scale) presented either horizontally or vertically at the middle top or the middle bottom of the SSD perceptual field (localization and orientation conditions) or three isolated dots placed randomly in the fovea of the SSD perceptual field (control condition) (see Figure2). It is worth noting that the spatial information was coded using arbitrary differences in frequencies in the 8 × 8 fovea and did not involve any auditory cues that convey naturally information about stimulus location or spatiality, like left/right stereo panning. Therefore, the access to the spatial information could only be done using the SSD code (and not intuitive auditory cues). Stimuli were provided in pairs and presented sequentially one by one with an interstimulus interval of one second. For the SSD experiment, the vertical and horizontal bars were presented successively at the predetermined locations in the SSD perceptual field (without any movement) and the corresponding sounds were then recorded to be delivered later on in the fMRI magnet using headphones (SDS device, the fMRI.pl group, http://www.fmri.pl/). It is worth noting that the sound corresponding to the same horizontal bar (and the same vertical bar) was different if presented at a different location in the SSD perceptual field and the sound corresponding to the horizontal bar was also different from the one of the vertical bar when presented in the same location in the SSD perceptual field (see Figure 3). In the additional visual experiment, the same “visual” stimuli as those encoded into sounds and used in the SSD conditions were backprojected using an MRI compatible projector placed at the rear of the magnet and viewed through a tilted mirror mounted on the head coil (Silent Vision System, Avotec, Inc., http://www.avotec.org). Stimuli were flashed one by one at the middle top or the middle bottom of the screen (experimental conditions), or three isolated square dots were displayed randomly in the half centre of the screen (control condition), with the same time course as in the SSD experiment. A fixation point was displayed at the centre of the screen during inter-stimulus intervals and resting periods. The subjects were instructed to keep their eyes on the fixation point whenever present and to avoid head or eye movements as much as possible throughout the experiment. Although the use of very simple stimuli may not be optimal to probe strong brain activation within the visual cortex, stimuli of a similar level of complexity were successfully used in previous studies on spatial processing in vision [3].Figure 2
Examples of stimuli. The stimuli used in the experiment, that is, a vertical or a horizontal bar, were presented at the middle-upper or middle-lower part of the artificial fovea of the SSD. Examples of pairs of stimuli that were sequentially presented are shown here. The control stimuli were three isolated pixels presented pseudorandomly in the artificial fovea of the SSD.Figure 3
fMRI paradigm (see Section2). We show here the temporal course of a representative run with the representation of some stimuli used in the conditions with the SSD. L: localization task, O: orientation detection task and C: control task. Visual representations and corresponding spectrograms of representative stimuli are displayed in the lower part of the figure in order to illustrate the image-to-sound conversion. On the left, one pair for the experimental condition (localization or orientation), where the sounds correspond to a horizontal bar followed by a vertical bar presented both in the middle-lower part of the artificial retina). On the right, one pair for the control condition, where the sounds correspond to three isolated pixels presented in the SSD artificial retina. The spectrograms below each stimulus are the spectral frequency display (top) and the waveform display (bottom) of 400 msec extracts of the corresponding sounds. In each display, the x-axis (horizontal ruler) measures time (expressed in hms, in a time window of 900 msec, for display purposes). In the spectral frequency display, the y-axis (vertical ruler) measures frequency (expressed in Hz), low-amplitude audio frequencies are in dark color and high-amplitude frequencies are in bright color. In the waveform display, the y-axis (vertical ruler) measures amplitude (the loudness of the audio signal on a decibel scale that ranges from [−Infinity] to zero dBFS).


## 2.4. Task and Procedure

We used a one-back comparison task in all conditions (see Figures3 and 4). During the experimental conditions (localization and orientation detection), the subjects had to determine whether the second element of each pair was the same or different regarding either its location or its orientation. During the SSD control condition, subjects had to determine whether each dot combination was the same or different as compared to the previous one. It is worth noting that unlike the localization and orientation conditions the control condition did not strictly require from the subject to decode the sensory substitution signal, but could rather be done based on a simple acoustic comparison of the pair of sounds. The spectral acoustic complexity of each stimulus was similar across the three conditions (see Figure 3).Figure 4
Illustration of an experimental (localization or orientation detection) and a control block for the visual modality. During an experimental block (left), the subjects had to determine whether the second element of each pair was the same or different regarding either its location or its orientation (in a one-back comparison task). During a control block (on the right), subjects had to determine whether each dot combination was the same or different as compared to the previous one.


## 2.5. Training Sessions

### 2.5.1. Learning the SSD Code

All subjects were trained to use the SSD in five 1-hour sessions with the protocol adapted from Arno and collaborators [23] and successfully used by Renier and collaborators [24–26]. Subjects were trained to recognize simple and complex 2D figures, formed by vertical, horizontal, and oblique lines. One additional 1-hour session was devoted to the familiarization to the task and stimuli that would be used in the fMRI experiment. In order to ensure a good performance level in the noisy environment of fMRI, a score of 85% of correct answers was required at the end of this training period. Subjects who would not reach this criterion at the end of the last training session or familiarization session would have been discarded from the experiment, which did not happen in the present study. All subjects were blindfolded during all conditions involving the sensory substitution device and never saw visually the stimuli before the end of the experimental conditions involving the SSD.


### 2.5.2. Learning the Acoustic Association

During three 1-hour sessions, the two subjects naive to the SSD code learnt to associate the sound of each stimulus coded by means of the SSD to its corresponding visual representation (the vertical and the horizontal bars presented at the top or the bottom of the SSD perceptual field). Each sound was presented several times, and subjects had to determine whether the sound corresponded to a horizontal or to a vertical bar or was placed in the upper or in the lower part of the SSD perceptual field. After each trial, the correct response was provided verbally to the subjects. We did not present the stimuli visually to prevent inducing visual memories that would have potentially influenced the brain activations during the subsequent fMRI experiment. At the end of the second training session, the two subjects were able to perform the task with >80% of accuracy. During the third session, subjects were familiarized to the fMRI task (one-back comparison). It is worth noting that these subjects received the same instructions as the other subjects for the fMRI experiment; they had to determine whether the second stimulus of each pair was the same or different than the preceding one regarding either its location or its orientation.


## 2.5.1. Learning the SSD Code

All subjects were trained to use the SSD in five 1-hour sessions with the protocol adapted from Arno and collaborators [23] and successfully used by Renier and collaborators [24–26]. Subjects were trained to recognize simple and complex 2D figures, formed by vertical, horizontal, and oblique lines. One additional 1-hour session was devoted to the familiarization to the task and stimuli that would be used in the fMRI experiment. In order to ensure a good performance level in the noisy environment of fMRI, a score of 85% of correct answers was required at the end of this training period. Subjects who would not reach this criterion at the end of the last training session or familiarization session would have been discarded from the experiment, which did not happen in the present study. All subjects were blindfolded during all conditions involving the sensory substitution device and never saw visually the stimuli before the end of the experimental conditions involving the SSD.


## 2.5.2. Learning the Acoustic Association

During three 1-hour sessions, the two subjects naive to the SSD code learnt to associate the sound of each stimulus coded by means of the SSD to its corresponding visual representation (the vertical and the horizontal bars presented at the top or the bottom of the SSD perceptual field). Each sound was presented several times, and subjects had to determine whether the sound corresponded to a horizontal or to a vertical bar or was placed in the upper or in the lower part of the SSD perceptual field. After each trial, the correct response was provided verbally to the subjects. We did not present the stimuli visually to prevent inducing visual memories that would have potentially influenced the brain activations during the subsequent fMRI experiment. At the end of the second training session, the two subjects were able to perform the task with >80% of accuracy. During the third session, subjects were familiarized to the fMRI task (one-back comparison). It is worth noting that these subjects received the same instructions as the other subjects for the fMRI experiment; they had to determine whether the second stimulus of each pair was the same or different than the preceding one regarding either its location or its orientation.


## 2.6. fMRI Paradigm

A block design paradigm was used with conditions alternating with baseline. Each condition was announced verbally at the end of each preceding resting period (i.e., “localization,” “orientation,” “control”). One pair of stimuli was presented in each block. Each stimulus was presented for four seconds with an inter-stimulus interval of one second. Subjects provided their response by pressing the button of one of two response pads, one held in each hand. In order to control any potential effect of the lateralization of brain activation related to the responding hand, we counterbalanced the response buttons across the subjects in the experimental conditions (e.g., right for “same” and left for “different”).


## 2.7. 3D-MRI and fMRI Acquisition

Structural brain imaging was obtained in all subjects in the bicommissural (AC-PC) orientation [21] on a 3 Tesla MRI unit (Achieva, Philips Medical systems) using a 3D fast T1-weighted gradient echo sequence with an inversion prepulse (Turbo field echo (TFE), TR [repetition time] = 9 ms, TE [echo time] = 4.6 ms, flip angle = 8 degree, 150 slices, 1 mm thickness, in plane resolution = 0.81 × 0.95 mm). The field of view was 220 × 197 mm, and the SENSE factor (parallel imaging) was 1.5. We used an 8-channel phased array head coil. Foam pads restrained the head, and eyes were blindfolded throughout the study, except in the visual control experiment.Blood-oxygen-level-dependent (BOLD) fMRI data were acquired using a 2D simple shot T2*-weighted gradient echo-planar imaging (EPI) sequence (TR = 3000 ms, TE = 32 ms) with 44 axial slices (slice thickness = 2.6 mm) in the AC-PC orientation. The matrix was 112 × 112 × 44, and the field of view was 220 mm. In plane resolution was 2.12 mm. The fMRI paradigms consisted in 4 runs of 21 alternating epochs of experimental or control conditions and rest (9 s per active epoch, 3 brain volume repetitions, alternating with 12 s [4 repetitions] resting periods). Each condition was assessed 7 times in a separate run, in counterbalanced order. The warning (the name of the task) was provided at the end of the preceding resting period, and stimulus onset was synchronized with the acquisition of the first slice of each epoch. Subject responses were recorded by key pressing at the end of each epoch, when the SSD sound or images were automatically turned off.


## 2.8. Data Analysis

Data were processed and analyzed using Statistical Parametric Mapping (SPM 2, The Wellcome Department of Imaging Neuroscience, London, UK,http://www.fil.ion.ucl.ac.uk/spm/), implemented in Matlab (MathWorks Inc., Sherborn, MA, USA). The first four volumes of each run were discarded to allow for T1 equilibration. The individual structural (TFE) brain volume was coregistered to the first remaining fMRI volume of the corresponding participant. The 3D structural volume was then spatially normalized into the referential defined by the MRI template supplied by the Montreal Neurological Institute (MNI) based on the atlas of Talairach and Tournoux [21] implemented in SPM. The fMRI data were spatially realigned using a least squares approach to estimate a six-parameter rigid body transformation for each scan (trilinear interpolation [27]) and further spatially normalized in one single step using the normalization parameters derived from the 3D TFE normalization. This procedure resulted in normalized fMRI scans with a cubic voxel size (2 × 2 × 2 mm) for individual and group analysis. Next, a spatial smoothing with a Gaussian kernel of 6 mm (full width at half maximum, FWHM) was applied in order to reduce the residual anatomical and functional variability between participants.Condition-related changes in regional brain activity were estimated for each participant by a general linear model (GLM) in which the responses evoked by each condition of interest were modeled by a standard hemodynamic response function. The contrasts of interest were first computed at the individual level to identify the cerebral regions significantly activated by each condition. To analyze the data from the sensory substitution experiment, significant brain activation patterns in the group were then examined at the whole-brain level in a random-effects analysis (RFX) using one-samplet-tests, with the statistical threshold set at P<0.05, corrected for false discovery rate (FDR) and extending to at least 10 contiguous voxels. Alternatively, a threshold of P<0.001 (uncorrected for multiple comparisons with a voxels cluster size extending to at least 10 contiguous voxels) was used in the condition comparisons depending on the strictness of the contrasts performed. Given the small size of the sample used in the visual and auditory control experiments, significant brain activation patterns averaged from the two tested subjects were estimated in a fixed-effects analysis (FFX), with the same statistical thresholds as in RFX analysis.


## 3. Results

### 3.1. Behavioral Performance

The behavioral performance during the SSD fMRI experiment was 77.8% (SD = 7.1), 78.1% (SD = 9.4), and 99.5% (SD = 2.3) of correct response for the localization, the orientation detection, and the SSD control condition, respectively. A Studentt-test performed on the scores showed no difference between the localization and the orientation condition (P=0.78), whereas the performance of the control task was significantly better (P<0.001). The mean percentages of correct responses were 80.3, 78.5, and 91.06 in the acoustic association experiment and 91.06, 92.84, and 96.42 in the visual control experiment for the localization, the orientation detection, and the control condition, respectively (all P values > 0.05, although there was a trend towards better scores in the control condition compared to the orientation condition (P=0.09) during the acoustic association experiment only).


### 3.2. fMRI Results

#### 3.2.1. Sensory Substitution Experiment

The one-back comparison of dot combinations used as control condition in SSD experiment, that is, the contrast {control minus rest} showed the expected brain activation in primary and secondary auditory brain areas, bilaterally (Table1). The brain activation pattern also included frontal brain areas such as the superior frontal gyrus bilaterally (BA 9), as well as the insula and the precuneus (BA 7) bilaterally. Additional recruitment of subcortical brain areas was observed in the lentiform nucleus (putamen) bilaterally and the left thalamus (see Figure 5 and Table 1).Table 1
Overview of brain activation foci during the sensory substitution conditions.
Brain areaCluster sizeZ scorex,y,z{mm}Control > restLeft postcentral gyrus (BA 1–3)14665.50−44; −28; 66Left precentral gyrus (BA 4–6)(*)4.73−50; −20; 62Right lentiform nucleus, putamen1924.7824;  2;  6Right insula(*)4.9140; −8; 0Left superior temporal gyrus (BA 22)1204.30−44; −18; −4Left lentiform nucleus, putamen1374.24−22;  0;  4Left claustrum/insula(*)3.66−34;  0;  0Right paracentral lobule/precuneus (BA 5–7)143.8220; −42; 50Left superior frontal gyrus (BA 9)1533.78−20; 46; 32Right middle frontal gyrus (BA 9)383.6464; 12; 34Left thalamus173.47−18; −26; 8Right superior frontal gyrus (BA 9)753.2026; 58; 28Right transverse temporal gyrus (BA 41)113.1634; −26; 12Left precuneus (BA 7)103.15−6; −60; 62Localization > controlRight superior frontal gyrus (BA 8)14595.284; 30; 48Right superior frontal gyrus (BA 6)(*)4.982; 22; 54Left cerebellum6494.85−2; −34; −2Right inferior frontal gyrus (BA 47)4464.8042; 18; −6Left insula (BA 13)4814.80−34; 22; 0Left inferior parietal lobule (BA 40)12454.76−42; −52; 50Left superior parietal lobule (BA 7)(*)4.53−32; −64; 46Left precuneus (BA 7)(*)4.37−28; −50; 50Left middle frontal gyrus (BA 6)584.71−28; 6; 56Right middle frontal gyrus (BA 9)10114.7154; 28; 30Left inferior frontal gyrus (BA 46)5594.47−48; 44; 2Right cerebellum2404.478; −46; −14Left cingulate gyrus (BA 23)1464.32−4; −18; 32Left caudate body333.84−18; −6; 24Right middle frontal gyrus (BA 6)1233.7428; 14; 54Right caudate body293.6110; 2; 10Right superior parietal lobule (BA 7)1373.5934; −64; 50Right precuneus (BA 7)(*)3.2918; −68; 50Right precuneus (BA 7)183.198; −72; 52Right inferior parietal lobule (BA 40)103.1540; −44; 44Orientation > controlRight inferior frontal gyrus (BA 47)1224.8138; 20; −4Left thalamus1374.76−16; −10; 8Left inferior parietal lobule (BA 40)5984.75−40; −50; 46Left superior parietal lobule (BA 7)(*)4.39−32; −48; 48Left superior parietal lobule/precuneus (BA 7)(*)4.08−30; −66; 44Left middle frontal gyrus (BA 9)4864.72−42; 22; 36Left inferior frontal gyrus (BA 45)1894.68−32; 26; 6Right superior frontal gyrus (BA 8)7934.522; 34; 50Left substantia nigra984.38−12; −16; −8Right cerebellum314.146; −48; −20Right middle frontal gyrus (BA 9)453.9244; 2; 40Right middle frontal gyrus (BA 9)723.8858; 28; 32Left middle frontal gyrus (BA 9-10)833.80−42; 48; 20Left inferior frontal gyrus (BA 46)503.61−46; 42; 4Left cingulate gyrus (BA 23)103.76−8; −24; 32Overview of brain activation peaks in the contrasts {SSD control condition minus rest}, {SSD localization minus SSD control, masked (inclusive) with SSD Localization minus rest}, and {SSD orientation minus SSD control, masked (inclusive) with SSD Orientation minus rest} from random-effect analysis (atP<0.05, FDR corrected). Coordinates are reported in MNI space, as given by SPM2, and correspond only approximately to Talairach and Tournoux space [21]. BA: Brodmann area. (*) belonging to the same voxels cluster.Brain activation foci elicited by control (dots comparison) and experimental tasks (localization or orientation processing) through a visual-to-auditory sensory substitution device (SSD) in blindfolded sighted subjects. The statistical parametric maps for these comparisons (in a random-effects analysis in the group) were superimposed on the axial, coronal, and sagittal section of an individual normalized brain MRI. Only voxels exceeding a threshold ofP<0.05 corrected for false discovery rate (FDR) at the whole brain level were displayed according to the color scales that code the T-values. (a) Brain activation observed in control condition, that is, in the contrast {control minus rest}. A bilateral activation in auditory brain areas was evident in this contrast. (b) Brain activation foci observed during the localization condition, that is, in the contrast {localization minus control masked (inclusive) by localization minus rest}. A bilateral activation in frontal brain areas and in the dorsal visual pathway was observed in this contrast. The images were displayed at the level of the inferior parietal lobule (BA 40), which was bilaterally recruited with a main focus in the left hemisphere, with a Z value of 4.76 at coordinates (x,y,z (mm) = −42, −52, 50). (c) Brain activation foci observed during the orientation condition, that is, in the contrast {stimulus orientation minus control masked (inclusive) by stimulus orientation minus rest}. This contrast elicited a left lateralization of brain activation in the dorsal visual pathway, with a main focus in the left inferior parietal lobule (BA 40), with a Z value of 4.75 at coordinates (x,y,z (mm) = −40, −50, 46). A bilateral recruitment of frontal brain areas was also observed. (d) Brain activation foci common to both spatial processing tasks with the SSD, that is, from a conjunction analysis of experimental conditions {localization minus control and orientation detection minus control}. The common neural substrate for localization and orientation processing with the SSD involved frontal brain areas bilaterally, including the inferior, middle, and superior frontal gyri, and parietal structures located in the left hemisphere such as the precuneus and the inferior and superior parietal lobules (see Table 2 for the detailed list of the activation foci). This part of the figure focused on the left inferior parietal lobule, with a Z value of 5.50 at coordinates (x,y,z (mm) = −30, −50, 48). Coordinates are in reference to the Talairach and Tournoux atlas [21] and the MNI template (see Section 2). L: left side of the brain.
(a)
Control (dots) minus rest(b)
Localization minus control(c)
Orientation minus control(d)
Localization conjunction orientationThe localization condition, that is, the contrast {localization minus control masked (inclusive) by localization minus rest} revealed a widespread activated network including the right superior frontal gyrus, the middle and inferior frontal gyri bilaterally, the left and right inferior parietal lobule (BA 40), as well as the left and right superior parietal lobule (BA 7) and precuneus (BA 7) and the cerebellum bilaterally (Table1).The orientation detection, that is, the contrast {orientation minus control masked (inclusive) by orientation minus rest} showed a roughly similar brain activation pattern including the right superior frontal gyrus, the middle and inferior frontal gyri bilaterally and the left inferior parietal lobule (BA 40), superior parietal lobule (BA 7), and precuneus (BA 7) (Table1). The left thalamus and the right cerebellum were also activated.A conjunction analysis between the localization and orientation conditions (minus control) confirmed the equivalent recruitment of the right superior frontal gyrus (BA 6, 8) and the middle and inferior frontal gyri (BA 9, 45, 47) bilaterally, as well as the activation of the precuneus (BA 7) and the superior and inferior parietal lobules (BA 7, 40) in the left hemisphere in both conditions (see Table2).Table 2
Brain activation foci specific to experimental tasks using the sensory substitution device.
Brain areaCluster sizeZ scorex,y,z{mm}Localization∩ orientationRight superior frontal gyrus (BA 8)10955.9810; 28; 50Right superior frontal gyrus (BA 6)(*)5.142; 22; 54Right inferior frontal gyrus (BA 47)2585.6738; 20;  −6Left superior parietal lobule/inferior parietal lobule (BA 7–40)10735.50−30; −50; 48Left superior parietal lobule (BA 7)(*)5.24−40; −54; 50Left precuneus/inferior parietal lobule (BA 7–40)(*)5.18−32; −70; 46Left thalamus9845.34−2; −32;  2Left middle frontal gyrus (BA 9)10145.07−44;  6; 42Right middle frontal gyrus (BA 9)5224.8656; 22; 38Left inferior frontal gyrus (BA 45)3674.36−36; 24;  2Localization > orientationRight superior occipital gyrus/precuneus (BA 19–7)1794.6942; −74; 34Right angular gyrus (BA 39)(*)3.3754; −70; 36Right inferior parietal lobule (BA 40)(*)3.3454; −68; 44Right middle frontal gyrus (BA 6)384.2040;  6;  46Right superior frontal gyrus (BA 6)224.0240; 24; 58Right middle temporal gyrus (BA 21)223.9870; −42;  −8Right middle temporal gyrus (BA 19–39)333.9050; −74; 20Right inferior frontal gyrus (BA 10–46)143.7348; 52;   2Right superior parietal lobule (BA 7)253.7252; −60; 52Right middle occipital gyrus (BA 19)213.6536; −80; 16Right superior parietal lobule (BA 7)113.558; −68; 62Left middle temporal gyrus (BA 19–39)173.38−40; −80; 18Overview of brain activation peaks in the contrasts {(SSD localization minus SSD control) conjunction (SSD orientation minus SSD control)} and {SSD localization minus SSD orientation masked (inclusive) by SSD localization minus rest}. Results are from a random-effect analysis, atP<0.05, FDR corrected, for the conjunction, and P<0.001 uncorrected, for the differential contrast between experimental tasks. Coordinates are reported in MNI space, as given by SPM2, and correspond only approximately to Talairach and Tournoux space [21]. BA: Brodmann area. (*) belonging to the same voxels cluster.The differential contrast between the two experimental conditions, that is, the contrast {localization minus orientation masked (inclusive) by localization minus rest} (at a threshold of uncorrectedP<0.001 in combination with a cluster size threshold of 10 continuous voxels), showed that the localization task tended to be associated with larger activation in frontal, parietal, and occipitoparietal brain areas mainly in the right hemisphere, as compared to the orientation detection (Table 2). These brain activation foci included the right superior, middle, and inferior frontal gyri (BA 6, 10, 46), the right superior occipital gyrus/precuneus (BA 19, 7) and inferior parietal lobule (BA 40), the right superior parietal lobule (BA 7), and the middle temporal gyrus bilaterally (BA 19, 21, 39) (see Figure 6 and Table 2). The inverted contrast {orientation minus localization masked (inclusive) by orientation minus rest} did not reveal any significant activation focus at the selected threshold.Brain activation patterns related to location as contrasted to orientation discrimination, that is, using the contrast {localization minus orientation masked (inclusive) by localization minus rest}, with the SSD and in vision. Positive differences exceeding an uncorrected threshold ofP<0.001 were displayed according to the color scales that code the T-values. The statistical parametric map for each comparison was superimposed on the axial, coronal, and sagittal sections of an individual normalized brain MRI. The brain activation pattern observed with the SSD (a) included brain areas mainly in the right dorsal visual pathway, including the superior occipital gyrus, precuneus, inferior, and superior parietal lobules (see Table 2 for the exhaustive list of the activation foci). The red circle indicates an activation focus in the right superior parietal lobule (x,y,z (mm) = 8, −68, 62; 11 voxels). The corresponding brain activation pattern observed in vision (b) also included brain areas in the dorsal visual pathway with a slight predominance in the right hemisphere (see Table 3). The red circle indicates an activation focus in the right superior parietal lobule (x,y,z (mm) = 26, −66, 66; 86 voxels).
(a)
Localization minus orientation (SSD)(b)
Localization minus orientation (Vision)


#### 3.2.2. Visual Control

We used an FFX analysis in the two subjects involved in this control, with a threshold ofP<0.05 FDR corrected in combination with a cluster size threshold of 10 continuous voxels. The visual control condition as contrasted to baseline was associated with the expected recruitment of a large frontoparietal network, known to be engaged in visuospatial working memory tasks, and including visual association areas (see Table 3). The contrast {localization minus control masked (inclusive) by localization minus rest} disclosed brain activation foci in occipital and parietal areas, included in the superior parietal lobule (BA 7) and the cuneus (BA 17, 18) bilaterally. The contrast {orientation minus control masked (inclusive) by orientation minus rest} did not reveal any significant activation focus at the selected threshold. When we compared the localization condition with the orientation condition, that is, using the contrast {localization minus orientation masked (inclusive) by localization minus rest} (at a threshold of P<0.001, uncorrected, in combination with a cluster size threshold of 10 continuous voxels), brain activation voxels clusters survived in the right superior parietal lobule (BA 7) and, to a lesser extent, in the right and left superior occipital gyrus/precuneus (BA 19, 7). Obviously the reverse contrast {orientation minus localization} did not reveal any activation focus at the selected threshold.Table 3
Brain activation during the visual conditions.
Brain areaCluster sizeZ scorex,y,z{mm}Control > restLeft precuneus/superior parietal lobule (BA 7)4435.98−14; −74; 54Left precuneus/superior parietal lobule (BA 7)(*)4.74−22; −60; 48Right middle frontal gyrus (BA 6–8)1055.8552; 8; 48Right superior frontal gyrus (BA 6)1035.662; 24; 58Right cuneus (BA 17-18)3465.4720; −90; 6Right middle frontal gyrus (BA 9)1165.4250; 28; 32Left inferior occipital gyrus (BA 18)1085.32−28; −94; −4Right precuneus/superior parietal lobule (BA 7)1264.8818; −68; 54Left superior frontal gyrus (BA 8)1064.29−4; 22; 48Left middle frontal gyrus (BA 9)403.78−16; 44; 34Localization > controlRight superior parietal lobule (BA 7)3525.2220; −66; 56Left superior parietal lobule (BA 7)2064.79−28; −60; 52Left precuneus/superior parietal lobule (BA 7)(*)4.62−18; −62; 52Left superior parietal lobule (BA 7)(*)4.24−30; −68; 52Left cuneus (BA 17-18)694.76−6; −96; 8Right cuneus (BA 17)303.9910; −90; 4Orientation > controlLocalization > orientationRight superior parietal lobule (BA 7)864.1226; −66; 66Right superior parietal lobule (BA 7)(*)3.8728; −56; 60Left superior occipital gyrus/precuneus (BA 19–7)414.06−36; −78; 42Right superior occipital gyrus/precuneus (BA 19–7)103.7738; −68; 44Overview of brain activation peaks during the additional visual control experiment.  When experimental conditions were compared to control condition, only the contrast {visual localization minus visual control, masked (inclusive) with visual localization minus rest} provided results atP<0.05, FDR corrected (fixed-effect analysis).  Results from the differential contrast between experimental conditions, that is, {visual localization minus visual orientation masked (inclusive) by visual localization minus rest} are also reported (from a fixed-effect analysis at P<0.001, uncorrected) for reference.  Coordinates are reported in MNI space, as given by SPM2, and correspond only approximately to Talairach and Tournoux space [21]. BA: Brodmann area. (*) belonging to the same voxels cluster.


#### 3.2.3. Acoustic Association

We used an FFX analysis in the two subjects who had learned to associate each sound with its visual representation to estimate the condition-related brain activity changes when based on acoustic association. None of the contrasts performed ({localization minus control}, {orientation minus control}, {localization minus orientation} and {orientation minus localization)} revealed any activation in the parieto-occipital cortex in any of the two subjects (at a threshold ofP<0.001, uncorrected, in combination with a cluster size threshold of 10 continuous voxels). Only the localization task compared to control condition induced some weak activation restricted to the auditory cortex and temporal brain areas (see Table 4).Table 4
Brain activation foci related to the acoustic association conditions.
Brain areaCluster sizeZ scoreX,Y,Z{mm}Control > restRight middle frontal gyrus (BA 9)5365.8252; 10; 36Left middle frontal gyrus (BA 8-9)8545.78−46; 28; 38Right transverse/superior temporal gyrus (BA 41-42)1024.6050; −24; 10Right superior temporal gyrus (BA 22)(*)4.0352; −8; 6Left superior temporal gyrus (BA 22)1454.28−50; −42; 18Left transverse/superior temporal gyrus (BA 41-42)(*)3.76−50; −20; 12Localization > controlLeft insula (BA 13)354.58−42; −18; −2Left superior temporal gyrus (BA 21-22)343.97−44; −30;  0Left transverse/superior temporal gyrus (BA 41-42)(*)3.58−38; −40; 12Right superior temporal gyrus (BA 42–22)463.9646; −38; 20Right superior temporal gyrus (BA 42–22)133.8856; −32; 16Right superior temporal/supramarginal gyrus (BA 22–40)163.4968; −48; 22Left supramarginal gyrus (BA 40)103.49−46; −48; 32Orientation > controlLocalization > orientationOverview of brain activation peaks during the acoustic association control experiment. When experimental conditions were compared to control condition (provided here atP<0.05, FDR corrected, fixed-effect analysis), only the contrast {acoustic localization minus acoustic control, masked (inclusive) by acoustic localization minus rest} provided results at P<0.001, uncorrected, in FFX analysis.  There was no significant difference in brain activation patterns between experimental conditions, even at this low threshold.  Coordinates are reported in MNI space, as given by SPM2, and correspond only approximately to Talairach and Tournoux space [21]. BA: Brodmann area. (*) belonging to the same voxels cluster.


## 3.1. Behavioral Performance

The behavioral performance during the SSD fMRI experiment was 77.8% (SD = 7.1), 78.1% (SD = 9.4), and 99.5% (SD = 2.3) of correct response for the localization, the orientation detection, and the SSD control condition, respectively. A Studentt-test performed on the scores showed no difference between the localization and the orientation condition (P=0.78), whereas the performance of the control task was significantly better (P<0.001). The mean percentages of correct responses were 80.3, 78.5, and 91.06 in the acoustic association experiment and 91.06, 92.84, and 96.42 in the visual control experiment for the localization, the orientation detection, and the control condition, respectively (all P values > 0.05, although there was a trend towards better scores in the control condition compared to the orientation condition (P=0.09) during the acoustic association experiment only).


## 3.2. fMRI Results

### 3.2.1. Sensory Substitution Experiment

The one-back comparison of dot combinations used as control condition in SSD experiment, that is, the contrast {control minus rest} showed the expected brain activation in primary and secondary auditory brain areas, bilaterally (Table1). The brain activation pattern also included frontal brain areas such as the superior frontal gyrus bilaterally (BA 9), as well as the insula and the precuneus (BA 7) bilaterally. Additional recruitment of subcortical brain areas was observed in the lentiform nucleus (putamen) bilaterally and the left thalamus (see Figure 5 and Table 1).Table 1
Overview of brain activation foci during the sensory substitution conditions.
Brain areaCluster sizeZ scorex,y,z{mm}Control > restLeft postcentral gyrus (BA 1–3)14665.50−44; −28; 66Left precentral gyrus (BA 4–6)(*)4.73−50; −20; 62Right lentiform nucleus, putamen1924.7824;  2;  6Right insula(*)4.9140; −8; 0Left superior temporal gyrus (BA 22)1204.30−44; −18; −4Left lentiform nucleus, putamen1374.24−22;  0;  4Left claustrum/insula(*)3.66−34;  0;  0Right paracentral lobule/precuneus (BA 5–7)143.8220; −42; 50Left superior frontal gyrus (BA 9)1533.78−20; 46; 32Right middle frontal gyrus (BA 9)383.6464; 12; 34Left thalamus173.47−18; −26; 8Right superior frontal gyrus (BA 9)753.2026; 58; 28Right transverse temporal gyrus (BA 41)113.1634; −26; 12Left precuneus (BA 7)103.15−6; −60; 62Localization > controlRight superior frontal gyrus (BA 8)14595.284; 30; 48Right superior frontal gyrus (BA 6)(*)4.982; 22; 54Left cerebellum6494.85−2; −34; −2Right inferior frontal gyrus (BA 47)4464.8042; 18; −6Left insula (BA 13)4814.80−34; 22; 0Left inferior parietal lobule (BA 40)12454.76−42; −52; 50Left superior parietal lobule (BA 7)(*)4.53−32; −64; 46Left precuneus (BA 7)(*)4.37−28; −50; 50Left middle frontal gyrus (BA 6)584.71−28; 6; 56Right middle frontal gyrus (BA 9)10114.7154; 28; 30Left inferior frontal gyrus (BA 46)5594.47−48; 44; 2Right cerebellum2404.478; −46; −14Left cingulate gyrus (BA 23)1464.32−4; −18; 32Left caudate body333.84−18; −6; 24Right middle frontal gyrus (BA 6)1233.7428; 14; 54Right caudate body293.6110; 2; 10Right superior parietal lobule (BA 7)1373.5934; −64; 50Right precuneus (BA 7)(*)3.2918; −68; 50Right precuneus (BA 7)183.198; −72; 52Right inferior parietal lobule (BA 40)103.1540; −44; 44Orientation > controlRight inferior frontal gyrus (BA 47)1224.8138; 20; −4Left thalamus1374.76−16; −10; 8Left inferior parietal lobule (BA 40)5984.75−40; −50; 46Left superior parietal lobule (BA 7)(*)4.39−32; −48; 48Left superior parietal lobule/precuneus (BA 7)(*)4.08−30; −66; 44Left middle frontal gyrus (BA 9)4864.72−42; 22; 36Left inferior frontal gyrus (BA 45)1894.68−32; 26; 6Right superior frontal gyrus (BA 8)7934.522; 34; 50Left substantia nigra984.38−12; −16; −8Right cerebellum314.146; −48; −20Right middle frontal gyrus (BA 9)453.9244; 2; 40Right middle frontal gyrus (BA 9)723.8858; 28; 32Left middle frontal gyrus (BA 9-10)833.80−42; 48; 20Left inferior frontal gyrus (BA 46)503.61−46; 42; 4Left cingulate gyrus (BA 23)103.76−8; −24; 32Overview of brain activation peaks in the contrasts {SSD control condition minus rest}, {SSD localization minus SSD control, masked (inclusive) with SSD Localization minus rest}, and {SSD orientation minus SSD control, masked (inclusive) with SSD Orientation minus rest} from random-effect analysis (atP<0.05, FDR corrected). Coordinates are reported in MNI space, as given by SPM2, and correspond only approximately to Talairach and Tournoux space [21]. BA: Brodmann area. (*) belonging to the same voxels cluster.Brain activation foci elicited by control (dots comparison) and experimental tasks (localization or orientation processing) through a visual-to-auditory sensory substitution device (SSD) in blindfolded sighted subjects. The statistical parametric maps for these comparisons (in a random-effects analysis in the group) were superimposed on the axial, coronal, and sagittal section of an individual normalized brain MRI. Only voxels exceeding a threshold ofP<0.05 corrected for false discovery rate (FDR) at the whole brain level were displayed according to the color scales that code the T-values. (a) Brain activation observed in control condition, that is, in the contrast {control minus rest}. A bilateral activation in auditory brain areas was evident in this contrast. (b) Brain activation foci observed during the localization condition, that is, in the contrast {localization minus control masked (inclusive) by localization minus rest}. A bilateral activation in frontal brain areas and in the dorsal visual pathway was observed in this contrast. The images were displayed at the level of the inferior parietal lobule (BA 40), which was bilaterally recruited with a main focus in the left hemisphere, with a Z value of 4.76 at coordinates (x,y,z (mm) = −42, −52, 50). (c) Brain activation foci observed during the orientation condition, that is, in the contrast {stimulus orientation minus control masked (inclusive) by stimulus orientation minus rest}. This contrast elicited a left lateralization of brain activation in the dorsal visual pathway, with a main focus in the left inferior parietal lobule (BA 40), with a Z value of 4.75 at coordinates (x,y,z (mm) = −40, −50, 46). A bilateral recruitment of frontal brain areas was also observed. (d) Brain activation foci common to both spatial processing tasks with the SSD, that is, from a conjunction analysis of experimental conditions {localization minus control and orientation detection minus control}. The common neural substrate for localization and orientation processing with the SSD involved frontal brain areas bilaterally, including the inferior, middle, and superior frontal gyri, and parietal structures located in the left hemisphere such as the precuneus and the inferior and superior parietal lobules (see Table 2 for the detailed list of the activation foci). This part of the figure focused on the left inferior parietal lobule, with a Z value of 5.50 at coordinates (x,y,z (mm) = −30, −50, 48). Coordinates are in reference to the Talairach and Tournoux atlas [21] and the MNI template (see Section 2). L: left side of the brain.
(a)
Control (dots) minus rest(b)
Localization minus control(c)
Orientation minus control(d)
Localization conjunction orientationThe localization condition, that is, the contrast {localization minus control masked (inclusive) by localization minus rest} revealed a widespread activated network including the right superior frontal gyrus, the middle and inferior frontal gyri bilaterally, the left and right inferior parietal lobule (BA 40), as well as the left and right superior parietal lobule (BA 7) and precuneus (BA 7) and the cerebellum bilaterally (Table1).The orientation detection, that is, the contrast {orientation minus control masked (inclusive) by orientation minus rest} showed a roughly similar brain activation pattern including the right superior frontal gyrus, the middle and inferior frontal gyri bilaterally and the left inferior parietal lobule (BA 40), superior parietal lobule (BA 7), and precuneus (BA 7) (Table1). The left thalamus and the right cerebellum were also activated.A conjunction analysis between the localization and orientation conditions (minus control) confirmed the equivalent recruitment of the right superior frontal gyrus (BA 6, 8) and the middle and inferior frontal gyri (BA 9, 45, 47) bilaterally, as well as the activation of the precuneus (BA 7) and the superior and inferior parietal lobules (BA 7, 40) in the left hemisphere in both conditions (see Table2).Table 2
Brain activation foci specific to experimental tasks using the sensory substitution device.
Brain areaCluster sizeZ scorex,y,z{mm}Localization∩ orientationRight superior frontal gyrus (BA 8)10955.9810; 28; 50Right superior frontal gyrus (BA 6)(*)5.142; 22; 54Right inferior frontal gyrus (BA 47)2585.6738; 20;  −6Left superior parietal lobule/inferior parietal lobule (BA 7–40)10735.50−30; −50; 48Left superior parietal lobule (BA 7)(*)5.24−40; −54; 50Left precuneus/inferior parietal lobule (BA 7–40)(*)5.18−32; −70; 46Left thalamus9845.34−2; −32;  2Left middle frontal gyrus (BA 9)10145.07−44;  6; 42Right middle frontal gyrus (BA 9)5224.8656; 22; 38Left inferior frontal gyrus (BA 45)3674.36−36; 24;  2Localization > orientationRight superior occipital gyrus/precuneus (BA 19–7)1794.6942; −74; 34Right angular gyrus (BA 39)(*)3.3754; −70; 36Right inferior parietal lobule (BA 40)(*)3.3454; −68; 44Right middle frontal gyrus (BA 6)384.2040;  6;  46Right superior frontal gyrus (BA 6)224.0240; 24; 58Right middle temporal gyrus (BA 21)223.9870; −42;  −8Right middle temporal gyrus (BA 19–39)333.9050; −74; 20Right inferior frontal gyrus (BA 10–46)143.7348; 52;   2Right superior parietal lobule (BA 7)253.7252; −60; 52Right middle occipital gyrus (BA 19)213.6536; −80; 16Right superior parietal lobule (BA 7)113.558; −68; 62Left middle temporal gyrus (BA 19–39)173.38−40; −80; 18Overview of brain activation peaks in the contrasts {(SSD localization minus SSD control) conjunction (SSD orientation minus SSD control)} and {SSD localization minus SSD orientation masked (inclusive) by SSD localization minus rest}. Results are from a random-effect analysis, atP<0.05, FDR corrected, for the conjunction, and P<0.001 uncorrected, for the differential contrast between experimental tasks. Coordinates are reported in MNI space, as given by SPM2, and correspond only approximately to Talairach and Tournoux space [21]. BA: Brodmann area. (*) belonging to the same voxels cluster.The differential contrast between the two experimental conditions, that is, the contrast {localization minus orientation masked (inclusive) by localization minus rest} (at a threshold of uncorrectedP<0.001 in combination with a cluster size threshold of 10 continuous voxels), showed that the localization task tended to be associated with larger activation in frontal, parietal, and occipitoparietal brain areas mainly in the right hemisphere, as compared to the orientation detection (Table 2). These brain activation foci included the right superior, middle, and inferior frontal gyri (BA 6, 10, 46), the right superior occipital gyrus/precuneus (BA 19, 7) and inferior parietal lobule (BA 40), the right superior parietal lobule (BA 7), and the middle temporal gyrus bilaterally (BA 19, 21, 39) (see Figure 6 and Table 2). The inverted contrast {orientation minus localization masked (inclusive) by orientation minus rest} did not reveal any significant activation focus at the selected threshold.Brain activation patterns related to location as contrasted to orientation discrimination, that is, using the contrast {localization minus orientation masked (inclusive) by localization minus rest}, with the SSD and in vision. Positive differences exceeding an uncorrected threshold ofP<0.001 were displayed according to the color scales that code the T-values. The statistical parametric map for each comparison was superimposed on the axial, coronal, and sagittal sections of an individual normalized brain MRI. The brain activation pattern observed with the SSD (a) included brain areas mainly in the right dorsal visual pathway, including the superior occipital gyrus, precuneus, inferior, and superior parietal lobules (see Table 2 for the exhaustive list of the activation foci). The red circle indicates an activation focus in the right superior parietal lobule (x,y,z (mm) = 8, −68, 62; 11 voxels). The corresponding brain activation pattern observed in vision (b) also included brain areas in the dorsal visual pathway with a slight predominance in the right hemisphere (see Table 3). The red circle indicates an activation focus in the right superior parietal lobule (x,y,z (mm) = 26, −66, 66; 86 voxels).
(a)
Localization minus orientation (SSD)(b)
Localization minus orientation (Vision)


### 3.2.2. Visual Control

We used an FFX analysis in the two subjects involved in this control, with a threshold ofP<0.05 FDR corrected in combination with a cluster size threshold of 10 continuous voxels. The visual control condition as contrasted to baseline was associated with the expected recruitment of a large frontoparietal network, known to be engaged in visuospatial working memory tasks, and including visual association areas (see Table 3). The contrast {localization minus control masked (inclusive) by localization minus rest} disclosed brain activation foci in occipital and parietal areas, included in the superior parietal lobule (BA 7) and the cuneus (BA 17, 18) bilaterally. The contrast {orientation minus control masked (inclusive) by orientation minus rest} did not reveal any significant activation focus at the selected threshold. When we compared the localization condition with the orientation condition, that is, using the contrast {localization minus orientation masked (inclusive) by localization minus rest} (at a threshold of P<0.001, uncorrected, in combination with a cluster size threshold of 10 continuous voxels), brain activation voxels clusters survived in the right superior parietal lobule (BA 7) and, to a lesser extent, in the right and left superior occipital gyrus/precuneus (BA 19, 7). Obviously the reverse contrast {orientation minus localization} did not reveal any activation focus at the selected threshold.Table 3
Brain activation during the visual conditions.
Brain areaCluster sizeZ scorex,y,z{mm}Control > restLeft precuneus/superior parietal lobule (BA 7)4435.98−14; −74; 54Left precuneus/superior parietal lobule (BA 7)(*)4.74−22; −60; 48Right middle frontal gyrus (BA 6–8)1055.8552; 8; 48Right superior frontal gyrus (BA 6)1035.662; 24; 58Right cuneus (BA 17-18)3465.4720; −90; 6Right middle frontal gyrus (BA 9)1165.4250; 28; 32Left inferior occipital gyrus (BA 18)1085.32−28; −94; −4Right precuneus/superior parietal lobule (BA 7)1264.8818; −68; 54Left superior frontal gyrus (BA 8)1064.29−4; 22; 48Left middle frontal gyrus (BA 9)403.78−16; 44; 34Localization > controlRight superior parietal lobule (BA 7)3525.2220; −66; 56Left superior parietal lobule (BA 7)2064.79−28; −60; 52Left precuneus/superior parietal lobule (BA 7)(*)4.62−18; −62; 52Left superior parietal lobule (BA 7)(*)4.24−30; −68; 52Left cuneus (BA 17-18)694.76−6; −96; 8Right cuneus (BA 17)303.9910; −90; 4Orientation > controlLocalization > orientationRight superior parietal lobule (BA 7)864.1226; −66; 66Right superior parietal lobule (BA 7)(*)3.8728; −56; 60Left superior occipital gyrus/precuneus (BA 19–7)414.06−36; −78; 42Right superior occipital gyrus/precuneus (BA 19–7)103.7738; −68; 44Overview of brain activation peaks during the additional visual control experiment.  When experimental conditions were compared to control condition, only the contrast {visual localization minus visual control, masked (inclusive) with visual localization minus rest} provided results atP<0.05, FDR corrected (fixed-effect analysis).  Results from the differential contrast between experimental conditions, that is, {visual localization minus visual orientation masked (inclusive) by visual localization minus rest} are also reported (from a fixed-effect analysis at P<0.001, uncorrected) for reference.  Coordinates are reported in MNI space, as given by SPM2, and correspond only approximately to Talairach and Tournoux space [21]. BA: Brodmann area. (*) belonging to the same voxels cluster.


### 3.2.3. Acoustic Association

We used an FFX analysis in the two subjects who had learned to associate each sound with its visual representation to estimate the condition-related brain activity changes when based on acoustic association. None of the contrasts performed ({localization minus control}, {orientation minus control}, {localization minus orientation} and {orientation minus localization)} revealed any activation in the parieto-occipital cortex in any of the two subjects (at a threshold ofP<0.001, uncorrected, in combination with a cluster size threshold of 10 continuous voxels). Only the localization task compared to control condition induced some weak activation restricted to the auditory cortex and temporal brain areas (see Table 4).Table 4
Brain activation foci related to the acoustic association conditions.
Brain areaCluster sizeZ scoreX,Y,Z{mm}Control > restRight middle frontal gyrus (BA 9)5365.8252; 10; 36Left middle frontal gyrus (BA 8-9)8545.78−46; 28; 38Right transverse/superior temporal gyrus (BA 41-42)1024.6050; −24; 10Right superior temporal gyrus (BA 22)(*)4.0352; −8; 6Left superior temporal gyrus (BA 22)1454.28−50; −42; 18Left transverse/superior temporal gyrus (BA 41-42)(*)3.76−50; −20; 12Localization > controlLeft insula (BA 13)354.58−42; −18; −2Left superior temporal gyrus (BA 21-22)343.97−44; −30;  0Left transverse/superior temporal gyrus (BA 41-42)(*)3.58−38; −40; 12Right superior temporal gyrus (BA 42–22)463.9646; −38; 20Right superior temporal gyrus (BA 42–22)133.8856; −32; 16Right superior temporal/supramarginal gyrus (BA 22–40)163.4968; −48; 22Left supramarginal gyrus (BA 40)103.49−46; −48; 32Orientation > controlLocalization > orientationOverview of brain activation peaks during the acoustic association control experiment. When experimental conditions were compared to control condition (provided here atP<0.05, FDR corrected, fixed-effect analysis), only the contrast {acoustic localization minus acoustic control, masked (inclusive) by acoustic localization minus rest} provided results at P<0.001, uncorrected, in FFX analysis.  There was no significant difference in brain activation patterns between experimental conditions, even at this low threshold.  Coordinates are reported in MNI space, as given by SPM2, and correspond only approximately to Talairach and Tournoux space [21]. BA: Brodmann area. (*) belonging to the same voxels cluster.


## 3.2.1. Sensory Substitution Experiment

The one-back comparison of dot combinations used as control condition in SSD experiment, that is, the contrast {control minus rest} showed the expected brain activation in primary and secondary auditory brain areas, bilaterally (Table1). The brain activation pattern also included frontal brain areas such as the superior frontal gyrus bilaterally (BA 9), as well as the insula and the precuneus (BA 7) bilaterally. Additional recruitment of subcortical brain areas was observed in the lentiform nucleus (putamen) bilaterally and the left thalamus (see Figure 5 and Table 1).Table 1
Overview of brain activation foci during the sensory substitution conditions.
Brain areaCluster sizeZ scorex,y,z{mm}Control > restLeft postcentral gyrus (BA 1–3)14665.50−44; −28; 66Left precentral gyrus (BA 4–6)(*)4.73−50; −20; 62Right lentiform nucleus, putamen1924.7824;  2;  6Right insula(*)4.9140; −8; 0Left superior temporal gyrus (BA 22)1204.30−44; −18; −4Left lentiform nucleus, putamen1374.24−22;  0;  4Left claustrum/insula(*)3.66−34;  0;  0Right paracentral lobule/precuneus (BA 5–7)143.8220; −42; 50Left superior frontal gyrus (BA 9)1533.78−20; 46; 32Right middle frontal gyrus (BA 9)383.6464; 12; 34Left thalamus173.47−18; −26; 8Right superior frontal gyrus (BA 9)753.2026; 58; 28Right transverse temporal gyrus (BA 41)113.1634; −26; 12Left precuneus (BA 7)103.15−6; −60; 62Localization > controlRight superior frontal gyrus (BA 8)14595.284; 30; 48Right superior frontal gyrus (BA 6)(*)4.982; 22; 54Left cerebellum6494.85−2; −34; −2Right inferior frontal gyrus (BA 47)4464.8042; 18; −6Left insula (BA 13)4814.80−34; 22; 0Left inferior parietal lobule (BA 40)12454.76−42; −52; 50Left superior parietal lobule (BA 7)(*)4.53−32; −64; 46Left precuneus (BA 7)(*)4.37−28; −50; 50Left middle frontal gyrus (BA 6)584.71−28; 6; 56Right middle frontal gyrus (BA 9)10114.7154; 28; 30Left inferior frontal gyrus (BA 46)5594.47−48; 44; 2Right cerebellum2404.478; −46; −14Left cingulate gyrus (BA 23)1464.32−4; −18; 32Left caudate body333.84−18; −6; 24Right middle frontal gyrus (BA 6)1233.7428; 14; 54Right caudate body293.6110; 2; 10Right superior parietal lobule (BA 7)1373.5934; −64; 50Right precuneus (BA 7)(*)3.2918; −68; 50Right precuneus (BA 7)183.198; −72; 52Right inferior parietal lobule (BA 40)103.1540; −44; 44Orientation > controlRight inferior frontal gyrus (BA 47)1224.8138; 20; −4Left thalamus1374.76−16; −10; 8Left inferior parietal lobule (BA 40)5984.75−40; −50; 46Left superior parietal lobule (BA 7)(*)4.39−32; −48; 48Left superior parietal lobule/precuneus (BA 7)(*)4.08−30; −66; 44Left middle frontal gyrus (BA 9)4864.72−42; 22; 36Left inferior frontal gyrus (BA 45)1894.68−32; 26; 6Right superior frontal gyrus (BA 8)7934.522; 34; 50Left substantia nigra984.38−12; −16; −8Right cerebellum314.146; −48; −20Right middle frontal gyrus (BA 9)453.9244; 2; 40Right middle frontal gyrus (BA 9)723.8858; 28; 32Left middle frontal gyrus (BA 9-10)833.80−42; 48; 20Left inferior frontal gyrus (BA 46)503.61−46; 42; 4Left cingulate gyrus (BA 23)103.76−8; −24; 32Overview of brain activation peaks in the contrasts {SSD control condition minus rest}, {SSD localization minus SSD control, masked (inclusive) with SSD Localization minus rest}, and {SSD orientation minus SSD control, masked (inclusive) with SSD Orientation minus rest} from random-effect analysis (atP<0.05, FDR corrected). Coordinates are reported in MNI space, as given by SPM2, and correspond only approximately to Talairach and Tournoux space [21]. BA: Brodmann area. (*) belonging to the same voxels cluster.Brain activation foci elicited by control (dots comparison) and experimental tasks (localization or orientation processing) through a visual-to-auditory sensory substitution device (SSD) in blindfolded sighted subjects. The statistical parametric maps for these comparisons (in a random-effects analysis in the group) were superimposed on the axial, coronal, and sagittal section of an individual normalized brain MRI. Only voxels exceeding a threshold ofP<0.05 corrected for false discovery rate (FDR) at the whole brain level were displayed according to the color scales that code the T-values. (a) Brain activation observed in control condition, that is, in the contrast {control minus rest}. A bilateral activation in auditory brain areas was evident in this contrast. (b) Brain activation foci observed during the localization condition, that is, in the contrast {localization minus control masked (inclusive) by localization minus rest}. A bilateral activation in frontal brain areas and in the dorsal visual pathway was observed in this contrast. The images were displayed at the level of the inferior parietal lobule (BA 40), which was bilaterally recruited with a main focus in the left hemisphere, with a Z value of 4.76 at coordinates (x,y,z (mm) = −42, −52, 50). (c) Brain activation foci observed during the orientation condition, that is, in the contrast {stimulus orientation minus control masked (inclusive) by stimulus orientation minus rest}. This contrast elicited a left lateralization of brain activation in the dorsal visual pathway, with a main focus in the left inferior parietal lobule (BA 40), with a Z value of 4.75 at coordinates (x,y,z (mm) = −40, −50, 46). A bilateral recruitment of frontal brain areas was also observed. (d) Brain activation foci common to both spatial processing tasks with the SSD, that is, from a conjunction analysis of experimental conditions {localization minus control and orientation detection minus control}. The common neural substrate for localization and orientation processing with the SSD involved frontal brain areas bilaterally, including the inferior, middle, and superior frontal gyri, and parietal structures located in the left hemisphere such as the precuneus and the inferior and superior parietal lobules (see Table 2 for the detailed list of the activation foci). This part of the figure focused on the left inferior parietal lobule, with a Z value of 5.50 at coordinates (x,y,z (mm) = −30, −50, 48). Coordinates are in reference to the Talairach and Tournoux atlas [21] and the MNI template (see Section 2). L: left side of the brain.
(a)
Control (dots) minus rest(b)
Localization minus control(c)
Orientation minus control(d)
Localization conjunction orientationThe localization condition, that is, the contrast {localization minus control masked (inclusive) by localization minus rest} revealed a widespread activated network including the right superior frontal gyrus, the middle and inferior frontal gyri bilaterally, the left and right inferior parietal lobule (BA 40), as well as the left and right superior parietal lobule (BA 7) and precuneus (BA 7) and the cerebellum bilaterally (Table1).The orientation detection, that is, the contrast {orientation minus control masked (inclusive) by orientation minus rest} showed a roughly similar brain activation pattern including the right superior frontal gyrus, the middle and inferior frontal gyri bilaterally and the left inferior parietal lobule (BA 40), superior parietal lobule (BA 7), and precuneus (BA 7) (Table1). The left thalamus and the right cerebellum were also activated.A conjunction analysis between the localization and orientation conditions (minus control) confirmed the equivalent recruitment of the right superior frontal gyrus (BA 6, 8) and the middle and inferior frontal gyri (BA 9, 45, 47) bilaterally, as well as the activation of the precuneus (BA 7) and the superior and inferior parietal lobules (BA 7, 40) in the left hemisphere in both conditions (see Table2).Table 2
Brain activation foci specific to experimental tasks using the sensory substitution device.
Brain areaCluster sizeZ scorex,y,z{mm}Localization∩ orientationRight superior frontal gyrus (BA 8)10955.9810; 28; 50Right superior frontal gyrus (BA 6)(*)5.142; 22; 54Right inferior frontal gyrus (BA 47)2585.6738; 20;  −6Left superior parietal lobule/inferior parietal lobule (BA 7–40)10735.50−30; −50; 48Left superior parietal lobule (BA 7)(*)5.24−40; −54; 50Left precuneus/inferior parietal lobule (BA 7–40)(*)5.18−32; −70; 46Left thalamus9845.34−2; −32;  2Left middle frontal gyrus (BA 9)10145.07−44;  6; 42Right middle frontal gyrus (BA 9)5224.8656; 22; 38Left inferior frontal gyrus (BA 45)3674.36−36; 24;  2Localization > orientationRight superior occipital gyrus/precuneus (BA 19–7)1794.6942; −74; 34Right angular gyrus (BA 39)(*)3.3754; −70; 36Right inferior parietal lobule (BA 40)(*)3.3454; −68; 44Right middle frontal gyrus (BA 6)384.2040;  6;  46Right superior frontal gyrus (BA 6)224.0240; 24; 58Right middle temporal gyrus (BA 21)223.9870; −42;  −8Right middle temporal gyrus (BA 19–39)333.9050; −74; 20Right inferior frontal gyrus (BA 10–46)143.7348; 52;   2Right superior parietal lobule (BA 7)253.7252; −60; 52Right middle occipital gyrus (BA 19)213.6536; −80; 16Right superior parietal lobule (BA 7)113.558; −68; 62Left middle temporal gyrus (BA 19–39)173.38−40; −80; 18Overview of brain activation peaks in the contrasts {(SSD localization minus SSD control) conjunction (SSD orientation minus SSD control)} and {SSD localization minus SSD orientation masked (inclusive) by SSD localization minus rest}. Results are from a random-effect analysis, atP<0.05, FDR corrected, for the conjunction, and P<0.001 uncorrected, for the differential contrast between experimental tasks. Coordinates are reported in MNI space, as given by SPM2, and correspond only approximately to Talairach and Tournoux space [21]. BA: Brodmann area. (*) belonging to the same voxels cluster.The differential contrast between the two experimental conditions, that is, the contrast {localization minus orientation masked (inclusive) by localization minus rest} (at a threshold of uncorrectedP<0.001 in combination with a cluster size threshold of 10 continuous voxels), showed that the localization task tended to be associated with larger activation in frontal, parietal, and occipitoparietal brain areas mainly in the right hemisphere, as compared to the orientation detection (Table 2). These brain activation foci included the right superior, middle, and inferior frontal gyri (BA 6, 10, 46), the right superior occipital gyrus/precuneus (BA 19, 7) and inferior parietal lobule (BA 40), the right superior parietal lobule (BA 7), and the middle temporal gyrus bilaterally (BA 19, 21, 39) (see Figure 6 and Table 2). The inverted contrast {orientation minus localization masked (inclusive) by orientation minus rest} did not reveal any significant activation focus at the selected threshold.Brain activation patterns related to location as contrasted to orientation discrimination, that is, using the contrast {localization minus orientation masked (inclusive) by localization minus rest}, with the SSD and in vision. Positive differences exceeding an uncorrected threshold ofP<0.001 were displayed according to the color scales that code the T-values. The statistical parametric map for each comparison was superimposed on the axial, coronal, and sagittal sections of an individual normalized brain MRI. The brain activation pattern observed with the SSD (a) included brain areas mainly in the right dorsal visual pathway, including the superior occipital gyrus, precuneus, inferior, and superior parietal lobules (see Table 2 for the exhaustive list of the activation foci). The red circle indicates an activation focus in the right superior parietal lobule (x,y,z (mm) = 8, −68, 62; 11 voxels). The corresponding brain activation pattern observed in vision (b) also included brain areas in the dorsal visual pathway with a slight predominance in the right hemisphere (see Table 3). The red circle indicates an activation focus in the right superior parietal lobule (x,y,z (mm) = 26, −66, 66; 86 voxels).
(a)
Localization minus orientation (SSD)(b)
Localization minus orientation (Vision)


## 3.2.2. Visual Control

We used an FFX analysis in the two subjects involved in this control, with a threshold ofP<0.05 FDR corrected in combination with a cluster size threshold of 10 continuous voxels. The visual control condition as contrasted to baseline was associated with the expected recruitment of a large frontoparietal network, known to be engaged in visuospatial working memory tasks, and including visual association areas (see Table 3). The contrast {localization minus control masked (inclusive) by localization minus rest} disclosed brain activation foci in occipital and parietal areas, included in the superior parietal lobule (BA 7) and the cuneus (BA 17, 18) bilaterally. The contrast {orientation minus control masked (inclusive) by orientation minus rest} did not reveal any significant activation focus at the selected threshold. When we compared the localization condition with the orientation condition, that is, using the contrast {localization minus orientation masked (inclusive) by localization minus rest} (at a threshold of P<0.001, uncorrected, in combination with a cluster size threshold of 10 continuous voxels), brain activation voxels clusters survived in the right superior parietal lobule (BA 7) and, to a lesser extent, in the right and left superior occipital gyrus/precuneus (BA 19, 7). Obviously the reverse contrast {orientation minus localization} did not reveal any activation focus at the selected threshold.Table 3
Brain activation during the visual conditions.
Brain areaCluster sizeZ scorex,y,z{mm}Control > restLeft precuneus/superior parietal lobule (BA 7)4435.98−14; −74; 54Left precuneus/superior parietal lobule (BA 7)(*)4.74−22; −60; 48Right middle frontal gyrus (BA 6–8)1055.8552; 8; 48Right superior frontal gyrus (BA 6)1035.662; 24; 58Right cuneus (BA 17-18)3465.4720; −90; 6Right middle frontal gyrus (BA 9)1165.4250; 28; 32Left inferior occipital gyrus (BA 18)1085.32−28; −94; −4Right precuneus/superior parietal lobule (BA 7)1264.8818; −68; 54Left superior frontal gyrus (BA 8)1064.29−4; 22; 48Left middle frontal gyrus (BA 9)403.78−16; 44; 34Localization > controlRight superior parietal lobule (BA 7)3525.2220; −66; 56Left superior parietal lobule (BA 7)2064.79−28; −60; 52Left precuneus/superior parietal lobule (BA 7)(*)4.62−18; −62; 52Left superior parietal lobule (BA 7)(*)4.24−30; −68; 52Left cuneus (BA 17-18)694.76−6; −96; 8Right cuneus (BA 17)303.9910; −90; 4Orientation > controlLocalization > orientationRight superior parietal lobule (BA 7)864.1226; −66; 66Right superior parietal lobule (BA 7)(*)3.8728; −56; 60Left superior occipital gyrus/precuneus (BA 19–7)414.06−36; −78; 42Right superior occipital gyrus/precuneus (BA 19–7)103.7738; −68; 44Overview of brain activation peaks during the additional visual control experiment.  When experimental conditions were compared to control condition, only the contrast {visual localization minus visual control, masked (inclusive) with visual localization minus rest} provided results atP<0.05, FDR corrected (fixed-effect analysis).  Results from the differential contrast between experimental conditions, that is, {visual localization minus visual orientation masked (inclusive) by visual localization minus rest} are also reported (from a fixed-effect analysis at P<0.001, uncorrected) for reference.  Coordinates are reported in MNI space, as given by SPM2, and correspond only approximately to Talairach and Tournoux space [21]. BA: Brodmann area. (*) belonging to the same voxels cluster.


## 3.2.3. Acoustic Association

We used an FFX analysis in the two subjects who had learned to associate each sound with its visual representation to estimate the condition-related brain activity changes when based on acoustic association. None of the contrasts performed ({localization minus control}, {orientation minus control}, {localization minus orientation} and {orientation minus localization)} revealed any activation in the parieto-occipital cortex in any of the two subjects (at a threshold ofP<0.001, uncorrected, in combination with a cluster size threshold of 10 continuous voxels). Only the localization task compared to control condition induced some weak activation restricted to the auditory cortex and temporal brain areas (see Table 4).Table 4
Brain activation foci related to the acoustic association conditions.
Brain areaCluster sizeZ scoreX,Y,Z{mm}Control > restRight middle frontal gyrus (BA 9)5365.8252; 10; 36Left middle frontal gyrus (BA 8-9)8545.78−46; 28; 38Right transverse/superior temporal gyrus (BA 41-42)1024.6050; −24; 10Right superior temporal gyrus (BA 22)(*)4.0352; −8; 6Left superior temporal gyrus (BA 22)1454.28−50; −42; 18Left transverse/superior temporal gyrus (BA 41-42)(*)3.76−50; −20; 12Localization > controlLeft insula (BA 13)354.58−42; −18; −2Left superior temporal gyrus (BA 21-22)343.97−44; −30;  0Left transverse/superior temporal gyrus (BA 41-42)(*)3.58−38; −40; 12Right superior temporal gyrus (BA 42–22)463.9646; −38; 20Right superior temporal gyrus (BA 42–22)133.8856; −32; 16Right superior temporal/supramarginal gyrus (BA 22–40)163.4968; −48; 22Left supramarginal gyrus (BA 40)103.49−46; −48; 32Orientation > controlLocalization > orientationOverview of brain activation peaks during the acoustic association control experiment. When experimental conditions were compared to control condition (provided here atP<0.05, FDR corrected, fixed-effect analysis), only the contrast {acoustic localization minus acoustic control, masked (inclusive) by acoustic localization minus rest} provided results at P<0.001, uncorrected, in FFX analysis.  There was no significant difference in brain activation patterns between experimental conditions, even at this low threshold.  Coordinates are reported in MNI space, as given by SPM2, and correspond only approximately to Talairach and Tournoux space [21]. BA: Brodmann area. (*) belonging to the same voxels cluster.


## 4. Discussion

This experimental design aimed to test how the brain codes orientation and localization of a visual stimulus transformed into sounds in healthy individuals who were using a sensory substitution device developed for the blind and known to allow visual-like perception through analytic cognitive operations. We measured the task-related activation patterns for two spatial tasks on visual stimuli encoded into sounds and observed a similar recruitment of frontoparietal and occipital brain areas as the corresponding tasks performed in vision. This provides compelling evidence that the parietal and occipital cortex activation during the use of the prosthesis is task related and it further indicates the multisensory recruitment of the dorsal visual pathway in spatial processing.


### 4.1. Task-Specific Activation beside SSD Signal Decoding

This experiment aimed to address one aspect of the use of a sensory substitution device: the division between activation related to decoding the SSD signal and this relating to the particular task being carried on the information once decoded. Differing from vision, the cognitive operations that allow visual-like perception through sensory substitution are mainly sequential and analytic. Auditory recognition of a visual stimulus through an SSD like the PSVA requires focusing attention on only parts of the stimulus at a time, while the total image has to be mentally synthesized over time. It is therefore not surprising that the use of such SSD to perceive complex environmental scenes, objects, or patterns constantly activated brain areas responsible for spatial processing (in the parietal cortex) and working memory (in frontal lobes) included in blind subjects, whereas little is known about the respective contribution of SSD signal decoding, stimulus perception, and task-specific processing in this brain activation pattern. The present study was designed to fill in this gab by showing how processing the location versus the orientation of identical “visual” stimuli encoded into sounds modulated the brain activity within the dorsal visual stream in proficient SSD users who did not use stimulus exploration strategies or self-generated exploratory movements. Furthermore, the same memorization process was used in each experimental and control condition, and, to minimize the brain activation relating to decoding the SSD signal during the tasks, we used very simple stimuli that could be apprehended globally and almost instantly. Although the use of very simple stimuli may not be optimal to probe strong brain activation within the visual cortex, stimuli of a similar level of complexity were successfully used in previous studies on spatial processing in vision [3] and in the tactile modality [28–30].


### 4.2. Task-Specific Activation and Hemispheric Specialization

Task-specific activation foci were observed in the dorsal visual pathway when using the SSD to compare the location or the orientation of successive stimuli, which were not present in the SSD control condition. The superior occipital gyrus, precuneus, and the inferior and superior parietal lobules (IPL, SPL) in the right hemisphere were slightly more strongly activated during the localization condition as compared to the orientation condition. Such task-related modulation of brain areas recruitment cannot be due to a difference in the difficulty level since the behavioral performance was equivalent between the localization and the orientation conditions. In addition, since the same stimuli were used in both experimental conditions, no difference in their nature or complexity could account for differences in brain activation patterns. Furthermore, most spatial aspects related to the use of the SSD were controlled and kept to a minimum in all conditions (there was no stimulus exploration and the stimuli did not comprise any spatial arrangement of isolated elements). In vision, the right SPL showed a similar preference for location over orientation processing. Although the activation focus in the right SPL did not perfectly overlap the one observed in the SSD experiment, it was located in the same structure and in its direct vicinity. In the visual location discrimination, as compared to the visual control condition (see Table3), the SPL was the main activated structure, with, albeit to a lesser extent, the cuneus, in both hemispheres. The weaker effect for the spatial processing tasks observed in the visual modality (i.e., the restricted recruitment of the dorsal stream during location discrimination and its absence during orientation detection as compared to control condition) could be due to the relative simplicity of the tasks in vision that could have led subjects to automatically recognize and localize the stimuli during all conditions, this resulting in a reduction of the specificity of the brain activation patterns. Nevertheless, the task-specific activation patterns for stimulus localization observed both with the SSD and in vision were located mainly in the right hemisphere, in full accordance with the hemispheric specialization hypothesis according to which the right hemisphere would be more involved than the left one in spatial processing, at least in right-handed subjects [31, 32]. Surprisingly, no task-specific activation was observed for the orientation processing (as compared to location discrimination) in any of the sensory modalities although previous studies on sensory substitution that used more complex stimuli revealed clear and strong brain activations in the ventral visual streams during shape recognition tasks [5, 6, 8, 10]. Several explanations could be proposed to that (1) the stimuli used (i.e., horizontal bars and vertical bars) were too simple to engage significantly shape processing, which also might explain the absence of lateral occipital complex (LOC) recruitment and any brain activation foci in the ventral stream during the orientation judgment as compared to control condition and (2) the stimulus orientation was automatically processed during the localization condition.


### 4.3. Multisensory Integration and Spatial Processing in the Parietal Cortex

The involvement of common parietal brain areas in spatial processing both with the SSD and in vision indicated that part of the parietal cortex could be specialized in the processing of spatial attributes at a supramodal level. This observation was consistent with previous studies. For instance, a bilateral functional recruitment of the SPL and the precuneus was previously reported in auditory and tactile studies involving spatial processing [33–37]. These two brain structures were also described as being part of the cerebral network involved in the processing of spatial information in vision [3, 4]. The IPL, which sustains a wide range of functions related to attention, motion processing, stereo vision, and spatial working memory [38–41], was widely reported as involved in spatial hearing and in tactile localization [36, 42–50]. All these observations indicated that parts of the parietal cortex should play a role of multisensory operator for the processing of spatial information [37]. It is worth noting that a brain area similarly specialized in the processing of object shape whatever the sensory input, the lateral occipital complex (LOC), was previously found in the ventral visual stream [51–55]. Such convergence of the same functional attributes of auditory, visual, and tactile stimuli into one centralized representation would constitute an optimal form of brain organization for the rapid and efficient processing and binding of perceptual information from different sensory modalities. This might play a vital role in our unified perception of the environment and in cross-modal adaptive changes observed in congenital blindness or deafness.


### 4.4. Potential Role of Mental Imagery and Working Memory in Sensory Substitution Studies

Several frontal and parietal regions located mainly in the left hemisphere were commonly activated during both experimental conditions involving the SSD and were more activated during the experimental conditions than during the control condition. In particular, the inferior, middle, and superior frontal gyri, the left precuneus, and the left IPL, and SPL were similarly recruited by both the localization and the orientation comparison tasks. Interestingly, a similar recruitment of the precuneus, the IPL and the SPL was already found in most studies during the use of either an SSD substituting vision with audition (e.g., the vOICe, the PSVA) or a tactile vision substitution system (e.g., the TVSS in its pneumatic, electric, or vibrotactile variant, the tongue display unit (TDU)) to perform various tasks, for example, to recognize 2D figures [5, 9], to process object shape [10], to estimate object distance in a 3D scene [6], to determine the orientation of a letter [7], to judge the direction of a stimulus in motion [11], or to recognize a virtual route [56]. These brain activation foci and other brain areas located in the ventral and dorsal visual streams were observed in studies that used an SSD, not only in the blind but also in normally sighted subjects (see Supplementary Table  1 for a list of activated voxels clusters that were previously reported on Supplementary material available online at doi:10.5402/2012/490950). Altogether, we cannot exclude that these as well as some of the brain activations observed in the present study were mediated by mental imagery mechanisms or working memory processes. The precuneus and the SPL were reported as involved in spatial mental imagery tasks [57–59], and the IPL and the SPL were also found to be involved in spatial working memory tasks (see [60] for a review). Although the subtraction of the control condition within the sensory substitution experiment was supposed to neutralize working memory aspects involved in the one-back comparison task, we cannot exclude that the experimental conditions (localization and orientation detection) engaged more strongly working memory than the control condition did. In the present study, no activation within the visual cortex was observed during the acoustic association conditions, which brought support to the hypothesis according to which sensory substitution would specifically recruit visual mechanisms [12] and engage different cognitive processes than a regular sound-object association task [5, 10]. The lack of brain modulation in the auditory paradigm was probably due to the fact that subjects used a same strategy in all tasks: the memorization of meaningless associations. Such strategy did not involve location or orientation processing. By contrast, during the sensory substitution conditions, subjects had to convert sounds into visual representations to perform the task based on the resulting “visual” representation they had built mentally [12]. Interestingly, a modulation of the brain activity according to the nature of the percept was previously observed in a study involving the same SSD and that contrasted the processing of 2D versus 3D attributes of the same stimuli [6]. The present study showed up a modulation of the brain activity according to the performed task on the same stimuli. Altogether, this leads us to favor the hypothesis according to which the brain activation patterns typically observed during the use of an SSD combine some general mechanisms related to sensory substitution and some specific processes related to the nature of the perception provided by the device (e.g., as in [6]) and to the nature of the task performed with the device (as presently).


### 4.5. Conclusions

Here we showed that regions in the parietal cortex were specifically recruited during spatial processing tasks both in vision and when using a visual-to-auditory sensory substitution device. In particular, the right superior parietal lobule was more activated during localization compared to orientation detection of a same stimulus in both sensory modalities. This indicates that part of the dorsal visual pathway would be a multisensory operator for spatial processing of information coming from different sensory modalities. Additional work with a greater variety of stimuli and paradigms is clearly needed to test whether other multisensory specialized regions can be found along the processing streams. Another challenge for the future is to connect the findings from sensory substitution studies in a coherent framework that defines the nature of such multisensory interactions in the healthy human and their alteration by visual or auditory deprivation.


## 4.1. Task-Specific Activation beside SSD Signal Decoding

This experiment aimed to address one aspect of the use of a sensory substitution device: the division between activation related to decoding the SSD signal and this relating to the particular task being carried on the information once decoded. Differing from vision, the cognitive operations that allow visual-like perception through sensory substitution are mainly sequential and analytic. Auditory recognition of a visual stimulus through an SSD like the PSVA requires focusing attention on only parts of the stimulus at a time, while the total image has to be mentally synthesized over time. It is therefore not surprising that the use of such SSD to perceive complex environmental scenes, objects, or patterns constantly activated brain areas responsible for spatial processing (in the parietal cortex) and working memory (in frontal lobes) included in blind subjects, whereas little is known about the respective contribution of SSD signal decoding, stimulus perception, and task-specific processing in this brain activation pattern. The present study was designed to fill in this gab by showing how processing the location versus the orientation of identical “visual” stimuli encoded into sounds modulated the brain activity within the dorsal visual stream in proficient SSD users who did not use stimulus exploration strategies or self-generated exploratory movements. Furthermore, the same memorization process was used in each experimental and control condition, and, to minimize the brain activation relating to decoding the SSD signal during the tasks, we used very simple stimuli that could be apprehended globally and almost instantly. Although the use of very simple stimuli may not be optimal to probe strong brain activation within the visual cortex, stimuli of a similar level of complexity were successfully used in previous studies on spatial processing in vision [3] and in the tactile modality [28–30].


## 4.2. Task-Specific Activation and Hemispheric Specialization

Task-specific activation foci were observed in the dorsal visual pathway when using the SSD to compare the location or the orientation of successive stimuli, which were not present in the SSD control condition. The superior occipital gyrus, precuneus, and the inferior and superior parietal lobules (IPL, SPL) in the right hemisphere were slightly more strongly activated during the localization condition as compared to the orientation condition. Such task-related modulation of brain areas recruitment cannot be due to a difference in the difficulty level since the behavioral performance was equivalent between the localization and the orientation conditions. In addition, since the same stimuli were used in both experimental conditions, no difference in their nature or complexity could account for differences in brain activation patterns. Furthermore, most spatial aspects related to the use of the SSD were controlled and kept to a minimum in all conditions (there was no stimulus exploration and the stimuli did not comprise any spatial arrangement of isolated elements). In vision, the right SPL showed a similar preference for location over orientation processing. Although the activation focus in the right SPL did not perfectly overlap the one observed in the SSD experiment, it was located in the same structure and in its direct vicinity. In the visual location discrimination, as compared to the visual control condition (see Table3), the SPL was the main activated structure, with, albeit to a lesser extent, the cuneus, in both hemispheres. The weaker effect for the spatial processing tasks observed in the visual modality (i.e., the restricted recruitment of the dorsal stream during location discrimination and its absence during orientation detection as compared to control condition) could be due to the relative simplicity of the tasks in vision that could have led subjects to automatically recognize and localize the stimuli during all conditions, this resulting in a reduction of the specificity of the brain activation patterns. Nevertheless, the task-specific activation patterns for stimulus localization observed both with the SSD and in vision were located mainly in the right hemisphere, in full accordance with the hemispheric specialization hypothesis according to which the right hemisphere would be more involved than the left one in spatial processing, at least in right-handed subjects [31, 32]. Surprisingly, no task-specific activation was observed for the orientation processing (as compared to location discrimination) in any of the sensory modalities although previous studies on sensory substitution that used more complex stimuli revealed clear and strong brain activations in the ventral visual streams during shape recognition tasks [5, 6, 8, 10]. Several explanations could be proposed to that (1) the stimuli used (i.e., horizontal bars and vertical bars) were too simple to engage significantly shape processing, which also might explain the absence of lateral occipital complex (LOC) recruitment and any brain activation foci in the ventral stream during the orientation judgment as compared to control condition and (2) the stimulus orientation was automatically processed during the localization condition.


## 4.3. Multisensory Integration and Spatial Processing in the Parietal Cortex

The involvement of common parietal brain areas in spatial processing both with the SSD and in vision indicated that part of the parietal cortex could be specialized in the processing of spatial attributes at a supramodal level. This observation was consistent with previous studies. For instance, a bilateral functional recruitment of the SPL and the precuneus was previously reported in auditory and tactile studies involving spatial processing [33–37]. These two brain structures were also described as being part of the cerebral network involved in the processing of spatial information in vision [3, 4]. The IPL, which sustains a wide range of functions related to attention, motion processing, stereo vision, and spatial working memory [38–41], was widely reported as involved in spatial hearing and in tactile localization [36, 42–50]. All these observations indicated that parts of the parietal cortex should play a role of multisensory operator for the processing of spatial information [37]. It is worth noting that a brain area similarly specialized in the processing of object shape whatever the sensory input, the lateral occipital complex (LOC), was previously found in the ventral visual stream [51–55]. Such convergence of the same functional attributes of auditory, visual, and tactile stimuli into one centralized representation would constitute an optimal form of brain organization for the rapid and efficient processing and binding of perceptual information from different sensory modalities. This might play a vital role in our unified perception of the environment and in cross-modal adaptive changes observed in congenital blindness or deafness.


## 4.4. Potential Role of Mental Imagery and Working Memory in Sensory Substitution Studies

Several frontal and parietal regions located mainly in the left hemisphere were commonly activated during both experimental conditions involving the SSD and were more activated during the experimental conditions than during the control condition. In particular, the inferior, middle, and superior frontal gyri, the left precuneus, and the left IPL, and SPL were similarly recruited by both the localization and the orientation comparison tasks. Interestingly, a similar recruitment of the precuneus, the IPL and the SPL was already found in most studies during the use of either an SSD substituting vision with audition (e.g., the vOICe, the PSVA) or a tactile vision substitution system (e.g., the TVSS in its pneumatic, electric, or vibrotactile variant, the tongue display unit (TDU)) to perform various tasks, for example, to recognize 2D figures [5, 9], to process object shape [10], to estimate object distance in a 3D scene [6], to determine the orientation of a letter [7], to judge the direction of a stimulus in motion [11], or to recognize a virtual route [56]. These brain activation foci and other brain areas located in the ventral and dorsal visual streams were observed in studies that used an SSD, not only in the blind but also in normally sighted subjects (see Supplementary Table  1 for a list of activated voxels clusters that were previously reported on Supplementary material available online at doi:10.5402/2012/490950). Altogether, we cannot exclude that these as well as some of the brain activations observed in the present study were mediated by mental imagery mechanisms or working memory processes. The precuneus and the SPL were reported as involved in spatial mental imagery tasks [57–59], and the IPL and the SPL were also found to be involved in spatial working memory tasks (see [60] for a review). Although the subtraction of the control condition within the sensory substitution experiment was supposed to neutralize working memory aspects involved in the one-back comparison task, we cannot exclude that the experimental conditions (localization and orientation detection) engaged more strongly working memory than the control condition did. In the present study, no activation within the visual cortex was observed during the acoustic association conditions, which brought support to the hypothesis according to which sensory substitution would specifically recruit visual mechanisms [12] and engage different cognitive processes than a regular sound-object association task [5, 10]. The lack of brain modulation in the auditory paradigm was probably due to the fact that subjects used a same strategy in all tasks: the memorization of meaningless associations. Such strategy did not involve location or orientation processing. By contrast, during the sensory substitution conditions, subjects had to convert sounds into visual representations to perform the task based on the resulting “visual” representation they had built mentally [12]. Interestingly, a modulation of the brain activity according to the nature of the percept was previously observed in a study involving the same SSD and that contrasted the processing of 2D versus 3D attributes of the same stimuli [6]. The present study showed up a modulation of the brain activity according to the performed task on the same stimuli. Altogether, this leads us to favor the hypothesis according to which the brain activation patterns typically observed during the use of an SSD combine some general mechanisms related to sensory substitution and some specific processes related to the nature of the perception provided by the device (e.g., as in [6]) and to the nature of the task performed with the device (as presently).


## 4.5. Conclusions

Here we showed that regions in the parietal cortex were specifically recruited during spatial processing tasks both in vision and when using a visual-to-auditory sensory substitution device. In particular, the right superior parietal lobule was more activated during localization compared to orientation detection of a same stimulus in both sensory modalities. This indicates that part of the dorsal visual pathway would be a multisensory operator for spatial processing of information coming from different sensory modalities. Additional work with a greater variety of stimuli and paradigms is clearly needed to test whether other multisensory specialized regions can be found along the processing streams. Another challenge for the future is to connect the findings from sensory substitution studies in a coherent framework that defines the nature of such multisensory interactions in the healthy human and their alteration by visual or auditory deprivation.



---

*Source: 490950-2012-02-06.xml*