# Exploration strategies influence visual-auditory sensory substitution in a passive guidance setup

Salim Akar, Loïc Deschamps, Vincent Roy

<figure><img src="image_1.png" /></figure>

## To cite this version:

Salim Akar, Loïc Deschamps, Vincent Roy. Exploration strategies influence visual-auditory sensory substitution in a passive guidance setup. Acta Psychologica, 2025, 260, pp.105619. 10.1016/j.actpsy.2025.105619. hal-05288612

HAL Id: hal-05288612

https://hal.science/hal-05288612v1

Submitted on 29 Sep 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
# Exploration strategies influence visual-auditory sensory substitution in a passive guidance setup

Salim Akar<sup>a, *</sup>, Loïc Deschamps<sup>a</sup>, Vincent Roy<sup>a</sup>

## ARTICLE INFO

### Keywords:

Sensory substitution

Perception

Passive guidance strategies

Passive exploration

Active exploration

## ABSTRACT

This study explores the origin of perception during passive exploration using a sensory substitution device (SSD) and investigates whether the type of passive guidance strategy influences participants' perception of 2D shapes. Forty-eight participants were divided into four groups, each guided passively with a distinct exploration strategy: macro-scanning (MAC), spatio-temporal (ST), micro-scanning (MIC), or contour exploration (C). After passive exploration, participants transitioned to active exploration with the SSD. Results showed that all participants improved in shape perception, with those exposed to the MIC strategy demonstrating the highest accuracy and confidence levels. This advantage was observed both during passive exploration and when participants actively explored the shapes. These findings suggest that the MIC strategy may have enhanced proprioceptive feedback and sensorimotor contingencies, leading to better perception and a more task-appropriate active exploration strategy adopted by participants. This research emphasizes the importance of passive guidance strategies in optimizing the use of SSDs and provides insights that could inform rehabilitation techniques and assistive technology design.

## 1. Introduction

Sensory substitution devices (SSDs) are defined as systems that aim to assist or replace one or more functions of a sensory organ through another sensory organ. The focus within this domain typically revolves around the creation of visual equivalents using auditory or tactile stimuli. These devices comprise three main components: a sensor, a processing unit that converts visual signals utilizing specific codes and algorithms, and a transmission system that delivers the information in tactile or auditory form (Kristjánsson et al., 2016).

Beyond their practical applications, these devices also contribute to advancing theoretical frameworks related to perception, attention, and brain plasticity (Lloyd-Esenkaya et al., 2020). What makes these interfaces particularly valuable is that they provide an experimental framework for simplifying and controlling human sensorimotor loops (Bach-y-Rita & Kercel, 2003; Froese et al., 2012; Lenay et al., 2003). They enable the transformation and extension of our perceptual abilities, offering novel ways to interact with the environment (Auvray & Myin, 2009; Lenay et al., 1997, 2003).

Based on this, several researchers have investigated how perception

evolves by manipulating the origin of movement during exploration with an SSD (Sangati et al., 2023). For instance, Froese and Ortiz-Garin (2020) developed an experimental setup that allowed two participants to simultaneously perceive the width of segments in one dimension using a shared SSD. One participant, in the active condition, used voluntary motor actions, proprioception, and vibrotactile feedback to explore the stimuli. The second participant, in the passive condition, was physically guided by the movements of the active participant and relied solely on proprioceptive and vibrotactile feedback, without initiating any movement. The study found perceptual improvements in both types of exploration.

Building on this approach, Akar et al. (2025) recently developed a new device to investigate the evolution of 2D shape perception under both active and passive conditions. They designed the PASS (PAntograph for Sensory Substitution), which allows the simultaneous exploration of 2D stimuli by two participants: one in an active condition and the other in a passive one. The active participant was free to move and explore the environment in order to recognize the target shape. In contrast, the passive participant's arm and hand were guided by the movements of their active counterpart. Results indicated that all

<footnote><sup>a</sup> Univ Rouen Normandie, CRFDP UR 7475, F-76000 Rouen, France</footnote>

<footnote>* Corresponding author at: Centre de Recherche sur les Fonctionnements et Dysfonctionnements Psychologiques (CRFDP UR 7475), Université de Rouen Normandie, 76 821, Mont-Saint-Aignan CEDEX, France.</footnote>

<footnote>E-mail address: salim.akar@univ-rouen.fr (S. Akar).</footnote>
participants significantly improved their ability to perceive 2D shapes. Interestingly, when the passive participants were placed in the active condition for the first time, a significant positive correlation was found between their perceptual performance and that of the active participants they had previously been paired with. The authors hypothesized that the exploration strategies employed by the active participants to perceive the shapes may have influenced their passive partners' ability to actively adapt to the device.

Exploration strategies refer to the movement patterns individuals adopt to perceive their environment through active exploration, which combines motor actions (such as hand or body movements) with the reception and processing of sensory feedback (tactile, auditory, or visual; see, for example, Lederman & Klatzky, 1987, for haptic object exploration). They emerge during the learning of new sensorimotor configurations, either through the natural exploration of objects or when using SSDs, enabling users to develop efficient movements for extracting task-relevant information (Noë, 2004; O'Regan & Noë, 2001; Varela et al., 2017). Researchers have classified these strategies according to the type of shape explored, based on visual analyses of perceptual trajectories from successful trials and participants' verbal reports. For instance, in the context of shape perception during active exploration with SSDs, Bermejo et al. (2015) identified two main exploration modes: general exploration, involving large movements crossing the shape from side to side while covering its entire area, and focalized exploration, characterized by small, precise movements targeting the shape's contour. A similar distinction was proposed by Ziat et al. (2007) for the perception of the letter “E” in the Raskin font, distinguishing between global and focalized exploration strategies.

Following this classification, Bermejo et al. (2015) further examined the exploration strategies developed by adult novices when actively using various vision-to-auditory SSDs to recognize simple shapes. They showed that, in general, focalized exploration strategies were more effective than general exploration strategies, and that their effectiveness depended on both the type of SSD and task-related features, such as the properties of the shapes.

A similar observation was made by Ziat et al. (2007), who used TACTOS, a minimalist SSD, and demonstrated the efficacy of focalized exploration strategies for fast learning and accurate perception of contours and letters. These findings were later confirmed in blind participants using the same system (Rovira et al., 2010).

More recently, Zilbershtain-Kra et al. (2021) studied how participants used active exploration strategies with the ASenSub, an active sensing SSD. They found that both sighted and blind participants significantly improved their perceptual performance, thanks to their ability to apply adaptive individual exploration strategies.

This question of how exploration strategies influence perception has primarily been addressed in the context of active exploration. In contrast, research on passive exploration has mainly focused on either investigating the nature of perception (Froese & Ortiz-Garin, 2020; Sangati et al., 2023) or developing rehabilitation methods involving passive guidance of patients' affected limbs to stimulate and improve proprioception (Chiyohara et al., 2020).

To our knowledge, no study has yet investigated the influence of exploration strategies on perceptual improvement during passive exploration. Therefore, we considered it essential to examine how perception evolves under passive conditions in a perceptual task, depending on the exploration strategies imposed. To this end, we used the PASS device, which enables passive exploration of 2D stimuli and allows the application of specific exploration strategies on participants (Akar et al., 2025). We hypothesize that perceptual improvement with the PASS device depends on the exploration strategy applied during passive guidance, with focalized exploration strategies leading to greater perceptual improvement.

Following the passive exploration phase, we assessed how participants adapted their exploration strategies once they became active users of the device, and how this affected their perceptual performance. While

we aimed to explore whether participants previously exposed to strategies associated with high perceptual performance would be more likely to adopt these strategies during active exploration, we did not formulate any specific hypothesis regarding this behavioral adaptation. Likewise, we had no prior hypothesis as to whether such adoption would lead to superior perceptual performance in the active exploration.

## 2. Method

### 2.1. Participants

We tested 48 participants (20 males and 28 females; mean age = 22.7, SD = 3.432), divided into four groups of 12 participants each, with a consistent male-to-female ratio of 5:7 per group. The mean ages across the groups ranged from 21 to 24 years. All participants were recruited from the student network at the University of Rouen, France. They agreed to participate without financial compensation and signed an informed consent form. All participants were right-handed and reported no psychiatric, auditory, or motor disorders. None had prior experience with sensory substitution experiments.

This study received approval from the local ethics advisory committee (CCE No. 2021-10-H), which is competent for reviewing research in psychology.

### 2.2. Apparatus

The PAntograph for Sensory Substitution (PASS) was used. This device consists of an SSD integrated with a pantograph system. The SSD used in this study links a participant's motor actions with auditory feedback based on the position of a tracking point. This is achieved using video-tracking software (ANY-maze, Stoelting, version 7.1) equipped with a webcam to monitor the movement of a contrasted shape, in this case, an LED pen producing a light reflection when directed onto a surface. To obtain optimal contrast, motion tracking was conducted under red light to minimize light interference and ensure precise LED detection. The software triggers a continuous auditory tone (500 Hz, 60 dB) through a loudspeaker when the tracking point enters a predefined ROI (Region of Interest) displayed on the computer screen. If the tracking point leaves this area, the auditory signal stops.

The pantograph system of the PASS consists of six aluminum segments connected by movable joints, with a fixed central point attached to the experiment table. This allows for precise transfer of movements across the pantograph. This system enables testing of users in pairs: one user operates in an active condition, freely moving their hand with the LED pen attached to the pantograph. The other participant operates in a guided passive condition, also holding an LED pen attached to the pantograph, while resting their hand on a support equipped with six nylon ball bearings for smooth hand movement without resistance. The passive participant's hand movement mirrors that of the active participant, guided by the pantograph's motion. Auditory feedback synchronized with the active participant's movements is provided to both users (see Fig. 1).

### 2.3. Experimental procedure

#### 2.3.1. Passive exploration phase

Participants were in a passive condition on the PASS, meaning that their movements were guided by those of the experimenter, who actively controlled the device. Prior to the task, all participants were briefed on the device operation and received a 30s training in the passive condition. They were instructed not to make voluntary arm movements, to keep their eyes closed, and to let the experimenter guide their movements. They were randomly assigned to four groups of 12 participants each. In each group, the experimenter manually applied a single exploration strategy, following a computer display showing the shape's exact location with a tracking point indicating the movement. A single
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** PAntograph for Sensory Substitution (PASS). The pantograph transfers the hand movements of an active user, the experimenter (bottom right), to a passive participant (top left). The experimenter explores the surface using a specific strategy, micro-scanning in this figure, while holding an LED pen attached to the pantograph. This pen creates a light halo of approximately 1 cm in diameter on the table surface. A camera positioned above the experimenter's hand transmits video to a PC running ANY-maze, which tracks the center of the light halo. An auditory signal is emitted through a speaker each time the tracking point enters a predefined region of interest, represented by the mirrored “E” in this figure. The passive participant, also holding an LED pen attached to the pantograph, rests their hand on a ball-bearing support, allowing it to follow the experimenter's movements in real time. This setup enables the experimenter to guide the passive participant according to a specific exploration strategy while ensuring identical sensory feedback at corresponding positions, without voluntary movement from the passive participant.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Examples of shape recognition performance in the passive condition. (a) Examples of shapes presented during each of the three sessions in the passive condition. (b) Participant's drawings of the perceived shape, made with a thick marker after each trial, overlaid on a frame with twelve segments where the shape can be arranged. The goal was to recognize the shapes in (a), with confidence level indicated for each drawing.</figcaption></figure>
experimenter conducted all the sessions and remained the same across all groups.

Each participant completed three exploration sessions (S1, S2, and S3). Each session consisted of eight trials, each lasting 45 s. In each trial, the participant's task was to attempt to recognize a shape composed of five segments, each representing 8 cm on the ground. We chose relatively similar shapes, differing mainly in orientation. This choice was influenced by previous studies involving shapes with different orientations (e.g., Tumbling E) (Ziat et al., 2007), to prevent subjects from using cognitive strategies to recognize the shape without actually perceiving it (such as identifying specific features like a right angle or curvature). In each set of eight trials, four different shapes were presented, each offered twice, without consecutive identical shapes. The shapes were entirely different between sessions.

Participants were shown a frame consisting of twelve segments, within which the five segments of the shape could be arranged. After each trial, participants indicated the perceived shape by drawing its segments in the empty frame on a sheet of paper, with no fixed time for drawing, and rated their confidence level on a scale from 1 to 4 (1 being the lowest and 4 being the highest), based on their conviction of correctly identifying the final shape (see Fig. 2).

### 2.3.1.1. Exploration strategies.
The experimenter passively guided the four participant groups using specific exploration strategies adapted from the experiments of Bermejo et al. (2015) and Ziat et al. (2007), and categorized into four types based on the visual characteristics of their movement trajectories within the frame containing the target shape (Fig. 3).

The criteria used to identify each strategy were as follows:

* **Macro-Scanning (MAC):** A strategy characterized by large, oscillatory vertical and/or horizontal sweeps, crossing the shape from side to side while covering the entire frame, allowing exploration of the full spatial extent of the shape.

* **Spatio-Temporal (ST):** A strategy involving back-and-forth movements between different segments of the shape, alternating between areas containing the shape and empty spaces, providing rhythmic feedback to detect the orientation of the segments relative to one another.

* **Micro-Scanning (MIC):** A focalized strategy involving small, precise sweeps closely following the shape's contour, allowing accurate tracking of the segments while maintaining constant proximity.

* **Contour Exploration (C):** A strategy consisting of continuous tracing along the shape's contour, avoiding deviations outside the shape and allowing complete outline tracing. When surpassing a segment's extremity and moving outside the shape, the movement is reversed to the extremity to regain contact and identify the appropriate direction to continue along the next segment.

Participant groups were labeled with a “P-” prefix to indicate the passive exploration phase, followed by the specific strategy with which they were passively guided (e.g., P-MAC, P-ST, P-MIC, P-C).

To ensure consistent passive guidance across participants within the same group, the experimenter received extensive training on the PASS device during pilot testing. Using the device's coupled webcam, the computer display showing the shape's exact location with a tracking point indicating movement, and the integrated video-tracking software ANY-maze (Stoelting, version 7.1), the experimenter monitored and recorded movement parameters such as average speed, time spent within the shape, and number of entries. Moreover, the experimenter made a deliberate effort to replicate the same movement trajectory and exploration pattern across participants (see Fig. 3). This procedure ensured that participants received consistent movement parameters within each strategy, with variability kept within 10 % standard deviation throughout the passive exploration phase (see Table 1).

### 2.3.2. Active exploration phase

After passive guidance, participants transitioned to active exploration in a single session (S4). Participant groups were labeled with an “A-” prefix to indicate the active exploration phase, followed by the strategy with which they were passively guided (e.g., A-MAC, A-ST, A-MIC, A-C). Before the test, they underwent a 30s training period in the active

<caption>**Table 1**</caption>

This table presents movement parameters that were controlled by the experimenter between participants for each exploration strategy during the passive exploration phase. These parameters include the average movement speed, the average time spent within the shape, and the average number of entries into the shape, along with their standard deviations across participants.

<table><thead><tr><td>Applied strategy</td><td>Average speed (cm/s)</td><td>Average time in ROI (s)</td><td>Average number of entries in ROI</td></tr></thead><tbody><tr><td>MAC</td><td>4.4 (±0.15)</td><td>10.628 (±0.94)</td><td>16.95 (±0.71)</td></tr><tr><td>ST</td><td>2.9 (±0.32)</td><td>13.601 (±1.37)</td><td>20.11 (±1.56)</td></tr><tr><td>MIC</td><td>1.9 (±0.18)</td><td>13.103 (±1.33)</td><td>48.277 (±4.31)</td></tr><tr><td>C</td><td>1.6 (±0.12)</td><td>25.881 (±1.4)</td><td>6.322 (±0.56)</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Examples of the experimenter's movement trajectories for the four exploration strategies, each assigned to a different group in the passive condition (S1-S3), as well as examples of the participants' movement trajectories for each strategy applied in the active condition (S4).</figcaption></figure>
condition. They were instructed to keep their eyes closed while freely navigating the environment. Similar to the passive exploration phase, their task was to recognize a shape composed of five segments. This session comprised eight 45s trials, during which four different shapes, distinct from those in S1-S3, were presented twice each, ensuring no consecutive repetitions. After each trial, participants indicated the perceived shape by drawing its segments in the empty frame on a sheet of paper, with no fixed time for drawing. At the end of the trial, they rated their confidence level on a scale from 1 to 4, based on their conviction of correctly identifying the final shape. Movement trajectories were recorded using the PASS coupled webcam, positioned just above the participant's moving hand, to monitor and analyze their exploration strategies.

## 2.4. EMG recordings

Four participants (two females aged 20 and 22, and two males aged 22 and 26), who were not part of the main experiment, were recruited for EMG recordings in both active and passive conditions. EMG settings and processing were adapted from Beets et al. (2012), who similarly verified that participants in the passive exploration kept wrist muscles relaxed and did not resist or assist imposed movements. We recorded EMG signals from the flexor and extensor carpi radialis muscles while participants interacted with the PASS. In the active exploration phase, participants were free to explore using the PASS voluntarily. In contrast, in the passive exploration phase, their movements were guided by the experimenter using different exploration strategies. Disposable Ag-AgCl surface electrodes (Blue Sensor SP) were placed over the middle portion of the muscle belly, aligned with the muscle's longitudinal axis. Signals were amplified using a MEGA MSPEC 8000 (61000), bandpass filtered between 4 and 500 Hz, and sampled at 1000 Hz using a Power 1404 CED device. Motion signals were simultaneously collected, and the experimenter monitored the data. As shown in Fig. 4, muscle activation levels were lower in the passive condition compared to the active condition, confirming the validity of the experimental setup. This setup ensures that participants are guided using different strategies without requiring active manipulation.

# 3. Data analysis

## 3.1. Shape recognition and confidence levels analysis

Participants' shape recognition of 5-segment shapes was assessed by counting the correctly drawn segments in their correct positions on the frame. Each correct segment earned +0.2 points, with a maximum score of 1/1 for a fully accurate shape. Missing or mispositioned segments

resulted in a deduction of -0.2 points each. Participants who mispositioned more than two segments received a score of 0/1 (see Fig. 5c). Overall shape recognition was then calculated as a percentage for each participant in each session. For confidence levels, participants rated their confidence on a scale from 1 to 4 after each trial, and an average confidence level was computed for each participant and session. Confidence levels were compared within groups across S1 to S3 and between groups during S4.

## 3.2. Exploration strategy analysis

In S4, using the PASS coupled webcam positioned just above the participant's moving hand and the ANY-maze software (Stoelting, version 7.1), movement trajectories were recorded in real time. The number of entries into the virtual shape was calculated automatically based on a predefined ROI displayed on the computer screen. These recordings were also used to assess the exploration strategies employed by participants during the active exploration phase (see Fig. 3). Each trial was visually analyzed based on the specific criteria previously described in the Exploration Strategies section (Section 2.3.1.1). If a single strategy was predominantly applied during a trial (over 75 % of the trial time, at least 35 out of 45 s), that strategy was assigned 1 point. If two strategies were used, the trial was classified as a mixed strategy (see Fig. 5b), with each strategy receiving 0.5 points. Classification was performed by a single researcher. To assess inter-rater reliability, a subset of trajectories representing 16.7 % (64/384) of all trials was independently classified by three researchers. Agreement between raters was 88.89 %, indicating a strong level of reliability. To assess the progression of exploration strategy usage frequency, the eight trials of S4 were divided into four phases (1: trials 1–2, 2: trials 3–4, 3: trials 5–6, and 4: trials 7–8).

## 3.3. Statistical analyses

All statistical analyses were performed using Jamovi (version 2.3.28), with the significance level set at $\alpha = 0.05$. Assumptions of normality (Shapiro-Wilk test), homogeneity of variances (Levene's test), and sphericity (Mauchly's test) were assessed for all relevant data. Depending on whether these assumptions were met, appropriate parametric or non-parametric tests were selected. Graphical representations were created using GraphPad Prism (version 8.0.1).

To compare shape recognition across passive sessions (S1 to S3), we used a mixed-design ANOVA. Significant main or interaction effects were followed by Holm-adjusted pairwise comparisons. Confidence levels across sessions within each group were analyzed using the Friedman test, followed by Durbin-Conover post hoc tests with Holm

<figure><img src="image_6.png" /><figcaption>**Fig. 4.** EMG activity of the flexor and extensor carpi radialis muscles across different conditions. For each condition, the average values were calculated for 4 participants. In the passive condition, the average corresponds to the participant's muscle activity under different exploration strategies (MIC, MAC, and ST) applied by the experimenter. In the active condition, the average reflects the participant's muscle activity while actively engaging in the task. The graphs illustrate how muscle activation evolves across the relaxation, passive, and active conditions.</figcaption></figure>

<figure><img src="image_7.png" /></figure>
<figure><img src="image_8.png" /><figcaption>Fig. 5. Examples of shape recognition performance in the active condition. (a) Example shapes presented during trials 1, 5, and 8 in the active condition. (b) Trajectories of active movement by the blindfolded participant, who freely chose to use the C strategy, a mixed strategy (MIC + C), and the MIC strategy for trials 1, 5, and 8, respectively. (c) Participants' drawings corresponding to the goal of recognizing the shapes in (a) following their active exploration, also indicating their confidence levels. The shape recognition level was calculated based on the scores assigned to the correctly and incorrectly drawn segments.</figcaption></figure>

correction.

During S4, between-group comparisons of shape recognition and confidence levels were performed using one-way ANOVAs, followed by Tukey's post hoc tests. The frequency of exploration strategy use within each group during S4 was analyzed with the Friedman test, followed by Durbin-Conover post hoc tests with Holm correction. Temporal changes in strategy use between the first and last phases of S4 were assessed using Wilcoxon signed-rank tests. Finally, between-group differences in the number of entries into regions of interest (ROI) during S4 were evaluated via one-way ANOVA, followed by Tukey's post hoc tests when appropriate.

# 4. Results

## 4.1. Results of the passive exploration phase (S1–S3)

The analysis of participants' shape recognition revealed a significant interaction between session and group ($F_{(3, 44)} = 2.90, p = 0.012$, partial $\eta^2 = 0.165$). Within group comparisons showed that the P-MAC group performed significantly better in S2 and S3 compared to S1 ($p_{Holm} < 0.001$). A similar pattern was observed in the P-ST group, with significant differences between S1 and S2 ($p_{Holm} < 0.001$) and between S1 and S3 ($p_{Holm} = 0.024$). In the P-MIC group, shape recognition also improved significantly from S1 to S2 ($p_{Holm} = 0.009$) and from S1 to S3 ($p_{Holm} < 0.001$). In contrast, the P-C group demonstrated a significant

improvement only between S1 and S3 ($p_{Holm} = 0.004$) (see Fig. 6a).

The analysis of shape recognition between groups within the same session revealed that in S1, the P-MIC group had significantly higher shape recognition compared to both the P-MAC and P-ST groups ($p_{Holm} = 0.041$ and $p_{Holm} = 0.034$, respectively). In S2, no significant differences in shape recognition were found between groups. However, in S3, significant differences emerged, with the P-MIC group showing significantly higher shape recognition compared to the P-MAC ($p_{Holm} = 0.015$) and P-ST groups ($p_{Holm} < 0.001$). Additionally, the P-C group exhibited significantly higher shape recognition compared to the P-ST group ($p_{Holm} = 0.034$) (see Fig. 6b). Regarding confidence levels, the P-MAC and P-ST groups showed no significant differences across sessions, while the P-MIC and P-C groups exhibited significant variations ($\chi^2(2) = 17.2, p < 0.001$; $\chi^2(2) = 6.2, p = 0.044$, respectively). In both groups, confidence levels were significantly higher in S3 compared to S1 ($p_{Holm} = 0.002$ and $p_{Holm} = 0.007$, respectively; see Fig. 6c).

## 4.2. Results of the active exploration phase (S4)

The analysis of exploration strategy frequencies in S4 revealed that participants predominantly used the strategy to which they had been previously exposed during S1 to S3. Specifically, the A-MAC group mainly used the MAC strategy (45 %), the A-ST group the ST strategy (49 %), the A-MIC group the MIC strategy (82 %), and the A-C group the C strategy (60 %). In the A-MAC group, a significant difference in
<figure><img src="image_9.png" /><figcaption>**Fig. 6.** Shape recognition and confidence levels by session and group in the passive condition. (a) Multiple comparisons of shape recognition between sessions within the same group. (b) Multiple comparisons of shape recognition between groups within the same session. (c) Confidence levels across sessions 1 to 3 within the same group. P-MAC, P-ST, P-MIC, and P-C denote participants in the passive condition. \*p < 0.05, \*\*p < 0.01. Error bars indicate the SEM.</figcaption></figure>

exploration strategy frequencies was found ($\chi^2$ (2) = 12.4, p = 0.006), with the MAC strategy used significantly more than the ST ($p_{Holm}$ = 0.002) and C ($p_{Holm}$ = 0.014) strategies. Similarly, the A-ST group showed a significant difference in strategy use frequencies ($\chi^2$ (2) = 19.3, p < 0.001), with the ST strategy used significantly more often than both the MAC and C strategies ($p_{Holm}$ < 0.001 for both). In the A-MIC group, frequencies also differed significantly ($\chi^2$ (2) = 29.8, p < 0.001), with the MIC strategy used significantly more than the MAC, ST, and C strategies ($p_{Holm}$ < 0.001 for all comparisons). Finally, the A-C group exhibited a significant difference in strategy frequencies ($\chi^2$ (2) = 23.14, p < 0.001), with the C strategy used significantly more often than the MAC ($p_{Holm}$ < 0.001), ST ($p_{Holm}$ < 0.001), and MIC ($p_{Holm}$ = 0.044) strategies (see Fig. 7).

The variation in the application of exploration strategies across the four phases (1, 2, 3, and 4) revealed that for the A-MAC, A-ST, and A-C groups, the frequency of using the predominantly applied strategy (MAC, ST, and C, respectively) decreased across the phases. In contrast, the use of the MIC strategy increased progressively with each phase. As a result, the frequency difference between the predominantly applied strategy and the MIC strategy diminished between phase 1 and phase 4.

In the A-MAC group, the frequency difference between the MAC and MIC strategies decreased from 27.04 % in phase 1 (MAC = 50 %, MIC = 22.92 %) to -6.25 % in phase 4 (MAC = 35.42 %, MIC = 41.67 %). However, this difference was not significant within the same phase. For the A-ST group, the difference reduced from 37.5 % in phase 1 (ST = 64.58 %, MIC = 27.08 %) to -10.42 % in phase 4 (ST = 39.58 %, MIC = 50 %). Again, no significant difference was found within the same phase. In the A-C group, the difference in frequency decreased from 70.84 % in

phase 1 (C = 72.92 %, MIC = 2.08 %) to 2.09 % in phase 4 (C = 47.92 %, MIC = 45.83 %). Although the frequency of C strategy use was significantly higher than that of MIC in phase 1 (p = 0.003), no significant difference was observed between the two strategies in phase 4 (p = 0.915). Participants in the A-MIC group consistently used the MIC strategy at a high frequency (88 ± 2 %) across all phases, in contrast to the other strategies. The frequency of MIC strategy use in this group was significantly higher than that of all other strategies in every phase (p < 0.001 for all comparisons; see Fig. 7).

Additionally, a significant group effect was found on the average number of entries into the ROI (F (3, 44) = 23.5, p < 0.001, partial $\eta^2$ = 0.57). Participants in the A-MIC group entered the shape's ROI significantly more frequently than those in the A-MAC, A-ST, and A-C groups ($p_{Tukey}$ < 0.001 for all comparisons; see Fig. 8).

Finally, the analysis of shape recognition and confidence levels during S4 revealed a significant group effect on both measures (shape recognition: F (3, 44) = 4.45, p = 0.008, partial $\eta^2$ = 0.233; confidence: F (3, 44) = 4.38, p = 0.009, partial $\eta^2$ = 0.23). The A-MIC group demonstrated significantly higher shape recognition and confidence levels than the A-MAC, A-ST, and A-C groups ($p_{Tukey}$ < 0.05 for all comparisons; see Fig. 9).

## 5. Discussion

The primary objective of our study was to investigate the influence of passive exploration strategies on perceptual improvement using an SSD. Forty-eight participants were divided into four groups, each assigned to a different exploration strategy imposed by the experimenter.
<figure><img src="image_10.png" /><figcaption>**Fig. 7.** The frequency of exploration strategies applied by participants across the active condition. The histograms show the frequencies of exploration strategy usage across the entire active condition, while the graphs represent the progression of exploration strategy choice. Trial phases are divided as follows: phase 1 (Trials 1–2), phase 2 (Trials 3–4), phase 3 (Trials 5–6), and phase 4 (Trials 7–8), with each phase consisting of two trials. Each data point represents the average strategy use across all participants. A-MAC, A-ST, A-MIC, and A-C denote participants in the active condition. \*p < 0.05, \*\*p < 0.01, \*\*\*p < 0.001. Error bars represent the SEM.</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>**Fig. 8.** Multiple comparisons of the number of entries into the ROI of the shape by participants from each group (A-MAC, A-ST, A-MIC, A-C) in the active condition. \*p < 0.05, \*\*p < 0.01, \*\*\*p < 0.001. Error bars represent the SEM.</figcaption></figure>

Participants were shown a frame composed of 12 segments and informed that the shape consisted of 5 connected segments within this frame. This approach enables participants to develop a specific intention, thereby focusing their attention on the shape and enhancing perceptual accuracy (Sangati et al., 2023).

The analysis of perceptual performance across the passive exploration phase revealed that all participants exhibited significant improvements in shape perception, regardless of the exploration strategy employed. Recognition rates increased significantly between the first and third passive exploration sessions. Similar improvements across passive sessions were also reported by Akar et al. (2025), who found a significant main effect of session, and by Froese and Ortiz-Garin (2020), who demonstrated that perception can be enhanced through passively experienced sensorimotor loops. These results echo the findings of Van Doorn et al. (2012), who argued that perception during passive guidance is not only possible but may even be optimized, as it reduces both cognitive and motor demands, thereby enabling more direct processing of sensory signals within somatosensory and associative brain regions.

Upon comparing shape recognition rates across groups, each passively guided using a different exploration strategy, we found a significant effect of strategy on perceptual performance. The group exposed to the MIC strategy exhibited higher recognition rates than the others from the first session. A possible explanation for these results is that the MIC strategy is characterized by oscillatory movements along the shape's contour, which led to a higher number of entries into the shape

compared to the other strategies used in this study. This results in a frequent alternation between auditory feedback and silence, thereby providing more movement-dependent auditory feedback and enhancing sensorimotor contingencies. Sensorimotor contingencies refer to the systematic relationship between an individual's movements and the resulting variations in sensory stimulation. According to the sensorimotor account of perception, strengthening sensorimotor contingencies helps to structure, sustain, and enhance perception (Noë, 2004; O'Regan & Noë, 2001).

Another hypothesis is that the MIC group may have received more relevant proprioceptive feedback, which could have facilitated the perception of arm position and movement, thereby improving the spatial attention necessary for perception in our task. Fuentes and Basťian (2010) examined how proprioception varies with arm configuration and task demands. Their study showed that limb position perception is more accurate when participants use their hand position as a reference rather than the angle of their elbow. They attributed this to the higher density of proprioceptive receptors in the fingers, which provide richer sensory feedback to the brain compared to the elbow. We estimate that in our study, the MIC strategy involved more hand and wrist movements compared to other strategies, such as MAC, ST, or SC, which primarily engage the forearm and elbow. This difference may have kept participants' attention in the MIC group more focused on the spatial movement of the hand and wrist, potentially providing them with more detailed proprioceptive feedback.

However, our results should be interpreted with caution, as the exploration strategies were applied by a human rather than a programmed system. The same experimenter guided all participants across and within groups. Prior to the experiment, extensive training on the PASS device was conducted using ANY-maze (v7.1, Stoelting) to monitor and standardize movement trajectories for each strategy, ensuring consistent speed, duration within shapes, and number of entries across trials. Nevertheless, since the guidance was provided by a human arm, some inherent variability remained. Although the standard deviation across participants stayed within 10 %, the impact of this variability on participants' perceptual performance remains unclear.

We are not the first to report that the MIC strategy is associated with enhanced perceptual performance. However, this association has primarily been observed in the context of active exploration. For instance, Bermejo et al. (2015) evaluated exploration strategies used to recognize geometric shapes with various SSDs and found that a focalized strategy involving micro-scanning along the object's contour was not only the most frequently employed but also the most effective to perceive. They proposed that this strategy is a preferred mode of exploration for extracting geometric information, as it establishes a consistent sensorimotor regularity between the agent and the object. This movement

<figure><img src="image_12.png" /><figcaption>**Fig. 9.** Multiple comparisons of shape recognition and confidence levels of participants from each group (A-MAC, A-ST, A-MIC, A-C) in the active condition. \*p < 0.05. Error bars represent the SEM.</figcaption></figure>
pattern facilitates rapid attentional shifts toward relevant features of the shape. Similarly, in a study using the TACTOS system, Ziat et al. (2007) found that a strategy involving micro-scanning was effective for tactile recognition of contours and letters, resulting in consistent and precise performance. Their findings indicated that this strategy reduces perception errors and adapts well to shapes of varying sizes.

In relation to natural visual perception, it is interesting to note that the MIC strategy, which primarily consists of a series of straight movements, has been described as similar to eye saccades (Bermejo et al., 2015). This resemblance between movements made using SSDs and visual saccades has also been demonstrated in previous studies (Chekhchoukh et al., 2013; Milne et al., 2014; Ward & Wright, 2014). This idea aligns with the framework proposed by O'Regan and Noë (2001), who argued that perception through an SSD is shaped by replicating exploration strategies analogous to the intrinsic patterns of the sensory modality being replaced.

Following the passive exploration phase, the second objective of our study was to examine how participants adapted their exploration strategies once they became active users of the device, and how this adaptation impacted their perceptual performance. This was assessed during a fourth and final session (S4) with the SSD, in which participants were allowed to use the device freely and actively. Our results revealed that, across all four groups, the exploration strategy encountered during the passive phase clearly influenced subsequent active behavior. Specifically, the dominant strategy adopted in S4 by each group reflected the one they had been exposed to during passive exploration. For example, participants trained with the MAC strategy exhibited a higher frequency of MAC usage during active exploration compared to the other groups. A similar pattern was observed for the ST, MIC, and C groups, whose dominant strategy in S4 matched the one previously experienced passively. This tendency to reproduce the passively experienced strategy echoes the findings of Wong et al. (2013), who demonstrated that humans form bidirectional mappings between motor actions and sensory feedback. That is, whether a movement is passively experienced or merely imagined, individuals not only perceive the associated sensation but also tend to reproduce the movement pattern.

When tracking the exploration strategies adopted by participants throughout the trials of S4, we found that those who had been passively exposed to the MIC strategy almost exclusively continued to use this strategy across all trials. In contrast, participants from the other groups A-MAC, A-ST, and A-C tended to alternate between different strategies. Interestingly, within these groups that were not initially exposed to the MIC strategy, we observed an overall decrease of approximately 20 % in the use of their dominant strategy between the first and last quartiles of S4 trials relative to other strategies. In parallel, there was an increase of about 20 % in the use of the MIC strategy for the A-MAC and A-ST groups, and an increase of about 50 % for the A-C group. Although these changes did not reach statistical significance, extending the active exploration phase across more sessions could clarify these trends and help determine if they become significant in a larger study.

A comparable observation was reported by Bermejo et al. (2015). In one of their experiments, participants were asked to perceive shapes composed of connected segments using a minimalist SSD. The authors tracked the progressive evolution of exploration strategies during the task. They found that participants generally started with broad exploration to form an overall impression, then gradually focused on specific details. During the first quarter of trials (trials 1 to 5), exploration strategies consisting of large movements that explored either the entire object or only parts, accounted for 56.2 % of usage, while the focalized exploration strategy involving micro-scanning significantly increased over time, becoming the predominant strategy (approximately 43.3 %) during the last quarter of trials (trials 16 to 20). According to the authors, this shift likely reflects dynamic sensory learning and optimization of exploration based on perceptual needs.

In our study, this pattern of shifting strategies was not observed in the group that was passively exposed to the MIC strategy. They

maintained this strategy almost exclusively throughout all trials in S4. We suggest that these participants, having achieved near-perfect perception scores at the end of the passive phase (82 %), retained the MIC strategy because it best suited the perceptual demands of the task. Furthermore, we suggest that the continued use of the MIC strategy during active exploration likely contributed to their significantly higher perceptual performance and confidence levels compared to all other groups in S4. In contrast, participants not passively exposed to the MIC strategy used a mix of exploration strategies during S4, likely to optimize their exploration according to the task's perceptual requirements.

These results offer promising perspectives for future research aimed at improving perception through SSDs. In particular, incorporating a task-adapted passive guidance phase may facilitate perceptual learning and promote a more effective appropriation of the device once participants become active users. This approach could be especially valuable given that perception through SSDs is known to emerge only after long periods of training (Maidenbaum et al., 2014) and often remains frustetringly slow even after extensive practice (Amedi et al., 2007). Moreover, these findings could have important implications for motor rehabilitation, where perception tasks using devices are being developed alongside passive guidance techniques with robotic or therapeutic assistance. Such approaches could support patients with limited ability to actively explore their environment, helping to improve perception while engaging proprioception, which is crucial for motor recovery (Yeh et al., 2021).

An important aspect of our study is that we used an SSD involving a transformation from vision to audition, which constitutes a cross-modal mapping. Such mappings typically require more extensive training and tend to rely more heavily on cognitive strategies, as they do not benefit from pre-existing sensorimotor couplings within a single modality (Ahissar & Assa, 2016). Moreover, our design excluded active sensing, since participants did not control the movement of the stimulated organ. This combination likely reduced the involvement of automatic, low-level motor-sensory loops and increased reliance on slower, cognitive processing (Ahissar & Assa, 2016). For example, Zilbershtain-Kra et al. (2021) developed an SSD based on active sensing, where participants actively explored with tactile feedback aligned to their hand movements. This closed-loop approach allowed the use of natural sensorimotor routines and greatly shortened the training time. Perception in their setup appeared to emerge from low-level motor-sensory loops without conscious planning, resulting in more precise and efficient use. An interesting avenue for future research would be to use an SSD based on active sensing in a passive guidance context, to examine whether, as in active exploration, it could improve perceptual accuracy and facilitate device appropriation.

It is important to acknowledge that our study did not account for interindividual variability. Mizrachi et al. (2022) demonstrated, in a tactile object perception task, that exploration strategies adapt to individual physiological parameters, such as spatial threshold (the smallest detectable difference by touch) and temporal threshold (related to tactile receptor adaptation time). Similarly, Arnold et al. (2017) highlighted that SSD users vary across multiple dimensions. In particular, SSD performance may depend on individual sensory and cognitive abilities, including auditory and visuospatial skills, as well as factors such as engagement and motivation. Subjective experiences also differ based on the task and user profile, indicating variability in how the device is perceived and integrated. Future research should consider these individual differences to better understand how perception develops uniquely for each participant.

## CRediT authorship contribution statement

**Salim Akar:** Writing – review & editing, Writing – original draft, Visualization, Validation, Project administration, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. **Loïc Deschamps:** Writing – review & editing, Visualization, Validation,
Supervision, Investigation, Conceptualization. Vincent Roy: Writing – review & editing, Visualization, Validation, Supervision, Investigation, Conceptualization.

## Acknowledgments

The authors would like to thank Virginie Beaucousin for her valuable assistance in the training and execution of the EMG measurements.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Data availability

All data related to participant performance on each trial, EMG measurements, and examples of participants' drawings and performances are available at https://doi.org/10.57745/OGGRMP.

## References

Ahissar, E., & Assa, E. (2016). Perception as a closed-loop convergence process. *eLife*, 5, Article e12830. https://doi.org/10.7554/eLife.12830

Akar, S., Beaucousin, V., Velin, L., Lenay, C., Deschamps, L., & Roy, V. (2025). Visual-to-auditory sensory substitution with passive movements in a double participant setup. *Quarterly Journal of Experimental Psychology*, , Article 17470218251334990. https://doi.org/10.1177/17470218251334990

Amedi, A., Stern, W. M., Camprodon, J. A., Bemmpohl, F., Merabet, L., Rotman, S., ... Pascual-Leone, A. (2007). Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. *Nature Neuroscience*, 10(6), 687–689. https://doi.org/10.1038/nn1912

Arnold, G., Pesnot-Lerousseau, J., & Auvray, M. (2017). Individual differences in sensory substitution. *Multisensory Research*, 30(6), 579–600. https://doi.org/10.1163/22134808-00002561

Auvray, M., & Myin, E. (2009). Perception with compensatory devices: From sensory substitution to sensorimotor extension. *Cognitive Science*, 33(6), 1036–1058. https://doi.org/10.1111/j.1551-6709.2009.01040.x

Bach-y-Rita, P., & Kercel, S. W. (2003). Sensory substitution and the human-machine interface. *Trends in Cognitive Sciences*, 7(12), 541–546. https://doi.org/10.1016/j.tics.2003.10.013

Beets, I. A. M., Macé, M., Meesen, R. L. J., Cuypers, K., Levin, O., & Swinnen, S. P. (2012). Active versus passive training of a complex bimanual task: Is prescriptive proprioceptive information sufficient for inducing motor learning? *PLoS One*, 7(5), Article e37687. https://doi.org/10.1371/journal.pone.0037687

Bermejo, F., Di Paolo, E. A., Hüg, M. X., & Arias, C. (2015). Sensorimotor strategies for recognizing geometrical shapes: A comparative study with different sensory substitution devices. *Frontiers in Psychology*, 6. https://doi.org/10.3389/fpsyg.2015.00679

Chekhchoukh, A., Vuillerme, N., Payan, Y., & Glade, N. (2013). Effect of saccades in tongue electrotactile stimulation for vision substitution applications. In *2013 35th annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC)* (pp. 3543–3546). https://doi.org/10.1109/EMBC.2013.6610307

Chiyohara, S., Furukawa, J.-I., Noda, T., Morimoto, J., & Imamizu, H. (2020). Passive training with upper extremity exoskeleton robot affects proprioceptive acuity and performance of motor learning. *Scientific Reports*, 10(1), Article 11820. https://doi.org/10.1038/s41598-020-68711-x

Froese, T., McGann, M., Bigge, W., Spiers, A., & Seth, A. K. (2012). The Enactive Torch: A new tool for the science of perception. *IEEE Transactions on Haptics*, 5(4), 365–375. https://doi.org/10.1109/TOH.2011.57

Froese, T., & Ortiz-Garin, G. U. (2020). Where is the action in perception? An exploratory study with a haptic sensory substitution device. *Frontiers in Psychology*, 11. https://doi.org/10.3389/fpsyg.2020.00809

Fuentes, C. T., & Bastian, A. J. (2010). Where is your arm? Variations in proprioception across space and tasks. *Journal of Neurophysiology*, 103(1), 164–171. https://doi.org/10.1152/jn.00494.2009

Kristjánsson, Á., Moldoveanu, A., Jóhannesson, Ó. I., Balan, O., Spagnol, S., Valgeirsódtírr, V. V., & Unnthorsson, R. (2016). Designing sensory-substitution devices: Principles, pitfalls and potential 1. *Restorative Neurology and Neuroscience*, 34(5), 769–787. https://doi.org/10.3233/RNN-160647

Lederman, S. J., & Klatzky, R. L. (1987). Hand movements: A window into haptic object recognition. *Cognitive Psychology*, 19(3), 342–368. https://doi.org/10.1016/0010-0285(87)90008-9

Lenay, C., Canu, S., & Villon, P. (1997). Technology and perception: The contribution of sensory substitution systems. In, *Vol. 0. Proceedings of the international conference on cognitive technology* (p. 53). https://doi.org/10.1109/CT.1997.617681

Lenay, C., Gapenne, O., Hanneton, S., Marque, C., & Genouëlle, C. (2003). Sensory substitution: Limits and perspectives. In *Touching for knowing: Cognitive psychology of haptic manual perception* (pp. 276–292). John Benjamins Publishing Company. https://doi.org/10.1075/aicr.53.22len

Lloyd-Esenkaya, T., Lloyd-Esenkaya, V., O'Neill, E., & Proulx, M. J. (2020). Multisensory inclusive design with sensory substitution. *Cognitive Research: Principles and Implications*, 5(1), 37. https://doi.org/10.1186/s41235-020-00240-7

Maidenbaum, S., Abboud, S., & Amedi, A. (2014). Sensory substitution: Closing the gap between basic research and widespread practical visual rehabilitation. *Neuroscience & Biobehavioral Reviews*, 41, 3–15. https://doi.org/10.1016/j.neubiorev.2013.11.007

Milne, J. L., Goodale, M. A., & Thaler, L. (2014). The role of head movements in the discrimination of 2-D shape by blind echolocation experts. *Attention, Perception & Psychophysics*, 76(6), 1828–1837. https://doi.org/10.3758/s13414-014-0695-2

Mizrachi, N., Nelinger, G., Ahissar, E., & Arieli, A. (2022). Idiosyncratic selection of active touch for shape perception. *Scientific Reports*, 12(1), 2922. https://doi.org/10.1038/s41598-022-06807-2

Noë, A. (2004). *Action in perception*. Cambridge, MA: MIT Press.

O'Regan, J. K., & Noë, A. (2001). A sensorimotor account of vision and visual consciousness. *The Behavioral and Brain Sciences*, 24(5), 939–973. https://doi.org/10.1017/s0140525x01000115 (discussion 973-1031).

Rovira, K., Gapenne, O., & Ammar, A. A. (2010). Learning to recognize shapes with a sensory substitution system: A longitudinal study with 4 non-sighted adolescents. In *2010 IEEE 9th international conference on development and learning* (pp. 1–6). https://doi.org/10.1109/DEVLRN.2010.5578875

Sangati, E., Lobo, L., Estelle, S., Sangati, F., Tavassoli, S., & Froese, T. (2023). Uncovering the role of intention in active and passive perception. *Proceedings of the Annual Meeting of the Cognitive Science Society*, 45(45). https://escholarship.org/uc/item/6wj762mq.

Van Doorn, G. H., Dubaj, V., Wuillemin, D. B., Richardson, B. L., & Symmons, M. A. (2012). Cognitive load can explain differences in active and passive touch. In P. Isokoski, & J. Springare (Eds.), *Haptics: Perception, devices, mobility, and communication* (pp. 91–102). Springer. https://doi.org/10.1007/978-3-642-31401-8_9.

Varela, F. J., Thompson, E., & Rosch, E. (2017). *The embodied mind: Cognitive science and human experience*. The MIT Press. https://doi.org/10.7551/mitpress/978026259365.001.0001

Ward, J., & Wright, T. (2014). Sensory substitution as an artificially acquired synaesthesia. *Neuroscience & Biobehavioral Reviews*, 41, 26–35. https://doi.org/10.1016/j.neubiorev.2012.07.007

Wong, L., Manson, G. A., Tremblay, L., & Welsh, T. N. (2013). On the relationship between the execution, perception, and imagination of action. *Behavioural Brain Research*, 257, 242–252. https://doi.org/10.1016/j.bbr.2013.09.045

Yeh, I.-L., Holst-Wolf, J., Elangovan, N., Cuppone, A. V., Lakshminarayan, K., Cappello, L., ... Konczak, J. (2021). Effects of a robot-aided somatosensory training on proprioception and motor function in stroke survivors. *Journal of Neuroengineering and Rehabilitation*, 18(1), Article 77. https://doi.org/10.1186/s12984-021-00871-x

Ziat, M., Gapenne, O., Stewart, J., & Lenay, C. (2007). Haptic recognition of shapes at different scales: A comparison of two methods of interaction. *Interacting with Computers*, 19(1), 121–132. https://doi.org/10.1016/j.intcom.2006.07.004

Zilbershtain-Kra, Y., Graffi, S., Ahissar, E., & Arieli, A. (2021). Active sensory substitution allows fast learning via effective motor-sensory strategies. *iScience*, 24(1), Article 101918. https://doi.org/10.1016/j.isci.2020.101918