# Paving the way for social touch at a distance: Sonifying tactile interactions and their underlying emotions

Alexandra de Lagarde, Catherine Pelachaud, Louise P Kirsch, Malika Auvray

► To cite this version:

Alexandra de Lagarde, Catherine Pelachaud, Louise P Kirsch, Malika Auvray. Paving the way for social touch at a distance: Sonifying tactile interactions and their underlying emotions. Proceedings of the National Academy of Sciences of the United States of America, 2025, 122 (19), 10.1073/pnas.2407614122. hal-05069633

HAL Id: hal-05069633

https://hal.sorbonne-universite.fr/hal-05069633v1

Submitted on 15 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Paving the way for social touch at a distance: sonifying tactile interactions and their underlying emotions

Alexandra de Lagarde<fnref n="1" />,<fnref n="2" />*, Catherine Pelachaud<fnref n="1" />, Louise P. Kirsch<fnref n="3" />§* & Malika Auvray<fnref n="1" />§*

§. Shared senior authorship.

Alexandra de Lagarde

E-mail: delagarde.alexandra@gmail.com

Louise P. Kirsch

E-mail: louise.kirsch@u-paris.fr

Malika Auvray

E-mail: malika.auvray@sorbonne-universite.fr

**Author Contributions:** M.A., L.P.K., and C.P. conceived and designed the experiment. A.D.L. collected the data. L.P.K. and A.D.L analyzed the data and created the figures. A.D.L. wrote the manuscript under the guidance of M.A, L.P.K., and C.P. All authors revised and approved the final version of the manuscript.

**Competing Interest Statement:** The authors declare no competing interests.

**Preprint:** The first submitted (non-peer-reviewed version) of this manuscript was published as preprint on PsyArXiv. https://osf.io/preprints/psyarxiv/mchzy_v1 under the CC-BY Attribution 4.0 International Licence.

**Keywords:** social touch, movement sonification, emotions, augmented perception

<footnote n="1">¹ Sorbonne Université, CNRS, Institut des Systèmes Intelligents et de Robotique, ISIR, F-75005 Paris, France</footnote>

<footnote n="2">² Humans Matter, Paris, France</footnote>

<footnote n="3">³ Integrative Neuroscience and Cognition Center, CNRS, UMR 8002, Université Paris Cité, Paris, France</footnote>

<footnote>* Corresponding authors</footnote>
**Abstract**

Social touch is crucial for human well-being, as a lack of tactile interactions increases anxiety, loneliness, and need for social support. To address the detrimental effects of social isolation, we build on cutting-edge research on social touch and movement sonification to investigate whether social tactile gestures could be recognized through sounds, a sensory channel giving access to remote information. Four online experiments investigated participants’ perception of auditory stimuli that were recorded with our ‘audio-touch’ sonification technique, which captures the sounds of touch. In the first experiment, participants correctly categorized sonified skin-on-skin tactile gestures (i.e., stroking, rubbing, tapping, hitting). In the second experiment, the audio-touch sample consisted of the sonification of six socio-emotional intentions conveyed through touch (i.e., anger, attention, fear, joy, love, sympathy). Participants categorized above chance the socio-emotional intentions of skin-on-skin touches converted into sounds and coherently rated their valence. In two additional experiments, the surface involved in the touches (either skin or plastic) was shown to influence participants’ recognition of sonified gestures and socio-emotional intentions. Thus, our research unveils that specific information about social touch (i.e., gesture, emotions, and surface) can be recognized through sounds, when they are obtained with our specific sonifying methodology. This shows significant promise for providing remote access, through the auditory channel, to meaningful social touch interactions.

**Significance**

We present a novel approach combining findings from research on social touch and on movement sonification, allowing us to extract meaningful characteristics from tactile interactions and convert them into sounds. In our experimental studies, participants listening to our skin-on-skin audio-touch stimuli accurately associated sounds with specific gestures and coherently identified socio-emotional intentions underlying touches converted into sounds. Moreover, the same movements were recorded with inanimate objects, and further experiments revealed that participants’ perception is shaped by the surfaces involved in the tactile interactions (skin vs. plastic). This novel audio-touch approach may be the steppingstone to giving access to the ineffable experience of social touch at a distance, with human or virtual agents.
## Introduction

Touch is the first sense to develop and remains essential throughout life for development and well-being (1–3). As the world becomes increasingly connected through virtual means, there is a growing need to integrate tactile interactions into distant communication modalities. This is all the more important that in situations of social isolation, the absence of tactile interactions has been linked to detrimental mental health effects, including increased anxiety and loneliness (4). In line with this need, the present study poses a stepping stone to allow tactile interactions at a distance by the necessary combination of state-of-the-art research from two research domains: social touch and movement sonification.

Research has evidenced that touch is one of the most effective non-verbal communication channels together with body gesture, especially to convey affective content (5, 6). Not only can the intention underlying tactile interactions be naturally understood by humans (7), but many intentions and emotions can be conveyed through specific tactile encounters (8, 9). As such, distinct types of tactile gestures, as well as their underlying affective intentions, can be recognized through touch. For instance, slow gentle stroking on the forearm is identified as the most pleasant and relaxing touch (10) and recognized as communicating love and social support (11). Moreover, Hertenstein et al. (6, 12) evidenced that people can differentiate basic emotions (e.g., sadness, anger) as well as prosocial ones (e.g., love, sympathy), when blindly receiving various touches on their forearm performed by a naive stranger. Thompson & Hampton (13) replicated Hertenstein et al.'s results with romantically involved couples, who were shown to be able to additionally convey three self-focused emotions: envy, embarrassment, and pride.

Following Hertenstein et al.'s methodology, McIntyre et al. (14) have defined a set of prototypical tactile gestures that are often spontaneously used in socio-affective contexts, to communicate different intentions or emotions (i.e., standardized tactile contacts such as a comforting caress or an attention-seeking poking). Features such as velocity of touch, duration, and points of contact on the skin as well as rhythm of the tactile interactions, were carefully reported and associated with certain tactile emotions (see Table 1 for an overview). Indeed, the authors precisely pinpointed the physical features of tactile interactions between strangers using only touch to convey love, calming, happiness, gratitude, sadness, and attention.

<table><caption>**Table 1.** Overview of studies investigating socio-emotional intentions recognition and their corresponding prototypical tactile gestures.</caption><thead><tr><td>Emotion</td><td>Tactile gesture</td><td>Velocity</td><td>Pressure</td><td>Tactile continuity</td><td>Points of contact</td><td>Rhythm</td><td>Reference</td></tr></thead><tbody><tr><td>Love</td><td>Stroke</td><td>Slow (6 cm/sec)</td><td>Light</td><td>No</td><td>All fingers</td><td>Regular</td><td>Hertenstein et al. (2009)<br>McIntyre et al. (2022)</td></tr><tr><td>Sympathy</td><td>Rub Pat</td><td>Moderate (12 cm/sec)</td><td>Light</td><td>Yes</td><td>Palm and all fingers</td><td>Regular</td><td>Hertenstein et al. (2009)</td></tr><tr><td>Calming</td><td>Stroke Hold</td><td>Slow (6 cm/sec)</td><td>Light</td><td>No</td><td>Palm and all fingers</td><td>Regular</td><td>McIntyre et al. (2022)</td></tr><tr><td>Joy</td><td>Tap</td><td>Fast</td><td>Light</td><td>No</td><td>Tip of fingers</td><td>Irregular</td><td>McIntyre et al. (2022)</td></tr></tbody></table>
<table><thead><tr><th>Emotion</th><th>Tactile gesture</th><th>Velocity</th><th>Pressure</th><th>Tactile continuity</th><th>Points of contact</th><th>Rhythm</th><th>Reference</th></tr></thead><tbody><tr><td>Gratitude</td><td>Stroke Hold Squeeze</td><td>Slow (6 cm/sec)</td><td>Light</td><td>No</td><td>Palm and all fingers</td><td>Regular</td><td>McIntyre et al. (2022)</td></tr><tr><td>Sadness</td><td>Hold Stroke</td><td>Slow</td><td>Light</td><td>Yes</td><td>Palm and all fingers</td><td>Regular</td><td>McIntyre et al. (2022)</td></tr><tr><td>Fear</td><td>Tremble</td><td>Fast (18 cm/sec)</td><td>Moderate</td><td>Yes</td><td>Palm and all fingers</td><td>Irregular</td><td>Hertenstein et al. (2009)</td></tr><tr><td>Anger</td><td>Hit Squeeze</td><td>Fast</td><td>Strong</td><td>No</td><td>Palm</td><td>Regular</td><td>Hertenstein et al. (2009)</td></tr><tr><td>Attention</td><td>Tap Shake</td><td>Fast</td><td>Moderate</td><td>No</td><td>Tip of fingers</td><td>Regular</td><td>McIntyre et al. (2022)</td></tr><tr><td>Disgust</td><td>Push</td><td>Fast</td><td>Strong</td><td>No</td><td>Palm</td><td>Regular</td><td>Hertenstein et al. (2009)</td></tr></tbody></table>

The articles included in the table respect several criteria: only articles published in peer-reviewed journals in English, in which participants recognized the socio-emotional intentions conveyed by the toucher above chance, and which characterized the physical features of the tactile gestures used to convey each socio-emotional intention were considered. Only gestures involving the hand of the person touching and the arm of the person being touched were kept (i.e., hugging, nuzzling, and other tactile behaviors reported in the literature were excluded based on this criterion). Tactile continuity refers to whether the gesture involves continuous contact between the touching body and the body being touched.

Different skin-on-skin tactile interactions specifically produce distinct tactile vibrations (i.e., tactile waves), which convey information on the level of applied pressure and the velocity of the specific touch that is received (15). Moreover, the vibrations recorded in the hand or on the wrist that is touched seems to contain information in their signals similar to those obtained from the direct exploration of various textures with the fingertips. Indeed, it has been shown that the texture or roughness of a surface can be accurately decoded by participants perceiving vibrations on their skin, due to the replication of the force of interaction between the fingertips and a surface (16), even when the recording happens further away from the points of contact (17, 18). Therefore, capturing vibrations on the touched body part may remain indicative of the characteristics of the performed gesture by the person touching, such as the type of the gesture, its speed, and its pressure. The possibility to record meaningful vibrations produced by touch opens avenues to (i) broaden the repertoire of suitable measuring instruments, as recording systems no longer need to be confined to the fingertips of the person touching, and (ii) improve the signal with details about the physical characteristics of the touched area.

Sonification, the process of translating information into a sound, and specifically movement sonification, is an interesting method to translate gestures and interactions into sounds. In this field, auditory content can be created by recording the vibrations produced by contact or friction between two materials, and then be replayed to provide auditory feedback accompanying participants doing specific gestures. However, even if this method is as ecological as possible, the contact or friction is not chosen randomly. For
instance, in Tajadura et al. (19), movements’ speed, pressure, texture, and even meaning were sonified, by
pre-recording the audio signal produced by different interactions with a cardboard. Tajadura et al.’s study
suggests that it is the precise motion-to-sound mapping which allows the accurate transformation in terms
of acoustics, and also in terms of perception and emotional responses. Regarding the latter, Crommett et al.
(20) suggested that the perception of tactile frequency actually depends on the neural circuits that also
process auditory frequency. Interestingly, the vibrations created by skin-on-skin touches are of similar nature
to auditory ones, hence the information does not necessarily need to be transformed. In Ramasamy et al.
(21), extracting acoustic and kinematic features from the vibratory signal produced by skin-on-skin touch
allowed a highly accurate classification of self- versus other-touch via a machine learning approach. Surface
dynamics, like hardness and roughness, also significantly impact interaction sonification, as shown by
Tajadura et al. (19) and by Petiot et al. (22), who recorded skin-on-skin sliding gestures on the receiving end
of touches. These works highlight the unique acoustic properties of human skin and the potential for
sonifying socio-emotional tactile interactions. Yet, to date, no studies have sonified different types of skin-
on-skin tactile interactions to convey specific socio-emotional tactile intentions.

To fill this gap, the 'audio-touch' methodology presented here combines insights from social touch
research and movement sonification to investigate whether social tactile gestures can be recognized through
sounds. The methodology consisted of recording vibrations from specific tactile interactions using a
piezoelectric transducer and processing these signals into sounds. Based on the prototypical gestures defined
in the literature on social touch, four distinct sets of audio-touch stimuli were recorded with the apparatus
presented in Figure 1.A. The obtained stimuli were then played back to participants via headphones in four
distinct online experiments, testing their perception of various dimensions of the sonified tactile interactions.
More specifically, Experiment 1 hypothesized that participants can recognize the four types of skin-on-skin
gestures (i.e., rubbing, stroking, tapping, hitting) which were sonified. Experiment 2 hypothesized that
participants can identify the six socio-emotional intentions underlying skin-on-skin gestures meant to convey
sympathy, joy, love, anger, fear, and attention. In Experiments 3 and 4, the same tasks and the same gestures
as in Experiments 1 and 2 were used, but this time, audio-touch stimuli were recorded with three different
skins and with three inanimate objects (aluminum on plastic). In these Experiments 3 and 4, participants
were expected to categorize differently gestures and socio-emotional intentions depending on the surface
involved in the stimuli on the one hand, and to accurately identify the surfaces on the other hand. Overall,
the present study aims to determine which components of touch (i.e., type of gestures, socio-emotional
intentions, and surfaces) can be recognized through sound and whether surfaces influence their perception.
<figure><img src="image_2.png" /><figcaption>**B. Audio-touch stimuli spectrograms from Experiment 1**</figcaption></figure>

<figure><img src="image_3.png" /></figure>

**Figure 1.** **A. Audio-touch stimuli sonification set-up.** The violin microphone wires were taped onto the ventral skin of the left forearm of the experimenter, between the elbow and the wrist. A 6-centimeter-long perimeter covering the left and the right fields of the violin's auditory recording delimited the area of tactile interaction, which was performed with the experimenter's right hand. **B. Examples of auditory raw spectrograms** expressing amplitude in frequency (Hz) over time in seconds (sampling frequency: 44100 Hz). They were produced by different tactile gestures recorded via the violin microphone and post-processed via Audacity: rubbing, tapping, hitting, and stroking. Spectrograms for all four experiments can be found in the supplementary materials (Figure S1, S2, S4, and S6). Each stimulus has a 5 seconds duration.

# Results

## Experiment 1. Perception of sonified skin-on-skin tactile gestures.

In a first online experiment, participants (N=51) listened to the audio recordings of four different skin-on-skin tactile interactions, named ‘audio-touch stimuli’ (see Figure 1). This experiment aimed to investigate whether participants could recognize four different prototypical tactile gestures (i.e., rub, stroke, tap, hit) through their corresponding audio recording. Supplementary Table S1 describes the physical characteristics of the four sonified tactile gestures, chosen from the prototypical gestures defined in the literature on social touch.

### Perception of the sonified skin-on-skin tactile gestures without any clue.

In a first block, participants were asked to answer in free text what each sound corresponds to without any clue or context (Figure 2.A.). To minimize bias in analyzing the data, five naive external annotators were recruited to categorize *a posteriori* the qualitative answers gathered into the categories of “Touch”, “Not a touch”, and “Not a sound” (see Methods). Altogether, the stimuli were rated in 58.6% of the cases as belonging to the “Touch” category, 38.7% were categorized as “Not a touch” and 2.6% as “Not a sound” ($\chi^2(2, N=191)=92.4, p<0.001$). A post-hoc test indicated that the “Touch” category was significantly more present than the “Not Touch” category ($\chi^2(2, N=191)=7.8, p=0.005$). The four different stimuli were identified as related to touch with the following percentages: 53.2% for Rubbing stimuli; 52.3% for Stroking; 64.7% for Tapping; and 63.3% for Hitting (see contingency table in Table S2).
<figure><img src="image_4.png" /><figcaption>**Figure 2.** Results of Experiment 1. **A. Free-text responses without context (Block 1): Word cloud representation** of the participants' raw free-text responses (translated from French) for all audio-touch stimuli, without any information on the sounds presented, following the semantic analysis method and visual representation proposed by Schirmer et al. (43). The size and colors of the words indicate their frequency in the participants' answers, with a minimum threshold established at two occurrences and the maximum occurrences presented in Table S3. **B. Free-text responses with context (Block 2): Word cloud representation** of the participants' raw free-text responses for all four audio-touch stimuli when being informed that the sounds correspond to skin-on-skin touches, following the same method as described above (find the maximum occurrences in Table S5). **C. Free-text responses with context (Block 2): Cumulative barplot** representing the distribution of participants' free-text responses into the four audio-touch stimuli categories, as per conducted by external annotators. **D. Forced-choice categorization (Block 3):** cumulative barplot representing the categorization of each type of touch per stimulus in the forced-choice categorization task of Experiment 1. Note that categories with less than 1% of the responses are not presented and values under 5% are not specified on the graph (see details of these percentages in the Table S6).</figcaption></figure>

*Perception of the sonified skin-on-skin tactile gestures knowing it is a skin-on-skin touch.*

In the second block, participants were told that the auditory stimuli were originating from tactile interactions on someone's forearm, however without being given any example or being prompted with the touch categories. They were asked to identify which type of tactile gesture the sound corresponds to (Figure 2.B). The five external annotators were again charged with categorizing each free-text response in the four gesture categories we had pre-established (i.e., “Stroking”, “Rubbing”, “Tapping” and “Hitting”). Based on this *a posteriori* categorization, participants associated the audio-touch stimuli with the correct categories, i.e., as corresponding to the correct tactile gesture, significantly above chance ($\chi^2(12, N=196)=314.5$,
p<0.001, V=0.7). Indeed, all categories were accurately chosen above 66% of the time by participants, even though they were not given to them at this point of the experiment. Rubbing was correctly interpreted 85.7% of the time; Stroking 68%; Tapping 66.7%; and Hitting 75.5% of the time (Figure 2.C, see contingency table in Table S4).

***Categorization of sonified skin-on-skin tactile gestures.***

In the third block, participants had to listen to each stimulus and categorize them in a four-alternative forced-choice task: Rubbing, Stroking, Tapping, Hitting. All four stimuli were presented ten times, in ten sub-blocks organized in a randomized order (the rationale for this repetitive design is explained in the Methods section). The Chi-square goodness-of-fit test showed that as participants were given the possible categories, responses were still not randomly distributed ($\chi^2(9, N=2040)=5010$, p<0.001, V=0.91). Within each stimulus category, a follow-up Chi-square test confirmed that responses were not randomly distributed (Rubbing: $\chi^2(3, N=510)=1281.2$, p<0.001; Stroking: $\chi^2(3, N=510)=1046.9$, p<0.001; Tapping: $\chi^2(3, N=510)=1325$, p<0.001; Hitting: $\chi^2(3, N=510)=1361.3$, p<0.001). Therefore, participants have associated each stimulus with a specific corresponding tactile category above chance.

Moreover, the audio-touch stimuli were categorized as corresponding to the appropriate tactile gesture, with all stimuli being correctly recognized in over 86% of the cases (93.5% of accurate recognition for Rubbing, 86.3% for Stroking, 94.7% for Tapping and 95.7% for Hitting, see details in Figure 2.D. and Table S6). The Hitting audio-touch stimuli were the best categorized, and significantly chosen correctly above the other response categories (as confirmed by post-hoc Wilcoxon rank tests, see Table S7).

**Experiment 2. Perception of sonified skin-on-skin tactile socio-emotional intentions.**

Our second experiment investigated participants’ perception of six distinct socio-emotional intentions conveyed by touch and sonified with our audio-touch methodology (i.e., Anger, Attention, Fear, Joy, Love, Sympathy; see Methods for the rationale justifying the selection of these socio-emotional intentions). To this aim, another sample of participants (N=50) listened to six 5-second audio-touch stimuli presented in a randomized order. Table S8 presents the physical characteristics of the gestures used in this experiment.

***Perception of sonified socio-emotional intentions underlying skin-on-skin touch without any clue.***

In the first block, participants were given neither a clue nor contextual information on the nature of the sounds they had to listen to. When asked to give the emotion each stimulus would evoke to them (Figure 3.A.), they gave significantly different answers for each of the six stimuli, as the Chi-square test results showed (after a post-hoc categorization of the participants’ free-text responses by five other external annotators, see Methods, $\chi^2(35, N=290)=94.3$, p<0.001, V=0.3, and see contingency Table S9). Hence, participants answered differently to the sounds of a gentle stroking on the arm (Love), of a gentle back-and-
forth rubbing of the forearm (Sympathy), of a light and melodic tapping gesture (Joy), of a brief and violent
hit (Anger), of a trembling (Fear) and of a fast and repetitive tapping gesture (Attention).

Perception of sonified socio-emotional intentions underlying touch knowing it is a skin-on-skin
touch.

In the second block, participants were given the information that the sounds correspond to the sonification of skin-on-skin tactile interactions on the forearm. They had to listen again to each stimulus and state what the person touching the other was trying to convey to them (Figure 3.B). The external annotators had to categorize each free-text response into our six pre-established emotional categories. This *a posteriori* categorization showed that participants’ interpretations of the six audio-touch stimuli were significantly different ($\chi^2(35, N=255)=191.3, p<0.001, V=0.4$). They recognized the Anger stimulus as communicating anger in 25.6% of the cases; Attention in 22.2%; Fear in 43.5%; Joy in 32.5%; Love in 38.5% and Sympathy in 27.7% of the cases (see details in Figure 3.B and in the contingency Table S11).

<figure><img src="image_5.png" /><figcaption>**Figure 3.** Results of Experiment 2. **A. Free-text responses without context (Block 1): Word cloud representation** of the participants’ raw free-text responses (translated from French) to the six emotional audio-touch stimuli without any information on the nature of the sounds. The size and colors of the words indicate their frequency in the participants’ answers, with a lower threshold established at two occurrences minimum (along 43) and the maximum occurrences are presented in Table S10. **B. Free-text responses with**</figcaption></figure>
**context (Block 2)** (Left): Word cloud representation of the participants' raw free-text responses to the six emotional audio-touch stimuli knowing the sounds correspond to skin-on-skin touch; (Right) Cumulative barplot representing the distribution of the participants' free-text responses into the six audio-touch stimuli categories, as per categorized by external annotators (see maximum occurrences in Table S12). **C. Forced-choice categorization (Block 3)**: Cumulative barplot representing the categorization of each type of touch per stimulus in the forced-choice categorization task. Note that categories with less than 1% of the responses are not presented and values under 5% are not specified on the graph (see details of percentages in Table S13). **D. Valence ratings per audio-touch stimuli (Block 4)**: Average valence ratings of the touch underlying the audio-touch stimuli. Ratings on a VAS scale ranging from "0 - Very unpleasant" to "100 - Very pleasant". Dots represent rating per condition for each participant. The half violin provides data distribution. Circular points represent the mean for each rating, and the error bars denote the standard error of the mean.

***Categorization of sonified socio-emotional intentions underlying skin-on-skin touch.***

In the third block of Experiment 2, participants had to listen to each stimulus and categorize them in one of the six categories provided to them: Anger, Attention, Fear, Joy, Love and Sympathy (forced-choice, presented ten times in ten sub-blocks in a randomized order, see Methods for the rationale for this repetitive design). The Chi-square test showed that overall observed frequencies in this categorization task were not randomly distributed ($\chi^2(25, N=3000)=3804.4, p<0.0001, V=0.5$). Within each stimulus category, a follow-up Chi-square test confirmed that responses were not randomly distributed (see details in Table S13, Anger: $\chi^2(5, N=500)=606.3, p<0.001$; Attention: $\chi^2(5, N=500)=460.2, p<0.001$; Fear: $\chi^2(5, N=500)=312.4, p<0.001$; Joy: $\chi^2(5, N=500)=962.9, p<0.001$; Love: $\chi^2(5, N=500)=704.4, p<0.001$; Sympathy: $\chi^2(5, N=500)=840.7, p<0.001$).

Moreover, the audio-touch stimuli were associated with the intended socio-emotional intention above chance. The audio-touch stimuli for Joy, Anger and Attention were the best categorized along our pre-established categories, and significantly chosen above the other response categories (as confirmed by post-hoc Wilcoxon rank tests, see Table S14). Indeed, Joy stimulus was categorized as such in 66% of the cases, Anger stimulus in 54% of the cases and Attention stimulus in 47% of the cases. Sympathy stimulus was recognized as such 56% of the times, although confused in 37% of the cases with a communication of Love. Similarly, Love stimulus was categorized as such in 35% of the cases and was confused with communicating Sympathy in 52% of the cases. Finally, Fear stimulus was categorized as such in 24% of the cases and confused with communicating Attention in 39% of the cases (see details in Table S13). Overall, the confusion with another category reported by the participants usually occurred within similar categories (e.g., Love and Sympathy).

***Valence ratings of sonified socio-emotional intentions underlying skin-on-skin touch.***

In this final block, participants were asked to rate each audio-touch stimulus on a visual analogue scale (VAS) going from "0 - Not pleasant at all" to "100 - Extremely pleasant". A repeated-measure ANOVA was conducted on the mean valence ratings for the six audio-touch stimuli and showed a significant difference depending on the emotion conveyed ($F(3.3, 159.2)=78.7, p<0.001, \eta^2_p=0.62$). Post-hoc analyses using t-tests with Bonferroni correction indicated that, on average, participants gave positive valence ratings to positive emotions (Sympathy: M=76.6, SE=3.3; Love: M=73.5, SE=3.3; Joy: M=59.9, SE=3.8) and
negative valence ratings to negative emotions (Anger: M=17.6, SE=2.8; Attention: M=26, SE=2.8; Fear: M=19.5, SE=2.8) (see Figure 3.D and see details in Table S15). Mean ratings for the valence of the sounds can be found in the Table S16.

**Experiment 3. Influence of surface on the recognition of four sonified tactile gestures.**

In Experiment 1, participants categorized the sounds obtained through our audio-touch sonification methodology as specific tactile gestures. In Experiment 2, they also associated four out of six stimuli with the intended socio-emotional intentions underlying the touches and coherently rated the valence of all sounds. Nonetheless, the question of which sonic characteristic(s) of our audio-touch stimuli allows participants' perception to be close to their intended content remains to be elucidated. Participants may have been relying mostly on the stimuli rhythm to infer the underlying tactile gesture or socio-emotional intention. This would be consistent with the high recognition results of the distinct gestures in the first experiment, as well as with the confusions between socio-emotional categories which relied on similar movement mechanics in the second experiment. Alternatively, the fact that the surfaces involved in the touches were skins may have influenced the perception of tactile gestures belonging to social contexts. Experiments 3 and 4 were tailored to investigate the potential influence of surface on the perception of our sonified tactile gestures (Experiment 3) and tactile socio-emotional intentions (Experiment 4).

Experiment 3 conducted online, and investigated how participants would perceive and distinguish among four sonified tactile gestures (i.e., rub, stroke, tap, hit). A new, enriched, set of audio-touch stimuli was proposed, replicating the same gestures as in Experiment 1 in two 'Surface' conditions: 'Skin-on Skin' (named 'Skin' hereafter) and 'Object-on-Object' (i.e., 'Object'). Moreover, there were three variations of surface within each surface category (e.g., in the 'Skin' condition, the gestures were performed by the experimenter on the forearm of three different persons, see Methods). So, in total, there were 24 unique stimuli in this experiment for participants (N=84) to listen to.

*Categorization of sonified skin-on-skin versus object-on-object tactile gestures.*

Participants were asked to listen to each sound stimulus and were told that the sounds were about tactile gestures, but the surfaces on which they happen were not specified. They had to categorize each sound in a four-alternative forced-choice (4-AFC) task, corresponding to the type of tactile gestures: Rubbing, Stroking, Tapping, Hitting.

A Generalized Linear Mixed Model was employed to study the combined effect of Surface and Stimulus on participants' Accuracy (participants' categorical responses were transformed into a binary variable). The analysis of deviance using a Type II Wald Chi-square test showed a significant effect of Surface ($\chi^2$(1, N=84)=8.5, p=0.004), of Stimulus ($\chi^2$(3, N=84)=225.5, p<0.001) and a significant interaction between Surface\*Stimulus ($\chi^2$(3, N=84)=49.9, p<0.001). Within each Stimulus category, post-hoc pairwise comparisons, using estimated marginal means adjusted with the Tukey method for multiple comparisons, were conducted to determine which Surface predicted Accuracy better. Accuracy was significantly higher in the Skin condition for Stroking (M$_{StrokingSkin}$=63.9% vs. M$_{StrokingObject}$=36.1%, SE=0.2, z-ratio=-6.5,
p<0.001) and Hitting (M<sub>HittingSkin</sub>=89.7% vs. M<sub>HittingObject</sub>=81%, SE = 0.3, z-ratio=-2.9, p=0.004). However, Accuracy was significantly higher in the Object condition for Rubbing (M<sub>RubbingSkin</sub>=85.7% vs. M<sub>RubbingObject</sub>=98.8%, SE=0.6, z-ratio=4.5, p<0.001), and not significantly different based on the Surface for the Tapping stimulus (M<sub>TappingSkin</sub>=91.3% vs. M<sub>TappingObject</sub>=92.9%, SE=0.3, z-ratio=0.7, p=0.5). Note that the effect size for the full model is moderate to large: marginal R<sup>2</sup>=0.353 and conditional R<sup>2</sup>=0.5.

<figure><img src="image_6.png" /><figcaption>**Figure 4.** Results of Experiments 3 and 4. **A. Forced-choice categorization in Experiment 3:** Cumulative barplot representing the categorization of each stimulus per type of touch and per surface. Percentages in bold represent frequencies of responses above chance (i.e., 25%). **B. Forced-choice categorization in Experiment 4:** Cumulative barplot representing the categorization of each stimulus per emotion and per surface. Percentages in bold represent frequencies of responses above chance (i.e., 25%). **C. Valence ratings per audio-touch stimuli in Experiment 4:** Average valence ratings of the touch underlying the audio-touch stimuli. Ratings on a VAS scale ranging from “0-Very unpleasant” to “100-Very pleasant”. Dots represent rating per condition for each participant. The half violin provides data distribution, and the error bars in black denote the standard error of the mean.</figcaption></figure>

## Surface categorization.

In the final block of Experiment 3, participants had to categorize each sound in terms of Surface, choosing whether each stimulus was the sound of a touch happening between two human skins or between two inanimate objects. Taking all types of gestures together, participants better categorized Object (M<sub>Accuracy</sub>=93.4%) than Skin stimuli (M<sub>Accuracy</sub>=55.8%). The better categorization of Object stimuli than Skin stimuli based on the Surface was confirmed by the GLMM conducted on participants' responses (i.e., transformed into a binary variable for Accuracy). The Wald test revealed a significant effect of Surface (χ<sup>2</sup>(1, N=84)=138.5, p<0.001), a significant effect of Stimulus (χ<sup>2</sup>(3, N=84)=25.2, p<0.001) and a significant interaction Surface\*Stimulus (χ<sup>2</sup>(3, N=84)=74.4, p<0.001). Pairwise post-hoc comparisons using estimated
marginal means showed that Object was indeed significantly better categorized than Skin for each type of gesture: for Rubbing, ($M_{RubbingSkin}=68.7\%$ vs. $M_{RubbingObject}=89.3\%$; SE=0.3, z-ratio=6.1, p<0.001); Stroking, ($M_{StrokingSkin}=68.7\%$ vs. $M_{StrokingObject}=89.3\%$; SE=0.3, z-ratio=6.1, p<0.001); Tapping, ($M_{TappingSkin}=48.4\%$ vs. $M_{TappingObject}=98\%$; SE=0.5, z-ratio=9.6, p<0.001); and Hitting, ($M_{HittingSkin}=37.3\%$ vs. $M_{HittingObject}=98.8\%$; SE=0.4, z-ratio=11.8, p<0.001).

**Experiment 4. Influence of surface on the recognition of six socio-emotional intentions underlying sonified tactile gestures.**

The fourth online experiment investigated how participants discriminate between four different sonified tactile socio-emotional intentions (i.e., Sympathy, Joy, Anger, Attention), depending on the nature of the Surface (Skin or Object). 24 corresponding audio-touch stimuli were played to participants (N=79).

*Categorization of sonified skin-on-skin versus object-on-object tactile emotions.*

Participants listened to each stimulus, knowing that the sounds were about tactile gestures, but not informed about the surfaces on which the gestures happened. They had to categorize them in a 4-AFC task. Because the wording of the categories could have led to confusions in Experiment 2, in this task, participants were told that each category proposed corresponds to a socio-emotional intention that the person who is performing the touch is trying to convey to the one receiving it: ‘Communicate their joy’ (for Joy), ‘Communicate their anger’ (for Anger), ‘Comfort’ (for Sympathy) and ‘Grab their attention’ (for Attention).

A GLMM was employed to study the combined effect of Surface and Stimulus on participants' Accuracy (participants' categorical responses were transformed into a binary variable). The analysis of deviance using a Type II Wald Chi-square test showed a significant effect of Surface ($\chi^2$(1, N=79)=13, p<0.001), of Stimulus ($\chi^2$(3, N=79)=162.1, p<0.001) and a significant interaction between Surface\*Stimulus ($\chi^2$(3, N=79)=106.5, p<0.001). Within each Stimulus category, post-hoc pairwise comparisons using estimated marginal means were conducted to determine which Surface predicted Accuracy better. Therefore, Accuracy was significantly better in the Skin condition for Sympathy ($M_{SympathySkin}=64.6\%$ vs. $M_{StrokingObject}=29.5\%$, SE=0.2, z-ratio=-7.7, p<0.001) and Anger ($M_{AngerSkin}=50.2\%$ vs. $M_{AngerObject}=34.6\%$, SE=-0.7, z-ratio=-3.6, p<0.001). However, Accuracy was significantly better in the Object condition for Joy ($M_{JoySkin}=61.2\%$ vs. $M_{JoyObject}=75.6\%$, SE=0.7, z-ratio=3.4, p<0.001) and for the Attention-grabbing stimulus ($M_{AttentionSkin}=49.4\%$ vs. $M_{AttentionObject}=71.3\%$, SE=1, z-ratio=5.1, p<0.001).

*Valence ratings of sonified skin-on-skin versus object-on-object tactile socio-emotional intentions.*

In this block, participants were asked to rate how pleasant the touch underlying each audio-touch stimulus was pleasant on a visual analogue scale (VAS) going from “0 - Not pleasant at all” to “100 - Extremely pleasant”. A repeated-measure ANOVA was conducted on the mean valence ratings and showed a significant main effect of Stimulus (F(1.6, 123.8)=14.3, p<0.001, $\eta_p^2=0.2$), a main effect of Surface (F(1, 78)=14.8, p<0.001, $\eta_p^2=0.2$), and a significant interaction Stimulus\*Surface (F(1.7, 131.7)=14.188, p<0.001,
$\eta^2_p=0.2$). A post-hoc analysis comparing the two levels of the Surface factor showed that Skin was globally rated higher than Object, by 6.7 points ($M_{Skin}=45.8$ vs. $M_{Object}=39.2$, SE=1.8, t-ratio=-3.8, p<0.001). However, a post-hoc analysis using Tukey tests to compare the four stimuli showed that this effect is driven by the Sympathy stimulus only, as there was no significant difference in the valence ratings of the three other emotion Stimuli. Figure 4 and Table S18 give the average valence ratings per Stimulus and per Surface, and Table S19 presents the post-hoc tests results.

## Surface categorization.

In the final block of Experiment 4, participants were asked to categorize each sound in terms of surface: they had to choose whether each stimulus was the sound of a touch arising from two human skins or two inanimate objects. Taking all types of emotions into account, participants better categorized Object ($M_{Object}=91.6\%$) than Skin ($M_{Skin}=65.9\%$) stimuli. This result was confirmed by the GLMM conducted on participants' responses (i.e., transformed into a binary variable for Accuracy). The Wald test revealed a significant effect of Surface ($\chi^2(1, N=79)=71.8$, p<0.001), a significant effect of Stimulus ($\chi^2(3, N=79)=70.1$, p<0.001) and a significant interaction Surface\*Stimulus ($\chi^2(3, N=79)=98.6$, p<0.001). Pairwise post-hoc comparisons using estimated marginal means showed that Object was significantly better recognized for the Joy ($M_{JoySkin}=65.8\%$ vs. $M_{JoyObject}=99.6\%$, SE=1, z-ratio=5.2, p<0.001), Anger ($M_{AngerSkin}=53.6\%$ vs. $M_{AngerObject}=98.3\%$, SE=0.5, z-ratio=8.5, p<0.001) and Attention stimuli ($M_{AttentionSkin}=67.1\%$ vs. $M_{AttentionObject}=100\%$, SE=17.5, z-ratio=1.1, p<0.001). On the contrary, Skin was significantly better categorized for the Sympathy stimuli ($M_{SympathySkin}=77.2\%$ vs. $M_{SympathyObject}=68.4\%$, SE=-0.6, z-ratio=-2.5, p=0.01).

## Discussion

Humans spontaneously and intuitively use specific tactile gestures to convey particular socio-emotional intentions (e.g., love, sympathy, anger) to other humans (6, 12), which can be described by their physical features (e.g., velocity, applied pressure, type of movement, see 14). Interestingly, the tactile and auditory modalities both rely on vibrations to communicate information, opening the opportunity of using sounds to provide remote access to the sense of touch. However, to date, no studies had investigated whether it is possible to sonify a variety of skin-on-skin social touches and whether the resulting auditory gestures and associated socio-emotional meanings can be recognized through their auditory counterparts. To fill this gap, we sonified tactile gestures with our audio-touch methodology, using a piezoelectric transducer to pick-up the vibrations happening on the surface being touched. Across four online experiments, participants were presented with audio-touch stimuli. Our hypothesis was that these sounds should carry information intrinsic to touch, such as type of gestures, emotions, and surface characteristics. The first experiment revealed that participants listening to our audio-touch stimuli accurately identified the tactile gestures underlying the sounds, even with little to no context. In the second experiment, participants categorized above chance the socio-emotional intentions of skin-on-skin touches converted into sounds and coherently rated their valence. Experiments 3 and 4 replicated the same tactile gestures both on various human skins, and on inanimate
objects, and found a significant influence of surface on participants' discrimination performances and valence ratings. Each of these main results will be discussed in detail below.

The first main result to emerge from our study is the participants' ability to identify tactile gestures through their sounds, with increased accuracy along with increasing information being given about the sounds. Particularly, for the first contextless condition in Experiment 1, the majority of participants (58.6%) spontaneously referred to touch when asked what the sounds were about. This result obtained with participants' free-text answers is especially interesting, as our 5-second musicless, voiceless noises of friction were associated with the notion of touch in a variety of ways. Their responses evoked humans touching other humans (e.g., "rubbing a hand", "caresses", translated from French), humans touching objects (e.g., "knocking on a door", "tapping a microphone") as well as objects touching other objects (e.g., "rubbing fabrics together", "sharp hit on a hard surface"). Moreover, when informed that the sounds are skin-on-skin touches without being prompted with any examples, participants accurately identified all four types of tactile gestures in our audio-touch stimuli (74% of accuracy on average across categories).

Importantly, when participants were given the choice between the four categories of socio-tactile gestures, they categorized them with even higher accuracy (86% of accuracy on average in all categories). No confusion was observed between categories, which is surprising given that participants were not given any definition of the various gestures and could have easily mistaken, for instance, stroking (i.e., one-way slow and gentle sliding) with rubbing (i.e., medium speed back-and-forth gentle sliding). It is important to note that categorization tasks are criticized in the literature (23, 24), as they can artificially constraint participants' spontaneous responses. However, in this experiment, the forced-choice distribution of answers was highly coherent with the free-text block. Therefore, our audio-touch stimuli facilitate associations between the notion of touch and corresponding sounds, and they can also be distinguished as being the sonified version of a specific tactile gesture naturally used by humans in social interactions: rubbing, stroking, tapping and hitting. Tactile waves have been shown to contain information on the level of applied pressure and the velocity of the specific touch that is received (15). Thus, our results suggest that tactile waves presented in the auditory domain also contain the necessary information to recognize socio-tactile gestures.

Experiment 2 shed light on two main results: participants were able to discriminate between six tactile socio-emotional intentions converted into sounds and they rated the valence of each audio-touch stimulus coherently with the intended underlying emotion. The six tactile socio-emotional intentions we sonified replicated the gestures shown to convey these socio-emotional intentions in the literature on social touch (12, 14). First, without being given any clue about the nature of the sounds, participants associated each audio-touch stimulus with an emotion which closely resembled the intended meaning. Indeed, the three negative socio-emotional intentions (i.e., Anger, Attention, Fear) mainly evoked negative free-text responses (e.g., "fear", "stress") while the three positive ones (i.e., Joy, Love, Sympathy) mainly appealed to positive appraisals ("joy", "calm", "soothing"). We estimate that the Attention stimulus presented in this study belongs to the negative emotions category, because it was presented to participants as 'Impatience' in French, a rather frustrating feeling. Giving more context to participants (i.e., that they were listening to skin-on-skin
touch, in the second block) increased accuracy between participants' free-text answers and the meaning
underlying each tactile gesture. As such, tactile interactions which can be meaningful in the tactile modality
(e.g., a gentle stroking expressing love) can convey some of the intended emotional content in the auditory
modality.

Then, participants were prompted with the six possible socio-emotional intentions and associated
each audio-touch stimulus with the intended socio-emotional intention above chance. However, some
confusions between certain dyads of socio-emotional intentions emerged, between Love and Sympathy and
between Fear and Attention. These confusions could be explained by several reasons. First, no definition of
the six socio-emotional intentions was given to participants in this forced-choice categorization block:
participants may have had a similar understanding of Love and Sympathy being expressed through touch,
this conceptual confusion impacting the categorization results. Second, the literature on social touch has
specified the physical characteristics of the tactile gestures most often used to convey these intentions, but
not unanimously. Hertenstein et al. (12) reckoned that humans rely on several different tactile gestures to
express one specific emotion (e.g., love can be expressed through hugging, stroking, patting or a simple
contact of the hand on the skin). Reciprocally, one specific tactile gesture can be used to convey different
emotions (e.g., shaking can be used to express anger, fear, happiness, gratitude, and disgust). Indeed, we
chose to replicate and to associate one tactile gesture with one specific socio-affective intention in our audio-
touch stimuli, following what the literature on social touch has identified as "prototypical gestures". With
our audio-touch methodology, we believe that the tactile gestures' physical features were accurately sonified
into our auditory signals. Nonetheless, it does not guarantee that, in the tactile modality itself, participants
would associate each of these gestures with the only meaning we chose to limit it to. Indeed, individual
differences and contextual factors may lead participants to interpret the socio-emotional content underlying
each gesture differently, that is then reflected in the categorization of our stimuli, which are auditory
representations of these gestures.

Importantly, when asked to categorize the socio-emotional intentions, participants did not confuse
stimuli of opposite valence or intensity. Specifically, they did not confuse positive socio-emotional intentions
with negative ones (e.g. Sympathy and Love vs. Attention and Fear), and an arousing emotion like Joy was
not confused with the slow and soft touch corresponding to Love. Hence, each sonified touch seemed to be
appropriately categorized along their valence, a finding that was confirmed in the final block of Experiment
2, in which participants rated the valence of the touch underlying the six audio-touch stimuli. Participants
indeed gave positive valence ratings to the auditory conversion of positive socio-emotional intentions
expressed through touch (i.e., Joy, Love, Sympathy) and reciprocally for the negative ones (i.e., Anger,
Attention, Fear). Notably, these valence ratings are coherent with the valence of the intended socio-emotional
intentions. Therefore, Experiment 2 showed that our six audio-touch stimuli had the necessary information
for participants to discriminate between distinct socio-emotional intentions usually communicated through
touch, or at least between emotions of opposite valence. It is worth highlighting that the obtained results
were better in Experiment 1 than in Experiment 2. Therefore, it is easier for participants to identify and
discriminate between tactile gestures through sounds rather than their socio-emotional functions. This is
unsurprising in our contextless, unimodal setting, since a tactile gesture can be associated with several distinct intentions depending on other cues which give context to the socio-affective touch. In line with this, Ellingsen et al. (25) showed how environmental settings and internal states can greatly modulate the appreciation of a touch, making people perceive the same stimulus as either very pleasant or repulsive. Sailer & Leknes (26) also highlighted that participants' perception of the affective value of a tactile stimulus is often more influenced by contextual factors of the real-world surrounding stimulus presentation and previous experiences, than the stimulus itself.

Surface dynamics have been shown to be an essential factor in shaping sonified haptic interactions (19), particularly when considering the intricate properties of human skin (22). In our study, Experiments 3 and 4 unveiled that the surfaces involved in tactile interactions also significantly influenced participants' perception of the audio-touch stimuli. Two newly enriched audio-touch stimulus databases were created, to expand the initial stimuli sets of Experiments 1 and 2. These additions were important for introducing variability between surface categories (e.g., skin-on-skin versus object-on-object) and within each surface category, encompassing three individuals' skins and three densities of plastic. With this increased stimulus pool, the experimental design was adapted to mitigate potential exposure or learning biases inherent to the repetitive designs of Experiments 1 and 2. Diversifying the skin-on-skin stimuli and limiting the alternative choices in the categorization task of Experiments 3 and 4 yielded categorization performances very similar to those observed in Experiments 1 and 2.

Experiments 3 and 4 revealed that participants' perception of the sonified gestures and socio-emotional intentions was different depending on whether they happened on Skin or on Object. In Experiment 3, participants better categorized the Stroking and the Hitting gestures when listening to the skin-on-skin audio-touch stimuli in comparison to the object-to-object stimuli. In Experiment 4, participants also better categorized the Sympathy and Anger stimuli due to skin-on-skin touch. These two results respond to one another, as the gesture underlying the 'Sympathy' stimulus in Experiment 4 is a stroking movement very similar to the one used in Experiment 3 to represent this category. Moreover, the gesture underlying the 'Anger' stimulus in Experiment 4 corresponds exactly to the hitting gesture used in Experiment 3. Therefore, whether the categorization task was about recognizing the content of (i.e., the gesture or the intention) or the surface involved in the touch, human skin-on-skin interactions improved participants' perception for these stimuli (Stroking/Sympathy and Hitting/Anger). A surprising result we wish to highlight, is that contrary to Experiment 2 in which no confusion was observed in the categorization of socio-emotional stimuli of opposite valence, participants in Experiment 4 mistook the Sympathy stimuli for Anger when it was in the Object condition. The Sympathy stimuli recorded on human skin did not yield such a confusion, emphasizing once again the pivotal role of surface in our audio-touch stimuli. Additionally, when asked about the nature of the surfaces interacting in the sounds, participants better recognized the Object than the Skin for all stimuli in Experiment 3, and for all stimuli except for Sympathy in Experiment 4. The skin-on-skin stroking gesture which conveys Sympathy again sets itself apart from the other categories (both stimulus and surface).

In Experiment 4, the valence ratings of the four socio-emotional audio-touch stimuli were globally higher for the Skin rather than for the Object stimuli. Nonetheless, this effect was only driven by the
Sympathy stimuli, which were rated as significantly more pleasant when happening on skin than on plastic. The other stimuli did not yield a significant valence rating difference based on surface. In order to properly convey the notion of social comfort, like a caress, through audio-touch, it is thus necessary to sonify the human skin, not an object.

The results of our study reveal that the audio-touch stimuli we created convey a variety of information about social touch through sounds (i.e., gestures, socio-emotional meanings, surface). In the literature on sensory conversion, participants often perform surprisingly well at recognizing information displayed through another sensory modality. Indeed, people are able to accurately decode signals belonging to the visual system when they are converted into sounds or tactile patterns (see 27, 28), so that they can recognize simple (29) and complex (30) shapes or can navigate through a certain environment (31, 32) without using their eyes. Atkinson et al. (33) proposed multi-touch interactive textiles where participants could correctly identify surfaces that were until then poorly represented only with visual cues. Moreover, Thoret et al. (34) showed that people can perceive specific shapes based on the sounds of their drawing because the frictions' acoustic properties respected the same kinematic rules as human movements. Analyzing participants' perception of emotions is of course much more complex, as socio-emotional intentions are multidimensional, and can be expressed via multiple potential movements. In that sense, future work should not impose a unique association between one touch, one sound and one emotional category, while still allowing quantitative and qualitative analysis of the responses. Different combinations of touch, sound, and emotion could be tailored for specific contexts. For instance, gentle stroking audio-touch stimuli could promote emotional well-being in older adults (35), but the same stimuli could induce discomfort or distress if paired with threatening virtual agents (36).

Our study thus unveils the relevance of building upon social touch and movement sonification findings to translate the content of one sensory modality, touch, into another, audition. It also opens the way to a new understanding of human perception of socio-affective touch, as well as offers a novel methodology for sonifying human skin and investigating its highly heterogenous properties (e.g., hairiness, firmness, hydration level). Our experimental recording and editing setup of audio-touch stimuli were validated at the behavioral level, as specific sensory correspondences between audio, touch, and the innermost socio-emotional intentions conveyed by tactile interactions were discovered. Indeed, recording the vibrations of various skin-on-skin interactions on the forearm, away from the fingertips entering in contact with the skin (following 17, 18), via a piezoelectric transducer, can pick-up the specific properties that enabled our participants to identify distinct tactile gestures, distinct tactile socio-emotional intentions, and distinct interacting surfaces through their auditory counterparts. Nonetheless, our sonification tool can be improved: only the vibrations happening on the surfaces were captured in this study, not what is perceived in the air by human ears. A new recording apparatus, such as an air microphone, could enable the sonification of gestures or associated with other socio-emotional intentions, such as the hugging conveying sadness (14), as they might generate a different type of signal. Furthermore, departing from standardized gestures to replicate more naturalistic and ecological gestures would allow conveying other socio-emotional intentions, as social touch outside of the lab setting encompasses complex tactile behaviors. For instance, while our microphone
cannot record a standardized still-holding hug, dynamic movements simultaneously involved in hugging, such as gently stroking the arms or lightly tapping the back, could be captured and sonified by another type of microphone, offering a richer representation of tactile interactions and their socio-emotional meaning.

Our study has implications at the fundamental level. First, it paves the way for a research area linking participants' recognition of audio-touch stimuli to their acoustic properties. Spectrograms provided in the Supplementary Materials (Figures S1, S2, S4, and S6) hint at differences in acoustic features, which may contribute to participants' ability to differentiate stimuli. By bridging tactile and auditory modalities, our audio-touch methodology broadens the scope of sensory research and raises new questions about the links between signals and perception. It encourages exploration of how acoustic properties—like spectral patterns and temporal dynamics—carry information about tactile and socio-emotional cues, and how these are interpreted by our sensory and cognitive systems. Exploring these questions requires further investigation using a larger set of audio-touch samples, combined with advanced computational tools such as machine learning-based classification models, which could in turn identify the acoustic features most predictive of participants' perceptual responses. In addition, previous research has suggested that sensory conversion gives rise to a multisensory perception not reducible to a single sensory modality (e.g., for a review see 37), but it had never been done in the audio-tactile domain. Showing that what happens in the intimacy of the human skin can now be grasped by the remote human ears asks the question of the nature of the resulting perception, auditory versus tactile.

Our study also has implications at the applied level. By establishing that information and socio-emotional intentions usually conveyed by a proximal sense can be recognized through a distal sensory modality, this research sets up the grounds for future possibilities to allow much needed socio-affective interactions at a distance. In particular, adding social touch in virtual reality appears timely and is a question of growing interest (see 38 for a review). The multisensory design of virtual environments would definitely benefit from providing mediated social touch (39), for instance through virtual characters who interact with human users (40). To this aim, research has since focused on complex haptic devices to allow humans and virtual characters to touch each other during virtual interactions (38). In parallel, decision models relying on machine learning are being developed for virtual agents to determine when and how it is relevant and appropriate to use touch from a virtual character toward humans (41). Future fundamental and applied studies could integrate our audio-touch stimuli to make virtual interactions more naturalistic and sensorily rich, whether they rely on haptic-based equipment or not. Indeed, based on the results of this first study, the more context was given to participants, the more they recognized the intended tactile gesture or socio-emotional intention. One can therefore hypothesize that pairing our audio-touch stimulations with congruent stimuli in another sensory modality would only strengthen the understanding and relevance of meaningful touch used in virtual settings. Therefore, our audio-touch stimuli could either be used on their own to support visual stimulations, or could complement the vibrations provided by haptic feedback technologies in audio-visual environments. In both cases, the versatility of our audio-touch stimuli aims at enhancing specific types of social touch gestures and at conveying specific socio-emotional intentions during virtual interactions, in the
hope of compensating for the detrimental consequences of the lack of social touch, whether they are due to personal or broader societal constraints.

# Materials and Methods

## Participants

Experiments 1 to 4 were approved by the INSEAD Institutional Review Board of Sorbonne Université, and were conducted in accordance with the Declaration of Helsinki. In Experiment 1 and 2, fifty-two participants were recruited; in Experiment 3 and 4, eighty-four participants were recruited. Based on outlier detection for failing the attention-checks, in Experiment 1, the final sample consisted of fifty-one participants (31 females, $M_{age} = 23$ years, SD = 3.45); of fifty participants in Experiment 2 (25 females, $M_{age} = 23.86$ years, SD = 3.76); of eighty-four participants in Experiment 3 (64 females, $M_{age} = 25$ years, SD = 3.8); of seventy-nine participants (39 females, $M_{age} = 25.27$ years, SD = 4.06). All participants were recruited by the INSEAD-Sorbonne Université Behavioral Lab, through their online participants' database (SONA System). All participants provided electronic consent before the study began and received monetary compensation for taking part in the experiment, which lasted around 20 minutes, provided they correctly answered the attention-check questions (accuracy rate > 80% on these trials).

## Stimuli

The very first database of skin-on-skin touches recorded through their vibratory signals (named 'audio-touch stimuli') was created for this study. For all experiments, the methodology to record the sounds of audio-touch was the same (details can be found in Figure S1). The vibrations happening on the surface of the skin were caught by the violin microphone, which was carefully taped to the ventral region of the forearm (see Figure 1). An additional 5-second stimulus found on a free sound database was added as an attention-check (i.e., a dog barking).

For Experiment 1, four tactile gestures were chosen and performed by the trained experimenter on the ventral region of her own forearm skin: stroking, rubbing, tapping, and hitting. Features extracted from the literature such as velocity, tactile continuity, pressure, points of contact, and rhythm of the tactile interactions were manipulated to differentiate the four prototypical gestures included in this first study (see Table S1). For Experiment 2, six socio-emotional intentions conveyed through touch were chosen and recorded: Anger, Attention, Fear, Joy, Love, and Sympathy. They were selected based on a comparative analysis of the literature describing the characteristics of tactile gestures expressing different socio-emotional intentions (12, 14, see S8).

Experiments 3 and 4 investigated the influence of the surface involved in the tactile interactions on participants' categorization of either the gestures or the socio-emotional intentions. Thus, the same tactile gestures as in Experiments 1 and 2 were reproduced in two different surface conditions: 'Skin' and 'Object'. In the 'Skin' condition, gestures were performed by the experimenter's hand on the ventral region forearm skin of three people (i.e., herself and two male volunteers). In the 'Object' condition, gestures were performed by the experimenter handling a finger-shaped aluminum tool on three polymethyl-methacrylate (PMMA) objects, varying in thickness (see Figure S3). In Experiment 4, gestures expressing socio-affective
intentions that were unambiguously categorized by participants in Experiment 2 were replicated once in each condition (i.e., 4 emotions: Anger, Attention, Joy and Sympathy). There were 24 audio-touch stimuli used in Experiment 3 and 4 (details can be found in Figure S4 and S6).

## Procedure

Participants were naive about the purpose of the experiments. They were only instructed that they were going to listen to short audio clips. Participants were asked to wear headphones and to be in a quiet environment to take part in the study. All experiments were run online and were thus autonomously completed by the participants on their computer. In all experiments, each trial included a sound stimulus (i.e., either one of the audio-touch stimuli or the attention-check dog-barking sound) and a question with a specific format to answer (i.e., free-text response, forced-choice categorization, valence rating on a visual analog scale). In each block, the audio-touch stimuli were presented in a randomized order to the participants. The details for each block's procedure can be found in Supplementary Information.

## Data analysis

For the first and second blocks of Experiments 1 and 2, a qualitative data analysis was necessary to process the participants' free-text responses (see Table S2, S4, S9 and S11). Chi-square goodness-of-fit tests distinguishing each target cue were then conducted to determine whether the distribution of participants' responses into these categories was random and, if not, showed participants' preferences. In the first block of Experiments 3 and 4, the number of free-text responses was high (1500+) and their length variable. Thus, no statistical test was done on these data. To visually represent the semantic concepts to which participants' free-text responses belonged and their frequency in each of these blocks, the Term Document Matrix R package was combined with the Word Clouds R package (along 43).

For the forced-choice categorization tasks (i.e., third block of Experiment 1 & 2, second block and last block of Experiment 3 & 4) Chi-square goodness-of-fit tests were conducted to determine whether the distribution of participants' responses into the target categories was random or not. Then, for each condition, post-hoc analysis using either t-tests (if there were two categories) or Wilcoxon rank tests was conducted to find the preferred category among the various ones that were presented. For the latter, the groups were paired in descending order (i.e., the category with the highest frequency was compared to the one with the second highest frequency, then this group was compared to the third highest, and so on), similarly to Kirsch et al. (11). For Experiment 1 & 2, the same measures were repeated ten times for each stimulus on the same participants, thus the assumption of independence between the trials was verified (see details in Supplementary Information). For Experiment 3 & 4, to assess the influence of Surface and Stimulus on participants' Accuracy in the 4-AFC task, we used a Generalized Linear Mixed-Effects Model (see detailed Data Analysis section in Supplementary Information).

All analyses were conducted using the softwares R (version 4.0.3) and JASP (version 0.16.1).
## Acknowledgments

We are grateful to the staff members of the INSEAD-Sorbonne Université Behavioural Lab where the data collection was done. We thank Jordan Cohen for his help in stimuli recordings and data collection for pilots of these experiments during his internship. We also thank Emmanuel Guigon and Perside Ngani for their helpful comments on earlier versions of this manuscript, Stéphane Genet and Thanh-loan Le for their suggestions on the statistical analyses, as well as Frederic Bevilacqua, Sylvain Argentieri, and Mohamed Chetouani for insightful discussions about complex auditory signal analyses. This work was funded by the French ANR (projects MATCH ANR-22-CE33-0010 and LackSocialTouch ANR-22-CE28-0005), the CNRS INS2I Emergence (“Multimodal simulation of touch: Application to virtual agents”), and Idex Sorbonne Université (11-IDEX-0004).

## References

1. T. Field, Infants' need for touch. *Hum. Dev.* **45**, 100–103 (2002).

2. A. J. Bremner, D. J. Lewkowicz, C. Spence, “The multisensory approach to development” in *Multisensory development*, A. J. Bremner, D. J. Lewkowicz, C. Spence, Eds. (Oxford University Press, 2012), pp. 1–26. https://doi.org/10.1093/acprof:oso/9780199586059.003.0001

3. A. Gallace, C. Spence, “In Touch with the Future: The Sense of Touch from Cognitive Neuroscience to Virtual Reality” (Oxford University Press, 2014).

4. M. von Mohr, L. P. Kirsch, A. Fotopoulou, Social touch deprivation during COVID-19: Effects on psychological wellbeing and craving interpersonal touch. *R. Soc. Open Sci.* **8**, 210287 (2021).

5. A. J. Bremner, C. Spence, The development of tactile perception. *Adv. Child Dev. Behav.* **52**, 227–268 (2017).

6. M. J. Hertenstein, D. Keltner, B. App, B. A. Bulleit, A. R. Jaskolka, Touch communicates distinct emotions. *Emotion* **6**, 528–533 (2006).

7. A. Schirmer, I. Croy, S. R. Schweinberger, Social touch: A tool rather than a signal. *Curr. Opin. Behav. Sci.* **44**, 101100 (2022).

8. B. App, D. N. McIntosh, C. L. Reed, M. J. Hertenstein, Nonverbal channel use in communication of emotion: How may depend on why. *Proc. Natl. Acad. Sci. U.S.A.* **11**, 603–617 (2011).

9. C. J. Cascio, D. Moore, F. McGlone, Social touch and human development. *Dev. Cogn. Neurosci.* **35**, 11 (2019).

10. R. Etzi, C. Carta, A. Gallace, Stroking and tapping the skin: Behavioral and electrodermal effects. *Exp. Brain Res.* **236**, 453–461 (2018).

11. L. P. Kirsch et al., Reading the mind in the touch: Neurophysiological specificity in the communication of emotions by touch. *Neuropsychologia* **116**, 136–149 (2018).

12. M. J. Hertenstein, R. Holmes, M. McCullough, D. Keltner, The communication of emotion via touch. *Emotion* **9**, 566–573 (2009).

13. E. H. Thompson, J. A. Hampton, The effect of relationship status on communicating emotions through touch. *Cogn. Emot.* **25**, 295–306 (2011).

14. S. McIntyre et al., The language of social touch is intuitive and quantifiable. *Psychol. Sci.* **33**, 1477–1494 (2022).

15. L. P. Kirsch, X. E. Job, M. Auvray, V. Hayward, Harnessing tactile waves to measure skin-to-skin interactions. *Behav. Res. Methods* **53**, 1469–1477 (2021).

16. M. Wiertlewski, J. Lozada, V. Hayward, “The spatial spectrum of tangential skin displacement can encode tactual texture” in *Proceedings of IEEE Transactions on Robotics* (2011), pp. 461–472.

17. B. Delhaye, V. Hayward, P. Lefèvre, J. L. Thonnard, Texture-induced vibrations in the forearm during tactile exploration. *Front. Behav. Neurosci.* **6**, 37 (2012).
18. Y. Shao, V. Hayward, Y. Visell, Spatial patterns of cutaneous vibration during whole-hand haptic interactions. *Proc. Natl. Acad. Sci. U.S.A.* **113**, 4188–4193 (2016).

19. A. Tajadura-Jimenez, N. Bianchi-Berthouze, E. Furfaro, F. Bevilacqua, "Sonification of surface tapping changes behavior, surface perception, and emotion" in *Proceedings of IEEE MultiMedia* (2015), pp. 48–57.

20. L.E. Crommett, A. Pérez-Bellido, J. M. Yau, Auditory adaptation improves tactile frequency perception. *J. Neurophysiol.* **117**, 1352–1362 (2017).

21. A. Ramasamy et al., "Human self-touch vs. other-touch resolved by machine learning" in *Haptics: Science, Technology, Applications. EuroHaptics 2022*, H. Seifi et al., Eds. (Lecture Notes in Computer Science, Vol. 13235, Springer, Cham, 2022).

22. J.-F. Petiot et al., "Characterization of human skin properties with sounds" in *The music of the skin. 16e Colloque National S-mart*. (France, 2019).

23. L. F. Barrett, Context reconsidered: Complex signal ensembles, relational meaning, and population thinking in psychological science. *Proc. Natl. Acad. Sci. U.S.A.* **77**, 894 (2022).

24. J. A. Russell, Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. *Psychol. Bull.* **115**, 102–141 (1994).

25. D. M. Ellingsen, S. Leknes, G. Løseth, J. Wessberg, H. Olausson, The neurobiology shaping affective touch: Expectation, motivation, and meaning in the multisensory context. *Front. Psychol.* **6**, 1986 (2016).

26. U. Sailer, S. Leknes, Meaning makes touch affective. *Curr. Opin. Behav. Sci.* **44**, 101099 (2022).

27. G. Arnold, J. Pesnot-Lerousseau, M. Auvray, Individual differences in sensory substitution. *Proc. Natl. Acad. Sci. U.S.A.* **30**, 579–600 (2017).

28. M. Auvray, Multisensory and spatial processes in sensory substitution. *Proc. Natl. Acad. Sci. U.S.A.* **37**, 609–619 (2019).

29. J. Pesnot-Lerousseau, G. Arnold, M. Auvray, Training-induced plasticity enables visualizing sounds with a visual-to-auditory conversion device. *Scientific Reports*, **11**, 14762 (2021).

30. M. Auvray, S. Hanneton, J. K. O'Regan, Learning to perceive with a visuo-auditory substitution system: Localization and object recognition with The Voice. *Proc. Natl. Acad. Sci. U.S.A.* **36**, 416–430 (2007).

31. D. R. Chebat, F. C. Schneider, R. Kupers, M. Ptito, Navigation with a sensory substitution device in congenitally blind individuals. *Neuroreport* **22**, 342–347 (2011).

32. D. R. Chebat, S. Maidenbaum, A. Amedi, Navigation using sensory substitution in real and virtual mazes. *PLoS ONE* **10**, e0126307 (2015).

33. D. Atkinson et al., "Tactile perceptions of digital textiles: A design research approach" in *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (ACM, New York, NY, 2013), pp. 1669–1678. https://doi.org/10.1145/2470654.2466221

34. E. Thoret, M. Aramaki, R. Kronland-Martinet, J. L. Velay, S. Ystad, From sound to shape: Auditory perception of drawing movements. *J. Exp. Psychol. Hum. Percept. Perform.* **40**, 983–994 (2014).

35. I. Sehlstedt et al., Gentle touch perception across the lifespan. *Psychol. Aging* **31**, 176–184 (2016).

36. T. Bosse, et al., "Virtually bad: A study on virtual agents that physically threaten human beings" in *Proceedings of the 17th International on Autonomous Agents and MultiAgent Systems (Stockholm, 2018)*, pp. 1258–1266.

37. L. P. Kirsch, X. Job, M. Auvray, Mixing up the senses: Sensory substitution is not a form of artificially induced synaesthesia. *Multisensory Research*, **34**, 297–322 (2020).

38. A. Gallace, M. Girondiny, Social touch in virtual reality, *Curr. Opin. Behav. Sci.* **43**, 249–254 (2022).

39. R. Héron, S. Safin, M. Baker, F. Détienne, "The functions of computer-mediated touch at a distance: An interactionist approach" in *Proceedings of the 21st Congress of the International Ergonomics Association (IEA 2021)*, N. L. Black, W. P. Neumann, I. Noy, Eds. (Lecture Notes in Networks and Systems, Vol. 223, Springer International Publishing, 2022), pp. 45–53.

40. T. Bickmore, "Health-related applications of socially interactive agents" in *The Handbook on Socially Interactive Agents: 20 years of Research on Embodied Conversational Agents, Intelligent Virtual Agents, and Social Robotics*, B. Lugrin, C. Pelachaud, D. Traum, Eds. (ACM Books, 2022), 2, pp. 403–436.
41. F. Boucaud, C. Pelachaud, I. Thouvenin, "Decision model for a virtual agent that can touch and be touched" in *Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (London, 2021)*, pp. 232-241.

42. F. Faul, E. Erdfelder, A.-G. Lang, A. Buchner, G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. *Behav. Res. Methods* **39**, 175-191 (2007).

43. A. Schirmer, C. Cham, Z. Zhao, I. Croy, What makes touch comfortable? An examination of touch giving and receiving in two cultures. *Pers. Soc. Psychol. Bull.*, in press.
**Supporting Information for**

Paving the way for social touch at a distance: sonifying tactile interactions and their underlying
emotions.

Alexandra de Lagarde, Catherine Pelachaud, Louise P. Kirsch & Malika Auvray.

Corresponding authors:
Alexandra de Lagarde
[delagarde.alexandra@gmail.com](mailto:delagarde.alexandra@gmail.com)

Louise P. Kirsch
louise.kirsch@u-paris.fr

Malika Auvray
malika.auvray@sorbonne-universite.fr

**Participants**

Gender parity was chosen for Experiments 2 and 4 to avoid any potential bias when perceiving, processing, and naming different emotions. Volunteers were screened so that they would participate in one of the experiments only and they met the following criteria: (i) no auditory deficit according to self-reports, (ii) French native speakers.

**Stimuli**

For all experiments, the methodology to record the sounds of audio-touch was the same. All tactile gestures were recorded via a violin microphone (i.e., Shadow SH-SV2, a piezoelectric twin-disc transducer), with a 44.1 kHz sampling rate, then converted by the Scarlett 2i2 3rd generation audio interface and saved on the experimenter's computer via Audacity® 3.0.0. (SI1) recording and editing software. All stimuli were shortened to 5 seconds, cut to correspond to where the signal was the most salient and presented the least noise. Some of the stimuli were also processed on Audacity® to reduce background noise and amplify the signal to be sufficiently audible, so that all stimuli would be of equal quality and intensity. Indeed, a slow stroking gesture does not produce an auditory signal as strong (in dB) as a fast repetitive tapping. It was the only editing that was conducted on the stimuli before use in the experiments. Each gesture was sonified once, via this apparatus.

**Procedure**

In the first block of all experiments (1 to 4), participants were not given any clue or context on the nature of the sound they had to listen to. Each stimulus was presented once. For each stimulus, they had to give a free-text response to the question “Please listen to this sound. What does this sound make you think of?” (for Experiments 1 and 3, in French in the study) or “What emotion does this sound evoke to you?” (for Experiments 2 and 4).

In the second block of Experiments 1 and 2, participants were informed that the sounds were of tactile interactions. Knowing that, they were asked to listen again to every stimulus once again and give a free-text response as to either what type of skin-on-skin tactile interaction each sound corresponded to (Experiment 1) or what the person touching the other on the forearm was trying to communicate (Experiment 2). In these first and second blocks, participants were not constrained and could listen to each stimulus as many times as they wanted to.

In the third block of Experiments 1 and 2, participants had to complete ten sub-blocks of a forced-choice categorization task with the same stimuli. They were given the name of the categories of tactile interactions corresponding to the sounds (i.e., Experiment 1: Stroking (*Caresse*, in French), Rubbing (*Frottement*), Tapping (*Tapotement*), Hitting (*Tape*) and Experiment 2: Love (*Amour*), Anger, (*Colère*), Sympathy (*Empathie/Réconfort*), Joy (*Joie*), Attention (*Impatience*), Fear (*Peur*)). In each sub-block, the auditory stimuli were presented once in a randomized order and participants were instructed to listen to each stimulus only once before categorizing it. Participants did not get any feedback during the task and all categorical options always
remained available after each trial. This forced-choice categorization task was also proposed as the second block of the Experiments 3 and 4: participants were informed that the sounds corresponded to tactile interactions. However, the surfaces involved in the touches were not specified here. As the initial number of stimuli was multiplied by six, due to the variation of the surfaces used for the stimuli (i.e., Skin * 3 persons, Object * 3 variations), only one occurrence of each stimulus was presented. In Experiment 3, categories were the same as in Experiment 1: Stroking, Rubbing, Tapping, Hitting. In Experiment 4, categories were rephrased in order to better capture the fact that certain touches were used by the toucher as a way to express their emotion (i.e., ‘The person touching is trying to convey their joy / anger’ for the ‘Joy’ (respectively the ‘Anger’) stimulus), while other touches underlying the sounds were used by the toucher to have an effect on the person receiving them (i.e., ‘The person touching is trying to get the other person’s attention / is trying to comfort the other person’ for the ‘Attention’ (respectively for the ‘Sympathy’) stimulus).

In the following block for Experiments 2 (i.e., fourth block) and 4 (i.e., third block), participants were asked to listen to each stimulus and rate their valence on two visual analog scales for Experiment 2 and one visual analog scale for Experiment 4. In Experiment 2, there was one scale to rate the pleasantness of the sound itself, “How pleasant is it to listen to this sound?”, and one for the touch underlying the sound, “How pleasant is the tactile experience evoked by this sound?”, ranging from “0 - Very unpleasant” to “100 - Very pleasant”. Only the latter was presented to participants in Experiment 4’s third block.

In the final block of Experiments 3 and 4, participants were told that stimuli were either skin-on-skin or object-on-object tactile interactions. They then had to categorize each stimulus in the ‘Surface’ category they thought it belonged to.

# Data analysis

## Experiments 1 and 2

For the first and second blocks of Experiments 1 and 2, a qualitative data analysis was necessary to process the participants’ free-text responses. 5 naive external raters were recruited for each experiment to classify participants’ answers. For Experiment 1, they were instructed to assign the first block’s answers into three categories: “*Touch*”, “*Not a touch*”, “*Not a sound*” and the second block’s answers into five categories: “*Stroking*”, “*Tapping*”, “*Rubbing*”, “*Hitting*”, “*Other*”. For Experiment 2, they were instructed to assign the first block’s answers into eight categories: “*Love*”, “*Anger*”, “*Sympathy*”, “*Joy*”, “*Attention*”, “*Fear*”, “*Other emotion*”, “*Not an emotion*” and the second block’s answers into nine categories; “*Love*”, “*Anger*”, “*Sympathy*”, “*Joy*”, “*Attention*”, “*Fear*”, “*Other emotion or intention*”, “*Not an emotion or intention*”. Coders were not given any definition during this annotation task, to avoid any bias.

Responses from participants that were classified in the same category by at least 60% of the raters (>=3) were considered for further analysis (Experiment 1: 93.6% of the answers for Block 1, 96.1% of the responses for Block 2; Experiment 2: 92.3% of the answers for Block 1, 81.7% of the answers for Block 2). All other responses were discarded from the analysis for these blocks.

For both experiments, for the first and second blocks, for each stimulus, the rate of responses attributed to each category was computed. Chi-square goodness-of-fit tests distinguishing each target cue were then conducted to determine whether the distribution of participants’ responses into these categories was random and, if not, showed participants’ preferences. To visually represent the semantic concepts to which participants’ free-text responses belonged and their frequency, the Term Document Matrix R package was combined with the Word Clouds R package (along Schirmer et al., 43).

For the third block, because the same measures were repeated ten times for each stimulus on the same participants, the assumption of independence between the trials could be violated. To control for a potential effect of being repetitively exposed to the same stimuli and the same question (which will be referred to as an “effect of time” hereafter), we relied on a Generalized Linear Mixed-Effects Model using the *glmer* function from the ‘lme4’ package (SI2), with a logit link between variables and a binomial distribution of errors. Type of stimulus and time effect were added as fixed effects and participant ID as a random effect, as measures were repeated per participant. Participants’ categorical answers for each trial were transformed into a binary variable reporting their accuracy at correctly discriminating the tactile gesture (for Experiment 1) or the intended emotion (for Experiment 2) or not. For each trial, the sub-block number in which the stimulus was presented was stored in a discrete ‘time’ variable, going from 1 to 10. At this point, only the right or wrong responses were to be considered, not the confusion
between categories. Then, a binomial logistic regression was conducted, as several criteria were met: the number of events (i.e., number of correct responses) was at least ten times greater than the number of parameters of the models for each experiment (i.e., Experiment 1: 8 parameters, 1881 events; Experiment 2: 12 parameters, 1433 events), and there was no overdispersion in the data (i.e., criterion φ<1.5; Experiment 1: φ=0.4; Experiment 2: φ=1.3). The results of this binary logistic regression indicated no significant effect of time on participants' accuracy in either experiment (Experiment 1: OR=1.08, p=0.4, 95% CI [0.9; 1.3]; Experiment 2: OR=1.1, p=0.1; 95% CI [1; 1.1]). Thus, we were able to conclude that the repetitive design inside of Block 3 did not significantly influence participants' responses, and then consider Block 3 trials in both experiments as independent for further analysis.

Chi-square goodness-of-fit tests were then conducted to determine whether the distribution of participants' responses into the target categories was random or not. Then, for each condition, post-hoc analysis using Wilcoxon rank tests was conducted to find the preferred category among the various ones that were presented. To do so, the groups were paired in descending order (i.e., the category with the highest frequency was compared to the one with the second highest frequency, then this group was compared to the third highest, and so on), similarly to Kirsch et al. (11).

For the fourth block of Experiment 2, a repeated-measures ANOVA was conducted using the target category (i.e., the different socio-emotional intentions) as a factor, to study potential significant differences in the ratings of the stimuli. A Greenhouse-Geisser correction was applied if there was any violation of sphericity. Finally, post-hoc t-tests using Bonferroni correction were run to compare the significantly different mean ratings of the various audio-touch stimuli.

### Experiments 3 and 4

In the first block of Experiments 3 and 4, the number of free-text responses was high (1500+) and their length variable. Thus, we did not conduct the same qualitative data analysis on these answers as in Experiments 1 and 2. No statistical test was done on these data, only a topic modeling package was used in RStudio to visually represent participants' answers (see Figure S5).

In Experiments 3 and 4, the second block was a forced-choice categorization task. For each stimulus, the rate of responses attributed to each category was computed. Chi-square goodness-of-fit tests distinguishing each target cue were then conducted to determine whether the distribution of participants' responses into these categories was random or not. In Experiment 3, results show that participants' accuracy was significantly different depending on whether the gesture was sonified on Skin or on Object for the Rubbing ($\chi^2$ (2, N=84)=30.3, p<0.0001), Stroking ($\chi^2$ (3, N=84)=38.3, p<0.0001) and Hitting categories ($\chi^2$ (2, N=84)=9.4, p=0.009). Nonetheless, the difference between surfaces was not significant for the sonified Tapping gesture ($\chi^2$ (3, N=84)=3.3, p=0.35). In Experiment 4, results show that participants' accuracy was significantly different depending on whether the gesture was sonified on Skin or on Object for all stimuli: for Joy: $\chi^2$ (3, N=79)=18.2, p=0.0004, for Sympathy: $\chi^2$ (3, N=79)=57.3, p<0.0001; for Anger: $\chi^2$ (3, N=79)=14.6, p=0.002; and for Attention: $\chi^2$ (3, N=79)=38.2, p<0.0001.

Then, for each condition, post-hoc analysis using Wilcoxon rank tests was conducted to find the preferred category among those that were presented. To do so, the groups were paired in descending order (i.e., the category with the highest frequency was compared to the one with the second highest frequency, then this group was compared to the third highest, and so on).

Additionally, to assess the impact of Surface and Stimulus on participants' Accuracy in the 4-AFC task, we used again a Generalized Linear Mixed-Effects Model. It is represented by the formula:

$$glmer(Accuracy ~ Stimulus * Surface + (1|ID))$$

This model considers Accuracy as a binary response variable and explores the interaction between Stimulus and Surface, both treated as fixed effects. The three variations within the Surface variable were regrouped (i.e., Surface was a 2-level categorical variable here). The decision to include these variables as fixed effects was informed by Chi-square goodness-of-fit tests, which indicated their significant impact on participants' responses. Additionally, the random effect (1|ID) was introduced to account for individual variations attributable to the repeated measures associated with each participant's ID. The GLMM was fitted with a logit link and a binomial distribution of errors. The residuals were examined for normality and homoscedasticity, confirming the appropriateness of the model. To enhance the efficiency of the optimization process, a control parameter was
specified (i.e., this configuration employed the 'Bound Optimization by Quadratic Approximations' (bobyqa) algorithm as the optimizer). To analyze the main effects and interactions, the *Anova* function (*car* package) was used to conduct a Wald Type II Chi-square test on the model. We further analyzed post-hoc results using marginal means (*emmeans* package) and Holm correction for multiple comparisons whenever applicable.

In Experiment 4, the third block consisted in the valence ratings of the audio-touch stimuli. Similarly to the analysis conducted in Experiment 2, a repeated-measures ANOVA was conducted using the target category (i.e., the different socio-emotional intentions) as a factor, to study potential significant differences in the ratings of the various stimuli. A Greenhouse-Geisser correction was applied because of the violation of the assumption of sphericity. Finally, post-hoc t-tests using Bonferroni correction were run to compare the significantly different mean ratings of the various audio-touch stimuli.

In both Experiments 3 and 4, the last block of the study was a two-alternative forced-choice categorization task. For each stimulus, the rate of responses attributed to either of the two categories was computed. Chi-square goodness-of-fit tests distinguishing each target cue were then conducted to determine whether the distribution of participants’ responses into these categories was random or not. Then, for each condition, post-hoc analysis using t-tests was conducted to find the preferred category between the two that were presented.
Experiment 1. Perception of sonified skin-on-skin tactile gestures.

<table><caption>Supplementary Table S1. Standardized features of four distinct prototypical tactile gestures used in Experiment 1: rubbing, stroking, tapping, and hitting.</caption><thead><tr><th>Type of touch</th><th>Rubbing</th><th>Stroking</th><th>Tapping</th><th>Hitting</th></tr></thead><tbody><tr><th>Velocity</th><td>Fast (18 cm/s)</td><td>Slow (6 cm/s)</td><td>Fast</td><td>Fast</td></tr><tr><th>Tactile continuity</th><td>Continuous back and forth</td><td>Long one-way, discontinuous</td><td>Brief, repeated several times</td><td>Brief, repeated twice</td></tr><tr><th>Pressure</th><td>Moderate / Strong</td><td>Light</td><td>Light</td><td>Strong</td></tr><tr><th>Points of contact</th><td>Palm and all fingers</td><td>Palm and all fingers</td><td>Index</td><td>All fingers but thumb</td></tr><tr><th>Rhythm</th><td>Regular</td><td>Regular</td><td>Irregular</td><td>Regular</td></tr></tbody></table>

Gestures were chosen to produce sounds loud enough to be picked up by the microphone and be distinct for at least one of the physical characteristics (i.e., velocity, continuity, pressure, points of contact, rhythm), as described by prior research in the social touch literature (i.e., Hertenstein et al., 2009; McIntyre et al., 2022). For rubbing vs. stroking, the qualitative differences are velocity (respectively fast vs. slow), continuity (resp. continuous vs. discontinuous) and pressure (moderate vs. light). For tapping vs. hitting, the qualitative differences are continuity (tapping repeated several times while hitting is just twice), points of contact (respectively index vs. all fingers and palm), and rhythm (resp. irregular vs. regular). These four types of touch, belonging to the prototypical tactile gestures defined by the literature on social touch, were selected as they are sufficiently distinct one from another from various perspectives: (i) semantically, each gesture has a specific name and a corresponding unique definition; (ii) physically, each gesture has specific properties and (iii) auditorily, each gesture produces a unique sound, thus has a unique auditory signature. The experimenter was trained to replicate each movement with a consistent velocity, pressure, and rhythm across repetitions.
<figure><img src="image_7.png" /><figcaption>Supplementary Figure S1. Spectrograms for our skin-on-skin audio-touch stimuli expressing amplitude in frequency (Hz) over time in seconds (sampling frequency: 44100 Hz). The four tactile gestures (from top to bottom: Hitting, Rubbing, Stroking, Tapping) - were recorded via the apparatus described above.</figcaption></figure>
<table><caption>**Supplementary Table S2. Perception of sonified skin-on-skin tactile gestures without any clue. Contingency table for the results of the first block of Experiment 1.**</caption><thead><tr><th rowspan="2">Type of gesture</th><th rowspan="2"></th><th colspan="3">Response</th></tr><tr><th>Not a sound</th><th>Not a touch</th><th>Touch</th></tr></thead><tbody><tr><td rowspan="3">Stroking</td><td>Observed</td><td>2</td><td>19</td><td><strong>23</strong></td></tr><tr><td>Category %</td><td>4.5%</td><td>43.2%</td><td><strong>52.3%</strong></td></tr><tr><td>Standardized Residuals</td><td>-4.1</td><td>1.4</td><td><strong>2.7</strong></td></tr><tr><td rowspan="3">Rubbing</td><td>Observed</td><td>1</td><td>21</td><td><strong>25</strong></td></tr><tr><td>Category %</td><td>2.1%</td><td>44.7%</td><td><strong>53.2%</strong></td></tr><tr><td>SR</td><td>-4.5</td><td>1.7</td><td><strong>2.9</strong></td></tr><tr><td rowspan="3">Hitting</td><td>Observed</td><td>2</td><td>16</td><td><strong>31</strong></td></tr><tr><td>Category %</td><td>4.1%</td><td>32.7%</td><td><strong>63.3%</strong></td></tr><tr><td>SR</td><td>-4.5</td><td>1.6</td><td><strong>2.9</strong></td></tr><tr><td rowspan="3">Tapping</td><td>Observed</td><td>0</td><td>18</td><td><strong>33</strong></td></tr><tr><td>Category %</td><td>0%</td><td>35.3%</td><td><strong>64.7%</strong></td></tr><tr><td>SR</td><td>-4.3</td><td>-0.1</td><td><strong>4.5</strong></td></tr></tbody></table>

Bold font emphasizes the *a posteriori* categorization of the participants' free-text responses considered as 'correct' in our setting.
<table><caption>**Supplementary Table S3.** Number of occurrences of the words with the most occurrences in the wordcloud representing the first block of Experiment 1 (for all audio-touch stimuli).</caption><thead><tr><td>Word with maximum occurrences</td><td>Number of occurrences</td></tr></thead><tbody><tr><td>Heartbeat</td><td>17</td></tr><tr><td>Rubbing</td><td>17</td></tr><tr><td>Tapping</td><td>14</td></tr><tr><td>Wind</td><td>13</td></tr><tr><td>Knocking</td><td>12</td></tr></tbody></table>
<table><caption>Supplementary Table S4. Perception of sonified skin-on-skin tactile gestures knowing it is a skin-on-skin touch. Contingency table for the results of the second block of Experiment 1.</caption><thead><tr><th rowspan="2">Type of gesture</th><th rowspan="2"></th><th colspan="5">Response</th></tr><tr><th>Stroking</th><th>Rubbing</th><th>Hitting</th><th>Tapping</th><th>Other</th></tr></thead><tbody><tr><td rowspan="3">Stroking</td><td>Observed</td><td><strong>34</strong></td><td>14</td><td>0</td><td>0</td><td>2</td></tr><tr><td>Category %</td><td><strong>68%</strong></td><td>28%</td><td>0%</td><td>0%</td><td>4%</td></tr><tr><td>Standardized Residuals</td><td><strong>10.3</strong></td><td>-0.1</td><td>-4.9</td><td>-4.2</td><td>-0.6</td></tr><tr><td rowspan="3">Rubbing</td><td>Observed</td><td>3</td><td><strong>42</strong></td><td>0</td><td>0</td><td>4</td></tr><tr><td>Category %</td><td>8.2%</td><td><strong>85.7%</strong></td><td>0%</td><td>0%</td><td>6.1%</td></tr><tr><td>SR</td><td>-2.6</td><td><strong>10.2</strong></td><td>-4.9</td><td>-4.1</td><td>0.9</td></tr><tr><td rowspan="3">Hitting</td><td>Observed</td><td>0</td><td>0</td><td><strong>37</strong></td><td>8</td><td>4</td></tr><tr><td>Category %</td><td>0%</td><td>0%</td><td><strong>75.5%</strong></td><td>16.3%</td><td>8.2%</td></tr><tr><td>SR</td><td>-3.9</td><td>-5.1</td><td><strong>9</strong></td><td>-0.8</td><td>0.9</td></tr><tr><td rowspan="3">Tapping</td><td>Observed</td><td>0</td><td>0</td><td>15</td><td><strong>32</strong></td><td>1</td></tr><tr><td>Category %</td><td>0%</td><td>0%</td><td>31.3%</td><td><strong>66.7%</strong></td><td>2.1%</td></tr><tr><td>SR</td><td>-3.9</td><td>-5</td><td>0.9</td><td><strong>9.2</strong></td><td>-1.2</td></tr></tbody></table>

Bold font emphasizes the *a posteriori* categorization of the participants' free-text responses considered as 'correct' in our setting.
<table><caption>**Supplementary Table S5.** Number of occurrences of the words with the most occurrences in the wordcloud representing the second block of Experiment 1 (for all audio-touch stimuli).</caption><thead><tr><td>Word with maximum occurrences</td><td>Number of occurrences</td></tr></thead><tbody><tr><td>Tapping</td><td>42</td></tr><tr><td>Hitting</td><td>32</td></tr><tr><td>Stroking</td><td>30</td></tr><tr><td>Rubbing</td><td>23</td></tr></tbody></table>
<table><caption>**Supplementary Table S6. Categorization of sonified skin-on-skin tactile gestures.** Contingency table for the forced-choice categorization task of experiment 1.</caption><thead><tr><th rowspan="2">Condition</th><th rowspan="2">χ<sup>2</sup></th><th rowspan="2">df</th><th rowspan="2">p-value</th><th colspan="4">Response</th></tr><tr><th>Stroking</th><th>Rubbing</th><th>Hitting</th><th>Tapping</th></tr></thead><tbody><tr><td rowspan="3">Stroking</td><td rowspan="3">1046.9</td><td rowspan="3">3</td><td rowspan="3">&lt; 0.001</td><td>Observed</td><td><strong>440</strong></td><td>70</td><td>0</td><td>0</td></tr><tr><td>Category %</td><td><strong>86.3%</strong></td><td>13.7</td><td>0</td><td>0</td></tr><tr><td>Standardized Residuals</td><td><strong>32</strong></td><td>-5.9</td><td>-13</td><td>-13</td></tr><tr><td rowspan="3">Rubbing</td><td rowspan="3">1281.2</td><td rowspan="3">3</td><td rowspan="3">&lt; 0.001</td><td>Observed</td><td>29</td><td><strong>477</strong></td><td>2</td><td>2</td></tr><tr><td>Category %</td><td>5.7%</td><td><strong>93.5%</strong></td><td>0.4%</td><td>0.4%</td></tr><tr><td>SR</td><td>-10.1</td><td><strong>35.7</strong></td><td>-12.8</td><td>-12.8</td></tr><tr><td rowspan="3">Hitting</td><td rowspan="3">1361.3</td><td rowspan="3">3</td><td rowspan="3">&lt; 0.001</td><td>Observed</td><td>1</td><td>0</td><td><strong>488</strong></td><td>21</td></tr><tr><td>Category %</td><td>0.2%</td><td>0</td><td><strong>95.7%</strong></td><td>4.1%</td></tr><tr><td>SR</td><td>-12.9</td><td>-13</td><td><strong>36.4</strong></td><td>-10.4</td></tr><tr><td rowspan="3">Tapping</td><td rowspan="3">1325</td><td rowspan="3">3</td><td rowspan="3">&lt; 0.001</td><td>Observed</td><td>1</td><td>0</td><td>26</td><td><strong>483</strong></td></tr><tr><td>Category %</td><td>0.2%</td><td>0</td><td>5.1%</td><td><strong>94.7%</strong></td></tr><tr><td>SR</td><td>-12.9</td><td>-13</td><td>-10.9</td><td><strong>36.9</strong></td></tr></tbody></table>

Contingency table presenting the observed counts, percentages and standardized residuals of each stimulus per type of touch in the forced-choice categorization task of experiment 1; as well as the Chi-square results for each category. Bold font emphasizes participants' responses accurately categorized.
<table><caption>Supplementary Table S7. Wilcoxon rank tests table for post-hoc Chi-square analysis of the third block of Experiment 1.</caption><thead><tr><th>Cue</th><th>Compared categories</th><th>Z (Wilcoxon W)</th><th>p-value</th></tr></thead><tbody><tr><td rowspan="3">Rubbing</td><td>Rubbing vs. Stroking</td><td>-9 (W= 2573)</td><td><strong>&lt; 0.0001</strong></td></tr><tr><td>Stroking vs. Tapping</td><td>-2.7 (W= 1533)</td><td><strong>0.007</strong></td></tr><tr><td>Tapping vs. Hitting</td><td>0 (W=1300)</td><td>1</td></tr><tr><td rowspan="3">Stroking</td><td>Stroking vs. Rubbing</td><td>-8.3 (W= 2514)</td><td><strong>&lt; 0.0001</strong></td></tr><tr><td>Rubbing vs. Tapping</td><td>-5.2 (W= 1861.5)</td><td><strong>&lt; 0.0001</strong></td></tr><tr><td>Tapping vs. Hitting</td><td>NA (W = 0)</td><td>NA</td></tr><tr><td rowspan="3">Tapping</td><td>Tapping vs. Hitting</td><td>-9.3 (W= 2594)</td><td><strong>&lt; 0.0001</strong></td></tr><tr><td>Hitting vs. Stroking</td><td>-2.7 (W= 1507.5)</td><td><strong>0.007</strong></td></tr><tr><td>Stroking vs. Rubbing</td><td>-1 (W= 1326)</td><td>0.3</td></tr><tr><td rowspan="3">Hitting</td><td>Hitting vs. Tapping</td><td>-9.4 (W= 2595.5)</td><td><strong>&lt; 0.0001</strong></td></tr><tr><td>Tapping vs. Stroking</td><td>-2.4 (W= 1480.5)</td><td>0.015</td></tr><tr><td>Stroking vs. Rubbing</td><td>-1 (W= 1326)</td><td>0.3</td></tr></tbody></table>

These tests were conducted to find the most chosen category among the four possible ones. To do so, the groups were paired in descending order (i.e., the category with the highest frequency was compared to the one with the second highest frequency, then this group was compared to the third highest, and so on). Z-scores, Wilcoxon W between parenthesis and p-values are reported. Significant p-values are in bold.
Experiment 2. Perception of sonified skin-on-skin tactile socio-emotional intentions.

<table><caption>Supplementary Table S8. Standardized features of prototypical tactile gestures conveying six distinct socio-emotional intentions: Anger, Attention, Fear, Joy, Love, Sympathy.</caption><thead><tr><th>Emotion</th><th>Anger</th><th>Attention</th><th>Fear</th><th>Joy</th><th>Love</th><th>Sympathy</th></tr></thead><tbody><tr><th>Tactile gesture</th><td>Hitting*</td><td>Tapping*</td><td>Trembling</td><td>Tapping</td><td>Stroking*</td><td>Rubbing</td></tr><tr><th>Velocity</th><td>Fast</td><td>Fast</td><td>Fast (18 cm/sec)</td><td>Fast</td><td>Slow (6 cm/sec)</td><td>Moderate (12 cm/sec)</td></tr><tr><th>Pressure</th><td>Strong</td><td>Strong</td><td>Moderate</td><td>Light</td><td>Light</td><td>Moderate</td></tr><tr><th>Points of contact</th><td>All fingers but thumb</td><td>Tip of fingers</td><td>Palm and all fingers</td><td>Tip of fingers</td><td>Palm and all fingers</td><td>Palm and all fingers</td></tr><tr><th>Rhythm</th><td>Regular</td><td>Regular</td><td>Irregular</td><td>Irregular</td><td>Regular</td><td>Regular</td></tr></tbody></table>

Gestures were chosen to be distinct across all physical characteristics (i.e., velocity, continuity, pressure, points of contact, rhythm), as described by prior research in the social touch literature (i.e., Hertenstein et al., 2009; McIntyre et al., 2022). The experimenter was trained to replicate each movement consistently across repetitions. The six socio-emotional intentions used in this study had to be sufficiently distinct from various perspectives: (i) conceptually, each emotion has a specific name and a corresponding definition; (ii) physically, each emotion can be associated with a tactile expression that bears specific properties; (iii) auditorily, each gesture conveying the emotion produces a unique sound, which can be caught by the violin microphone. In that sense, for instance, even though McIntyre et al. (14) offered a detailed account of the tactile interaction spontaneously used by participants to express sadness, the associated gesture (i.e., a gentle “gripping with the whole hand” on the forearm) does not produce enough tactile waves to be picked up by the contact microphone, so this emotion was not sonified. Each gesture conveying a specific socio-emotional intention was sonified once, via this apparatus. * These stimuli are the same as those presented in Experiment 1.
<figure><img src="image_8.png" /><figcaption>**Supplementary Figure S2.** Spectrograms for our skin-on-skin audio-touch stimuli expressing amplitude in frequency (Hz) over time in seconds (sampling frequency: 44100 Hz). They were produced by different tactile gestures previously reported to convey specific socio-emotional intentions, from top to bottom: Anger, Attention, Fear, Joy, Love and Sympathy.</figcaption></figure>

<caption>**Supplementary Table S9.** Perception of sonified skin-on-skin tactile socio-emotional intentions without any clue. Contingency table for the results of the first block of Experiment 2.</caption>
<table><thead><tr><th>Emotion</th><th></th><th>Anger</th><th>Attention</th><th>Fear</th><th>Joy</th><th>Love</th><th>Sympathy</th><th>Not an emotion</th><th>Other emotion</th></tr></thead><tbody><tr><td rowspan="3">Anger</td><td>Observed</td><td><strong>10</strong></td><td>1</td><td>14</td><td>0</td><td>0</td><td>0</td><td>8</td><td>16</td></tr><tr><td>Category %</td><td><strong>20.4%</strong></td><td>2%</td><td>28.6%</td><td>0%</td><td>0%</td><td>0%</td><td>16.3%</td><td>32.6%</td></tr><tr><td>Standardised Residuals (SR)</td><td><strong>1.7</strong></td><td>-2.2</td><td>3.4</td><td>-2.7</td><td>-2.7</td><td>-2.7</td><td>0.8</td><td>4.3</td></tr><tr><td rowspan="3">Attention</td><td>Observed</td><td>5</td><td><strong>6</strong></td><td>21</td><td>1</td><td>2</td><td>0</td><td>4</td><td>8</td></tr><tr><td>Category %</td><td>10.6%</td><td><strong>12.8%</strong></td><td>44.7%</td><td>2.1%</td><td>4.3%</td><td>0%</td><td>8.5%</td><td>17%</td></tr><tr><td>SR</td><td>-0.4</td><td><strong>0.1</strong></td><td>6.7</td><td>-2.2</td><td>-1.7</td><td>-2.6</td><td>-0.8</td><td>0.9</td></tr><tr><td rowspan="3">Fear</td><td>Observed</td><td>7</td><td>6</td><td><strong>26</strong></td><td>3</td><td>0</td><td>0</td><td>3</td><td>1</td></tr><tr><td>Category %</td><td>15.2%</td><td>13%</td><td><strong>56.5%</strong></td><td>6.5%</td><td>0%</td><td>0%</td><td>6.5%</td><td>2.2%</td></tr><tr><td>SR</td><td>0.6</td><td>0.1</td><td><strong>9</strong></td><td>-1.2</td><td>-2.6</td><td>-2.6</td><td>-1.2</td><td>-2.1</td></tr><tr><td rowspan="3">Joy</td><td>Observed</td><td>1</td><td>3</td><td>12</td><td><strong>13</strong></td><td>0</td><td>1</td><td>8</td><td>11</td></tr><tr><td>Category %</td><td>2%</td><td>6.1%</td><td>24.5%</td><td><strong>26.5%</strong></td><td>0%</td><td>2%</td><td>16.3%</td><td>22.5%</td></tr><tr><td>SR</td><td>-2.2</td><td>-2.2</td><td>2.5</td><td><strong>-1.4</strong></td><td>3</td><td>-2.7</td><td>0.8</td><td>2.1</td></tr><tr><td rowspan="3">Love</td><td>Observed</td><td>1</td><td>2</td><td>16</td><td>5</td><td><strong>1</strong></td><td>1</td><td>13</td><td>13</td></tr><tr><td>Category %</td><td>1.9%</td><td>3.9%</td><td>30.8%</td><td>9.6%</td><td><strong>1.9%</strong></td><td>1.9%</td><td>25%</td><td>25%</td></tr><tr><td>SR</td><td>-2.3</td><td>-1.9</td><td>4</td><td>-0.6</td><td><strong>-2.3</strong></td><td>-2.3</td><td>2.7</td><td>2.7</td></tr><tr><td rowspan="3">Sympathy</td><td>Observed</td><td>2</td><td>1</td><td>14</td><td>3</td><td>1</td><td><strong>2</strong></td><td>6</td><td>18</td></tr><tr><td>Category %</td><td>4.3%</td><td>2.3%</td><td>29.8%</td><td>6.5%</td><td>2.3%</td><td><strong>4.3%</strong></td><td>12.8%</td><td>38.3%</td></tr><tr><td>SR</td><td>-1.7</td><td>-2.1</td><td>3.6</td><td>-1.3</td><td>-2.2</td><td><strong>-1.7</strong></td><td>0.1</td><td>5.4</td></tr></tbody></table>

Bold font emphasizes the *a posteriori* categorization of the participants' free-text responses considered as 'correct' in our setting.
<table><caption>**Supplementary Table S10.** Number of occurrences of the words with the most occurrences in the wordcloud representing the first block of Experiment 2.</caption><thead><tr><th>Emotion</th><th>Word with maximum occurrences</th><th>Number of occurrences</th></tr></thead><tbody><tr><td rowspan="3">Anger</td><td>Fear</td><td>9</td></tr><tr><td>Surprise</td><td>7</td></tr><tr><td>Anger</td><td>4</td></tr><tr><td rowspan="2">Attention</td><td>Fear</td><td>5</td></tr><tr><td>Stress</td><td>4</td></tr><tr><td rowspan="2">Fear</td><td>Stress</td><td>13</td></tr><tr><td>Impatience</td><td>5</td></tr><tr><td rowspan="3">Joy</td><td>Joy</td><td>7</td></tr><tr><td>Stress</td><td>4</td></tr><tr><td>Fear</td><td>4</td></tr><tr><td rowspan="2">Love</td><td>Soothing</td><td>6</td></tr><tr><td>Anxiety</td><td>3</td></tr><tr><td rowspan="3">Sympathy</td><td>Calm</td><td>7</td></tr><tr><td>Stress</td><td>4</td></tr><tr><td>Soothing</td><td>3</td></tr></tbody></table>
<table><caption>Supplementary Table S11. Perception of sonified skin-on-skin tactile socio-emotional intentions knowing it is skin-on-skin touch. Contingency table for the results of Experiment 2 second block.</caption><thead><tr><th>Emotion</th><th></th><th>Anger</th><th>Attention</th><th>Fear</th><th>Joy</th><th>Love</th><th>Sympathy</th><th>Not an emotion</th><th>Other emotion</th></tr></thead><tbody><tr><td rowspan="3">Anger</td><td>Observed</td><td><strong>10</strong></td><td>0</td><td>6</td><td>3</td><td>0</td><td>1</td><td>7</td><td>12</td></tr><tr><td>Category %</td><td><strong>25.6%</strong></td><td>0%</td><td>15.4%</td><td>7.7%</td><td>0%</td><td>2.6%</td><td>18%</td><td>31%</td></tr><tr><td>SR</td><td><strong>2.3</strong></td><td>-2.4</td><td>0.5</td><td>-0.9</td><td>-2.4</td><td>-1.9</td><td>1</td><td>3.5</td></tr><tr><td rowspan="3">Attention</td><td>Observed</td><td>2</td><td><strong>10</strong></td><td>7</td><td>1</td><td>1</td><td>1</td><td>4</td><td>19</td></tr><tr><td>Category %</td><td>4.4%</td><td><strong>22.2%</strong></td><td>15.6%</td><td>2.2%</td><td>2.2%</td><td>2.2%</td><td>8.9%</td><td>42.2%</td></tr><tr><td>SR</td><td>-1.6</td><td><strong>2</strong></td><td>0.6</td><td>-2.1</td><td>-2.1</td><td>-2.1</td><td>-0.7</td><td>6</td></tr><tr><td rowspan="3">Fear</td><td>Observed</td><td>4</td><td>6</td><td><strong>20</strong></td><td>1</td><td>2</td><td>4</td><td>3</td><td>5</td></tr><tr><td>Category %</td><td>8.9%</td><td>13.3%</td><td><strong>44.4%</strong></td><td>2.2%</td><td>4.4%</td><td>8.9%</td><td>6.7%</td><td>11.1%</td></tr><tr><td>SR</td><td>-0.7</td><td>0.2</td><td><strong>6.5</strong></td><td>-2.1</td><td>-1.6</td><td>-0.7</td><td>-1.2</td><td>-0.3</td></tr><tr><td rowspan="3">Joy</td><td>Observed</td><td>0</td><td>6</td><td>2</td><td><strong>13</strong></td><td>3</td><td>1</td><td>7</td><td>8</td></tr><tr><td>Category %</td><td>0%</td><td>15%</td><td>5%</td><td><strong>32.5%</strong></td><td>7.5%</td><td>2.5%</td><td>17.5%</td><td>20%</td></tr><tr><td>SR</td><td>-2.4</td><td>0.5</td><td>-1.4</td><td><strong>3.8</strong></td><td>-1</td><td>-1.9</td><td>1</td><td>1.4</td></tr><tr><td rowspan="3">Love</td><td>Observed</td><td>1</td><td>0</td><td>1</td><td>2</td><td><strong>15</strong></td><td>8</td><td>6</td><td>6</td></tr><tr><td>Category %</td><td>2.6%</td><td>0%</td><td>2.6%</td><td>5.1%</td><td><strong>38.5%</strong></td><td>20.5%</td><td>15.4%</td><td>15.4%</td></tr><tr><td>SR</td><td>-1.9</td><td>-2.4</td><td>-1.9</td><td>-1.4</td><td><strong>4.9</strong></td><td>1.5</td><td>0.5</td><td>0.5</td></tr><tr><td rowspan="3">Sympathy</td><td>Observed</td><td>1</td><td>0</td><td>0</td><td>3</td><td>14</td><td><strong>13</strong></td><td>7</td><td>9</td></tr><tr><td>Category %</td><td>2.1%</td><td>0%</td><td>0%</td><td>6.4%</td><td>29.8%</td><td><strong>27.7%</strong></td><td>14.9%</td><td>19.2%</td></tr><tr><td>SR</td><td>-2.2</td><td>-2.6</td><td>-2.6</td><td>-1.3</td><td>3.6</td><td><strong>3.1</strong></td><td>0.5</td><td>1.4</td></tr></tbody></table>

Bold font emphasizes the *a posteriori* categorization of the participants' free-text responses considered as 'correct' in our setting.
<table><caption>**Supplementary Table S12.** Number of occurrences of the words with the most occurrences in the wordcloud representing the first block of Experiment 2.</caption><thead><tr><th>Emotion</th><th>Word with maximum occurrences</th><th>Number of occurrences</th></tr></thead><tbody><tr><td rowspan="2">Anger</td><td>Anger</td><td>6</td></tr><tr><td>Hitting</td><td>3</td></tr><tr><td rowspan="2">Attention</td><td>Impatience</td><td>5</td></tr><tr><td>Awaken</td><td>5</td></tr><tr><td rowspan="3">Fear</td><td>Stress</td><td>6</td></tr><tr><td>Fear</td><td>3</td></tr><tr><td>Warm</td><td>3</td></tr><tr><td rowspan="3">Joy</td><td>Music</td><td>7</td></tr><tr><td>Joy</td><td>3</td></tr><tr><td>Impatience</td><td>2</td></tr><tr><td rowspan="2">Love</td><td>Tenderness</td><td>7</td></tr><tr><td>Calm</td><td>4</td></tr><tr><td rowspan="3">Sympathy</td><td>Comforting</td><td>6</td></tr><tr><td>Tenderness</td><td>5</td></tr><tr><td>Affection</td><td>5</td></tr></tbody></table>
<table><caption>Supplementary Table S13. Categorization of sonified socio-emotional intentions underlying skin-on-skin touch.</caption><thead><tr><th rowspan="2">Emotion</th><th rowspan="2">χ<sup>2</sup></th><th rowspan="2">df</th><th rowspan="2">p-value</th><th colspan="6">Response</th></tr><tr><th>Anger</th><th>Attention</th><th>Fear</th><th>Joy</th><th>Love</th><th>Sympathy</th></tr></thead><tbody><tr><td rowspan="3">Anger</td><td rowspan="3">606.3</td><td rowspan="3">5</td><td rowspan="3">&lt; 0.001</td><td>Observed</td><td><strong>271</strong></td><td>56</td><td>121</td><td>24</td><td>12</td><td>16</td></tr><tr><td>Category %</td><td><strong>54.2%</strong></td><td>11.2%</td><td>24.2%</td><td>4.8%</td><td>2.4%</td><td>3.2%</td></tr><tr><td>SR</td><td><strong>22.5</strong></td><td>-3.3</td><td>4.5</td><td>-7.1</td><td>-8.6</td><td>-8.1</td></tr><tr><td rowspan="3">Attention</td><td rowspan="3">460.2</td><td rowspan="3">5</td><td rowspan="3">&lt; 0.001</td><td>Observed</td><td>83</td><td><strong>236</strong></td><td>128</td><td>25</td><td>18</td><td>10</td></tr><tr><td>Category %</td><td>16.6%</td><td><strong>47.2%</strong></td><td>25.6%</td><td>5%</td><td>3.6%</td><td>2%</td></tr><tr><td>SR</td><td>-0.04</td><td><strong>18.3</strong></td><td>5.4</td><td>-7</td><td>-7.8</td><td>-8.8</td></tr><tr><td rowspan="3">Fear</td><td rowspan="3">312.4</td><td rowspan="3">5</td><td rowspan="3">&lt; 0.001</td><td>Observed</td><td>87</td><td>194</td><td><strong>143</strong></td><td>27</td><td>17</td><td>32</td></tr><tr><td>Category %</td><td>17.4%</td><td>38.8%</td><td><strong>28.6%</strong></td><td>5.4%</td><td>3.4%</td><td>6.4%</td></tr><tr><td>SR</td><td>0.4</td><td>13.3</td><td><strong>7.2</strong></td><td>-6.8</td><td>-8</td><td>-6.1</td></tr><tr><td rowspan="3">Joy</td><td rowspan="3">962.9</td><td rowspan="3">5</td><td rowspan="3">&lt; 0.001</td><td>Observed</td><td>9</td><td>113</td><td>20</td><td><strong>329</strong></td><td>15</td><td>14</td></tr><tr><td>Category %</td><td>1.8%</td><td>22.6%</td><td>4%</td><td><strong>65.8</strong><br/><strong>%</strong></td><td>3%</td><td>2.8%</td></tr><tr><td>SR</td><td>-8.9</td><td>3.6</td><td>-7.6</td><td><strong>29.5</strong></td><td>-8.2</td><td>-8.3</td></tr><tr><td rowspan="3">Love</td><td rowspan="3">704.4</td><td rowspan="3">5</td><td rowspan="3">&lt; 0.001</td><td>Observed</td><td>7</td><td>20</td><td>17</td><td>19</td><td><strong>175</strong></td><td>262</td></tr><tr><td>Category %</td><td>1.4%</td><td>4%</td><td>3.4%</td><td>3.8%</td><td><strong>35%</strong></td><td>52.4%</td></tr><tr><td>SR</td><td>-9.2</td><td>-7.6</td><td>-8</td><td>-7.7</td><td><strong>11</strong></td><td>21.4</td></tr><tr><td rowspan="3">Sympathy</td><td rowspan="3">840.7</td><td rowspan="3">5</td><td rowspan="3">&lt; 0.001</td><td>Observed</td><td>6</td><td>13</td><td>7</td><td>12</td><td>183</td><td><strong>279</strong></td></tr><tr><td>Category %</td><td>1.2%</td><td>2.6%</td><td>1.4%</td><td>2.4%</td><td>36.6%</td><td><strong>55.8%</strong></td></tr><tr><td>SR</td><td>-9.3</td><td>-8.4</td><td>-9.2</td><td>-8.6</td><td>12</td><td><strong>23.5</strong></td></tr></tbody></table>

Contingency table presenting the observed counts, percentages and standardized residuals of each stimulus per type of emotion conveyed through touch in the forced-choice categorization task of Experiment 2. Bold font indicates participants' accurate categorization.
<table><caption>Supplementary Table S14. Post-hoc analysis on the categorization of sonified socio-emotional intentions underlying skin-on-skin touch.</caption><thead><tr><th>Cue</th><th>Compared categories</th><th>Z (Wilcoxon W)</th><th>p-value</th></tr></thead><tbody><tr><td rowspan="5">Anger</td><td>Anger vs. Fear</td><td>-3.3 (W= 1708.5)</td><td><strong>&lt;0.0001</strong></td></tr><tr><td>Fear vs. Attention</td><td>-1 (W= 1385.5)</td><td>0.3</td></tr><tr><td>Attention vs. Joy</td><td>-3.5 (W= 1637.5)</td><td><strong>&lt;0.0001</strong></td></tr><tr><td>Joy vs. Sympathy</td><td>-0.3 (W= 1228.5)</td><td>0.8</td></tr><tr><td>Sympathy vs. Love</td><td>-0.3 (W= 1272)</td><td>0.8</td></tr><tr><td rowspan="5">Attention</td><td>Attention vs. Fear</td><td>-3.2 (W= 1705)</td><td><strong>0.001</strong></td></tr><tr><td>Fear vs. Anger</td><td>-1.2 (W= 1409.5)</td><td>0.2</td></tr><tr><td>Anger vs. Joy</td><td>-2.8 (W= 1588.5)</td><td><strong>0.006</strong></td></tr><tr><td>Joy vs. Love</td><td>-1.5 (W= 1390)</td><td>0.1</td></tr><tr><td>Love vs. Sympathy</td><td>-0.5 (W= 1209.5)</td><td>0.6</td></tr><tr><td rowspan="5">Fear</td><td>Fear vs. Attention</td><td>-1.6 (W= 1026)</td><td>0.1</td></tr><tr><td>Attention vs. Anger</td><td>-3.3 (W= 1707)</td><td><strong>&lt;0.0001</strong></td></tr><tr><td>Anger vs. Sympathy</td><td>-2.8 (W= 1576.5)</td><td><strong>0.005</strong></td></tr><tr><td>Sympathy vs. Joy</td><td>-0.8 (W= 1168)</td><td>0.4</td></tr><tr><td>Joy vs. Love</td><td>-2 (W= 1437.5)</td><td>0.04</td></tr><tr><td rowspan="5">Joy</td><td>Joy vs. Attention</td><td>-5 (W=1957)</td><td><strong>&lt;0.0001</strong></td></tr><tr><td>Attention vs. Fear</td><td>-3.9 (W=1727.5)</td><td><strong>&lt;0.0001</strong></td></tr><tr><td>Fear vs. Love</td><td>-0.6 (W= 1301)</td><td>0.6</td></tr><tr><td>Love vs. Sympathy</td><td>-0.23 (W= 1228)</td><td>0.8</td></tr><tr><td>Sympathy vs. Anger</td><td>-0.2 (W= 1234)</td><td>0.9</td></tr><tr><td rowspan="5">Love</td><td>Sympathy vs. Love</td><td>-2.4 (W=1597.5)</td><td>0.016</td></tr><tr><td>Love vs. Attention</td><td>-5.4 (W=1954.4)</td><td><strong>&lt;0.0001</strong></td></tr><tr><td>Attention vs. Joy</td><td>-0.2 (W=1271)</td><td>0.8</td></tr><tr><td>Joy vs. Fear</td><td>-0.8 (W=1322.5)</td><td>0.4</td></tr><tr><td>Fear vs. Anger</td><td>-0.4 (W=1280)</td><td>0.7</td></tr><tr><td rowspan="5">Sympathy</td><td>Sympathy vs. Love</td><td>-2.5 (W=1615)</td><td>0.011</td></tr><tr><td>Love vs. Attention</td><td>-5.9 (W=1994.5)</td><td><strong>&lt;0.0001</strong></td></tr><tr><td>Attention vs. Joy</td><td>-0.6 (W= 1204.5)</td><td>0.6</td></tr><tr><td>Joy vs. Fear</td><td>-0.1 (W= 1257)</td><td>0.9</td></tr></tbody></table>
Fear vs. Anger

-0.9 (W=1321.5)

0.4

Wilcoxon rank tests table for post-hoc Chi-square analysis of the categorization task. These tests were conducted to find the preferred category among the six presented ones. To do so, the groups were paired in descending order (i.e., the category with the highest frequency was compared to the one with the second highest frequency, then this group was compared to the third highest, and so on). Z-scores, Wilcoxon W between parenthesis and p-values are reported here. Significant p-values are in bold.
<table><caption>Supplementary Table S15. Valence ratings of sonified socio-emotional intentions underlying skin-on-skin touch.</caption><thead><tr><th></th><th></th><th>Mean Difference</th><th>SE</th><th>t</th><th>pbonf</th></tr></thead><tbody><tr><td rowspan="4">Anger</td><td>Attention</td><td>-8.4</td><td>4.4</td><td>-1.9</td><td>0.9</td></tr><tr><td>Fear</td><td>-1.9</td><td>4.4</td><td>-0.4</td><td>1</td></tr><tr><td>Joy</td><td>-42.3</td><td>4.4</td><td>-9.6</td><td><strong>&lt; 0.001</strong></td></tr><tr><td>Love</td><td>-57.9</td><td>4.4</td><td>-13.1</td><td><strong>&lt; 0.001</strong></td></tr><tr><td></td><td>Sympathy</td><td>-59</td><td>4.4</td><td>-13.3</td><td><strong>&lt; 0.001</strong></td></tr><tr><td rowspan="4">Attention</td><td>Fear</td><td>6.5</td><td>4.4</td><td>1.5</td><td>1</td></tr><tr><td>Joy</td><td>-34</td><td>4.4</td><td>-7.7</td><td><strong>&lt; 0.001</strong></td></tr><tr><td>Love</td><td>-49.5</td><td>4.4</td><td>-11.2</td><td><strong>&lt; 0.001</strong></td></tr><tr><td>Sympathy</td><td>-50.6</td><td>4.4</td><td>-11.4</td><td><strong>&lt; 0.001</strong></td></tr><tr><td rowspan="3">Fear</td><td>Joy</td><td>-40.4</td><td>4.4</td><td>-9.1</td><td><strong>&lt; 0.001</strong></td></tr><tr><td>Love</td><td>-56</td><td>4.4</td><td>-12.7</td><td><strong>&lt; 0.001</strong></td></tr><tr><td>Sympathy</td><td>-57.1</td><td>4.4</td><td>-12.9</td><td><strong>&lt; 0.001</strong></td></tr><tr><td rowspan="2">Joy</td><td>Love</td><td>-15.6</td><td>4.4</td><td>-3.5</td><td><strong>0.008</strong></td></tr><tr><td>Sympathy</td><td>-16.7</td><td>4.4</td><td>-3.8</td><td><strong>0.003</strong></td></tr><tr><td>Love</td><td>Sympathy</td><td>-1.1</td><td>4.4</td><td>-0.3</td><td>1</td></tr></tbody></table>

Post-hoc t-tests comparing the valence ratings of the touch underlying each audio-touch stimulus of Experiment 2. The p-values are adjusted for comparing a family of 15 (i.e., multiple comparison correction). Significant p-values are in bold.
<table><caption>Supplementary Table S16. Descriptive results of the valence ratings.</caption><thead><tr><th>Emotion</th><th>Mean Sound</th><th>SEM</th><th>Mean Touch</th><th>SEM</th></tr></thead><tbody><tr><td>Anger</td><td>16.3</td><td>3</td><td>17.6</td><td>2.8</td></tr><tr><td>Attention</td><td>29.4</td><td>2.8</td><td>26</td><td>2.8</td></tr><tr><td>Fear</td><td>19.1</td><td>2.9</td><td>19.5</td><td>2.8</td></tr><tr><td>Joy</td><td>58.3</td><td>3.5</td><td>59.9</td><td>3.8</td></tr><tr><td>Love</td><td>61.3</td><td>3.9</td><td>75.5</td><td>3.3</td></tr><tr><td>Sympathy</td><td>67.9</td><td>3.6</td><td>76.6</td><td>3.3</td></tr></tbody></table>

Descriptive statistics presenting the mean and standard error of the mean of the valence ratings of the sounds (left) produced by audio-touch stimuli and of the touch underlying them (right), depending on the emotion conveyed.
<table><caption>Supplementary Table S17. Power analyses for Experiments 1 & 2.</caption><thead><tr><th></th><th>Number of observations required by a priori power analysis</th><th>Number of observations theoretically obtained with our N=52 sample size</th><th>Actual number of observations after responses categorization by external annotators</th><th>χ<sup>2</sup></th><th>df</th><th>Effect size obtained (Cramer's V, Cohen's ω)</th><th>Power obtained (1-β) and critical χ<sup>2</sup></th></tr></thead><tbody><tr><td colspan="8" style="text-align: center; font-weight: bold;">Experiment 1</td></tr><tr><td>Block 1</td><td>108</td><td>208</td><td>191</td><td>92.3</td><td>2</td><td>NA (1x3 contingency table)</td><td>NA (1x3 contingency table)</td></tr><tr><td>Block 2</td><td>193</td><td>208</td><td>196</td><td>314.5</td><td>12</td><td>V=0.7<br>ω=1.3</td><td>1-β=1<br>critical χ<sup>2</sup>=32.9</td></tr><tr><td>Block 3</td><td>174</td><td>2080</td><td>2040</td><td>5010</td><td>9</td><td>V=0.9<br>ω=1.6</td><td>1-β=1<br>critical χ<sup>2</sup>=27.9</td></tr><tr><td colspan="8" style="text-align: center; font-weight: bold;">Experiment 2</td></tr><tr><td>Block 1</td><td>291</td><td>312</td><td>290</td><td>94.3</td><td>35</td><td>V=0.3<br>ω=0.6</td><td>1-β=0.99<br>critical χ<sup>2</sup>=66.6</td></tr><tr><td>Block 2</td><td>291</td><td>312</td><td>255</td><td>191.3</td><td>35</td><td>V=0.4<br>ω=0.9</td><td>1-β=1<br>critical χ<sup>2</sup>=66.6</td></tr><tr><td>Block 3</td><td>254</td><td>3120</td><td>3000</td><td>3804.4</td><td>25</td><td>V=0.5<br>ω=1.1</td><td>1-β=1<br>critical χ<sup>2</sup>=52.6</td></tr></tbody></table>

In both experiments, for all three blocks, an a priori power analysis was conducted on G*Power to determine the required sample size necessary to achieve 80% power for detecting a medium effect (Cohen's ω = 0.3), at a significance level of α = 0.05. This is compared to the number of observations we were supposed to obtain with our samples made of 52 participants in each experiment, and to the actual number of observations included in the statistical analyses after participants' free-text responses (in block 1 and 2) were categorized by external annotators. The Chi-square test results and degrees of freedom for each block are presented, as well as the effect sizes obtained with the Chi-square goodness-of-fit test in both experiments for each block. Cramer's V and Cohen's ω were computed thanks to Cohen (1988) formulas. A post-hoc analysis was conducted with G*Power to compute the achieved power for each block as well as the critical Chi-square values, to check that the number of observations included in each block's analysis was sufficient to detect an effect.
Experiment 3. Influence of surface on the recognition of four sonified tactile gestures.

<figure><img src="image_9.png" /><figcaption>**Supplementary Figure S3. Material used for the object-on-object tactile gesture sonification.**</figcaption></figure>

The object-on-object tactile interactions were performed using the aluminum rounded piece (on the left), as a tool somewhat replicating the experimenter's finger. This piece entered in contact with each of the three polymethyl methacrylate pieces. They were identical in terms of composition but were differentiated by their thickness (from left to right: 0.5cm, 1.5cm, 2.5cm). This combination of inanimate surfaces was chosen after extensive testing of various materials touching each other, as it was the one producing sound loud enough to be recorded by our microphone with all chosen gestures.
<figure><img src="image_10.png" /><figcaption>**Supplementary Figure S4. Audio-touch stimuli spectrograms.** From top to bottom: 3 types of skin, then 3 types of object. A. Hitting stimuli. B. Rubbing stimuli. C. Stroking stimuli. D. Tapping stimuli. In Experiment 3, the exact same four gestures as in the main experiment were replicated once in each condition (i.e., 4 gestures (stroking, rubbing, tapping, and hitting) * 2 surfaces (Skin or Object) * 3 variations (3 different people's skin or 3 different pieces of PMMA). There were 24 audio-touch recordings in this Experiment 3.</figcaption></figure>
Perception of sonified skin-on-skin versus object-on-object tactile gestures without any clue.

A.

<figure><img src="image_11.png" /></figure>

B.

<figure><img src="image_12.png" /></figure>

C.

<figure><img src="image_13.png" /></figure>

**Supplementary Figure S5.** Wordclouds built with the Term Document Matrix R package combined with the Word Clouds R package (along Schirmer et al., 2022). All participants' responses were translated from French to English with DeepL.com. **A.** Global wordcloud taking all participants' responses. The threshold was set at 20 occurrences. **B.** Wordcloud after selecting participants' responses for the skin-on-skin stimuli only. The threshold was set at 15 occurrences. **C.** Wordcloud after selecting participants' responses for the object-on-object stimuli only. The threshold was set at 15 occurrences.
**Experiment 4. Influence of surface on the recognition of six socio-emotional intentions underlying sonified tactile gestures.**

<figure><img src="image_14.png" /></figure>

**Supplementary Figure S6. Audio-touch stimuli spectrograms.** From top to bottom: 3 types of skin, then 3 types of object. A. Anger stimuli. B. Attention stimuli. C. Joy stimuli. D. Sympathy stimuli. In Experiment 4, gestures expressing socio-affective intentions that were unambiguously categorized by participants in Experiment 2 were replicated once in each condition (i.e., 4 emotions (Anger, Attention, Joy and Sympathy) * 2 surfaces (Skin or Object) * 3 variations (3 different people's skin or 3 different pieces of PMMA). There were also 24 audio-touch stimuli in Experiment 4.

**Supplementary Table S18. Valence ratings of sonified socio-emotional intentions underlying skin-on-skin and object-on-object touch.**
<table><thead><tr><th>Emotion</th><th>Surface</th><th>Mean rating</th><th>SEM</th></tr></thead><tbody><tr><td rowspan="2">Anger</td><td>Skin</td><td>33.8</td><td>2.4</td></tr><tr><td>Object</td><td>31.4</td><td>2.1</td></tr><tr><td rowspan="2">Attention</td><td>Skin</td><td>44.9</td><td>2.5</td></tr><tr><td>Object</td><td>40.4</td><td>2.3</td></tr><tr><td rowspan="2">Joy</td><td>Skin</td><td>51</td><td>2.1</td></tr><tr><td>Object</td><td>48.7</td><td>2.5</td></tr><tr><td rowspan="2">Sympathy</td><td>Skin</td><td>36.1</td><td>3.1</td></tr><tr><td>Object</td><td>53.5</td><td>3.0</td></tr></tbody></table>

Descriptive statistics presenting the mean and standard error of the mean of the valence ratings of the touch underlying our audio-touch stimuli, depending on the emotion conveyed, and on the surface involved in the touch.
<table><caption>**Supplementary Table S19.** Post-hoc tests on valence ratings, using estimated marginal means.</caption><thead><tr><th>Stimulus</th><th>Contrast</th><th>Estimate</th><th>SE</th><th>df</th><th>Ratio</th><th>P-value</th></tr></thead><tbody><tr><td>Anger</td><td>Object - Skin</td><td>-2.4</td><td>3.1</td><td>155</td><td>-0.8</td><td>0.4</td></tr><tr><td>Attention</td><td>Object - Skin</td><td>-4.5</td><td>3.3</td><td>155</td><td>-1.3</td><td>0.2</td></tr><tr><td>Joy</td><td>Object - Skin</td><td>-2.3</td><td>3.3</td><td>155</td><td>-0.7</td><td>0.5</td></tr><tr><td>Sympathy</td><td>Object - Skin</td><td>-17.4</td><td>4.2</td><td>155</td><td>-4.1</td><td>0.0001</td></tr></tbody></table>
<table><caption>Supplementary Table S20. Power analyses for Experiments 3 & 4.</caption><thead><tr><th></th><th>Number of observations required by a priori power analysis</th><th>Number of observations theoretically obtained with our N=84 sample size</th><th>Actual number of observations with our effective sample size (N<sub>E3</sub>=84, N<sub>E4</sub>=79)</th><th>χ<sup>2</sup></th><th>df</th><th>Effect size obtained (Cramer's V, Cohen's ω)</th><th>Power obtained (1-β) and critical χ<sup>2</sup></th></tr></thead><tbody><tr><td colspan="8" style="text-align: center; font-weight: bold;">Experiment 3</td></tr><tr><td>4-AFC task</td><td>238</td><td>2016</td><td>2010</td><td>3750.5</td><td>21</td><td>V=0.8<br>ω=1.4</td><td>1-β=1<br>critical χ<sup>2</sup>=32.7</td></tr><tr><td colspan="8" style="text-align: center; font-weight: bold;">Experiment 4</td></tr><tr><td>4-AFC task</td><td>238</td><td>2016</td><td>1895</td><td>1411</td><td>21</td><td>V=0.5<br>ω=0.9</td><td>1-β=1<br>critical χ<sup>2</sup>=32.7</td></tr></tbody></table>

In both experiments, for the forced-choice categorization blocks, an a priori power analysis was conducted on G*Power to determine the required sample size necessary to achieve 80% power for detecting a medium effect (Cohen's ω = 0.3), at a significance level of α = 0.05. This is compared to the number of observations we were supposed to obtain with our samples made of 84 participants in each experiment, and to the actual number of observations included in the statistical analyses. The Chi-square test results and degrees of freedom for each block are presented, as well as the effect sizes obtained with the Chi-square goodness-of-fit test in both experiments for each block. Cramer's V and Cohen's ω were computed thanks to Cohen (1988) formulas. A post-hoc analysis was conducted with G*Power to compute the achieved power for each block as well as the critical Chi-square values, to check that the number of observations included in each block's analysis was sufficient to detect an effect.

References for SI

SI1. Audacity Team, *Audacity(R): (Version 3.0.0)* [Computer application] (2021).

SI2. D. Bates, M. Mächler, B. Bolker, S. Walker, Fitting linear mixed-effects models using lme4.

*Proc. Natl. Acad. Sci. U.S.A.* **67**, 1–48 (2015).