# Pushed by Sound: Effects of Sound and Movement Direction on Body Perception, Experience Quality, and Exercise Support

Aneesha Singh, Marusa Hrobat, Suxin Gui, Nadia Bianchi-Berthouze, Judith

Ley-Flores, Frédéric Bevilacqua, Joaquin R Diaz Duran, Elena Márquez

Segura, Ana Tajadura-Jiménez

► To cite this version:

Aneesha Singh, Marusa Hrobat, Suxin Gui, Nadia Bianchi-Berthouze, Judith Ley-Flores, et al. "Pushed by Sound: Effects of Sound and Movement Direction on Body Perception, Experience Quality, and Exercise Support." ACM Transactions on Computer-Human Interaction, 2024, 31 (4), pp.1-36. 10.1145/3648616. hal-04929076

HAL Id: hal-04929076

https://hal.science/hal-04929076v1

Submitted on 4 Feb 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Pushed by Sound: Effects of Sound and Movement Direction on Body Perception, Experience Quality, and Exercise Support

ANEESHA SINGH, University College London, London, UK

MARUSA HROBAT, University College London, London, UK

SUXIN GUI, University College London, London, UK

NADIA BIANCHI-BERTHOUZE, University College London, London, UK

JUDITH LEY-FLORES, Universidad Carlos III de Madrid, Getafe, Spain

FREDERIC BEVILACQUA, STMS IRCAM-CNRS-Sorbonne Université, Paris, France

JOAQUIN R. DIAZ DURAN, Universidad Carlos III de Madrid, Getafe, Spain

ELENA MÁRQUEZ SEGURA, Universidad Carlos III de Madrid, Getafe, Spain

ANA TAJADURA-JIMÉNEZ, Universidad Carlos III de Madrid, Getafe, Spain

Wearables integrating movement sonification can support body-perception changes and related physical activity; yet, we lack design principles for such sonifications. Through two mixed-methods studies, we investigate sound pitch and movement direction interaction effects on self-perception during squat exercises. We measured effects on body perception, affective quality of the experience, and actual and perceived movement, and compared them with two control conditions: no-sound and vibrotactile feedback. Results show that regardless of movement direction, ascending pitch enhances several body feelings and overall experience quality, while descending pitch increases movement acceleration. These effects were moderated by exercise physical demand. Sound and vibrotactile feedback enhanced flexibility and strength feelings, respectively, and contributed to exercise completion in different ways. Sound was perceived as an internal-to-body force while vibrotactile feedback was perceived as an external-to-body force. Feedback effects were stronger in people with lower fitness levels. We discuss results in terms of malleability of body perceptions and highlight opportunities to support demanding physical activity through wearable devices.

CCS Concepts: • **Human-centered computing** → **Human computer interaction (HCI)**; **Interaction design process and methods**; **Systems and tools for interaction design**; • **Applied computing** → **Sound and music computing**; **Psychology**;

Additional Key Words and Phrases: Embodied interaction, sports/exercise, wearable computers

## ACM Reference Format:

Aanesha Singh, Marusa Hrobat, Suxin Gui, Nadia Bianchi-Berthouze, Judith Ley-Flores, Frederic Bevilacqua, Joaquin R. Diaz Duran, Elena Márquez Segura, and Ana Tajadura-Jiménez. 2024. Pushed by Sound: Effects of Sound and Movement Direction on Body Perception, Experience Quality, and Exercise Support. *ACM Trans. Comput.-Hum. Interact.* 31, 4, Article 53 (September 2024), 36 pages. https://doi.org/10.1145/3648616

Authors' addresses: A. Singh, M. Hrobat, S. Gui, and N. Bianchi-Berthouze, University College London, Gower Street, London, WC1E 6BT, UK; e-mails: aneesha.singh@ucl.ac.uk, marusa.hrobat.19@alumni.ucl.ac.uk, holaguusuu@gmail.com, Nadia.berthouze@ucl.ac.uk; J. Ley-Flores, J. R. Diaz Duran, E. Márquez Segura, and A. Tajadura-Jiménez, Universidad Carlos III de Madrid, 28903 Getafe, Madrid, Spain; e-mail: jley@inf.uc3m.es, jodiazd@pa.uc3m.es, emarquez@inf.uc3m.es, atajadur@inf.uc3m.es; F. Bevilacqua, STMS IRCAM-CNRS-Sorbonne Université, place Igor-Stravinsky, 75004 Paris, France; e-mail: frederic.bevilacqua@ircam.fr.

<figure><img src="image_2.png" /></figure>
# 1 INTRODUCTION

Low **physical activity (PA)** is a global public health problem contributing to almost 3.2 million deaths each year [72]. To combat this, various activity tracking devices and applications (e.g., Fitbit, GoogleFit) have been developed, primarily based on personal informatics principles [76, 108] and implementations of behaviour change techniques [40]. However, insights obtained by self-tracking can be insufficient to facilitate physical activity [108]; research suggests that factors associated with body perception should also be taken into consideration [4, 54, 57]. In particular, recent work has shown how sound feedback associated with body movement (sonification) can affect people's body perceptions (e.g., altering footstep sounds [91]) and, in turn, their emotional state and movement behaviour [57, 95]. However, we lack design principles for such sonification in relation to body transformation experiences (i.e., changes in body perception) and physical activity.

Recent work indicates clear opportunities for sonification-based technologies using artificial sounds changing in pitch for altering body perceptions and facilitating movement. A first study using such sounds showed that an ascending pitch sound triggered by the action of pulling one's finger can make one's finger feel longer [67, 97]. Such an "auditory Pinocchio" illusion could result from associating sound pitch changes with changes in height, size, and motion along the vertical plane [22, 58]. Building on this, in a recent study we investigated the effect of pairing ascending and descending pitch sounds with an upward body movement (arm raise) on the perceived body position (i.e., proprioceptive awareness) and movement [55]. The investigation comprised three quantitative experiments with 25 participants in the first two studies and 20 in the final one. Participants synchronised an arm raise with sounds changing pitch in simple tones (Experiment 1), rich musical sounds (Experiment 2), and within different frequency ranges (Experiment 3), while their movement, body representations and feelings were recorded. Results showed that ascending pitch sound induces a sense of lightness and ease, accelerating movement and motivating exercise completion [55]. Conversely, descending pitch creates a sense of heaviness and slowness. But, it remains to be seen whether these effects extend to exercises requiring larger effort [55]. High-intensity exercises may lead individuals to focus more on overwhelming physiological sensations rather than on the sound [41]. In this article, we address three gaps by exploring, quantitatively and qualitatively, the effect of sound on a strength exercise on inactive and quasi-inactive people. These gaps are summarized here:

Firstly, few studies have focused on strength exercises. These studies were either not focused on altering body perception and capabilities [69] or explored natural sound alterations, such as footstep sounds, rather than augmented movement sonification [91, 95]. A more related study examining the effects of various metaphorical sounds [57] demonstrated positive effects of sounds changing in pitch in a small sample of inactive participants performing strength exercises (squats and step ups). However, the study had a small sample size (5 inactive people vs. 7 highly active people) and employed only qualitative methods, lacking measures of behavioural effects. While such small qualitative studies provide some initial understanding of how sound impacts body perception in strength exercises, quantitative approaches are necessary to understand the effects of specific sonification qualities.

Secondly, studies on sound and body size perception (e.g., finger length) [97] overlook interactions between sonification direction and movement direction. Further, interaction between the movement direction (i.e., up and down) and the perceived sound pitch (i.e., ascending and descending) is not studied in the physical exercise context. This gap is particularly relevant in strength exercises, in which downward and upward movements may pose different challenges. While upward movement in exercises (e.g., arm raise) has been studied (e.g., [55]), no single study has explored the effects of the descending pitch sound on downward movement. Whether the potential effects
<figure><img src="image_3.png" /><figcaption>Fig. 1. Overview of the experimental setups, including the musical sounds (can be heard by scanning the QR codes), the movement sonification app, the wearable vibration device, and other materials used in this study.</figcaption></figure>

mirror those of upward movement is not obvious. While congruence between movement direction and sound direction may facilitate a demanding exercise, it is uncertain how sound direction may produce different changes in body perceptions and how such perceptions influence movement execution.

Finally, the third gap is methodological: studies focusing on the effect of sound on perceived body size and movement capabilities contrast sound feedback with a control sound condition or no feedback (no-sound condition). We propose adding another sensory modality as a control condition, which may help to better understand the effect of sound through a contrasting experience of feedback. While one quantitative study [11] explored the combination of sound with another modality (smell), it focused on the effect of smell on body perception, not movement representation. While both haptic and visual feedback have been studied for their facilitating effects in physical activity and movement trajectory (see related works in other contexts [12, 30, 81]), to our knowledge, no studies have yet investigated the effects of sounds changing in pitch on body perception alongside those sensory feedbacks in our application context. Indeed, existing studies on physical activity primarily use feedback from multiple modalities mainly to compare their effects on adhering to movement trajectory and target only, not to alter people's perception of their own body and movement.

To address the identified gaps, we conducted two studies. First, a mixed-methods remote study (n = 22) focused on participants performing squat exercises (Experiment 1), in which ascent and descent were sonified by ascending or descending musical sounds (Figure 1). Squats are a physically demanding exercise involving two directional phases of movement requiring significant effort, upwards and downwards, unlike the arm raise in [55], in which only the upward movement is required great effort. We analysed self-report and sensor data to understand how sound direction affects perception of movement and movement characteristics. The second study (Experiment 2) aimed for a deeper understanding of the effect of sound on movement perception and facilitation and perceptions across the whole body through another mixed-methods study (n = 21). Squat exercises were accompanied by movement sonification, with two control conditions: (i) no sensory feedback and (ii) haptic feedback. The two conditions aimed to trigger comparative reflections in participants on the different experiences. Haptic feedback, like sound, doesn't interfere with movement, as people do not need to visually fixate on it. It has been shown to reduce perceived workload and improve performance [13, 38]. Further, haptics is a primary sense developed in infancy [64], increasingly used in user interfaces to convey rich information [9, 10, 29]. It is widely shown in neuroscience research to contribute to people's perception of their own body and action [5, 105] and is used in the context of physical activity for movement guidance [84] (see [82] for an overview). The use of haptics is also highlighted in the human-computer interaction (HCI) domain of soma design (e.g., work on the *Soma Corset* [47] and the *Breathing Wings* [106]). Our focus in
Experiment 2 was on participants with low or moderate levels of physical activity because they may experience larger effects of the feedback on their body and movement [56]. This article makes three contributions that are discussed next.

First, we show that directional movement sonification can facilitate a vertical strength exercise by influencing body perception and overall experience quality (in this article, this term encompasses affective experience and other aspects related to people's bodies and movement as an outcome of the sound effects). However, the psychological mechanisms triggered depend on sound direction. The upward sonification triggers feelings of being lighter, less tired and more capable, whereas the downward sonification triggers feelings of being heavier and of greater movement acceleration, independent of the movement direction. This extends previous literature [55] by suggesting that movement sonification with sounds changing in pitch can facilitate strength exercises, not just simple movements. Moreover, departing from the constancy effect observed in [97], this study shows that the congruence between the movement direction and sound amplifies related effects in strength exercises. For example, feeling heavier in an upward movement could be perceived as negative but the same feeling may be enjoyed and facilitate a downward movement. Thus, ascending and descending sounds can generate different body changes and changes in experience quality to support movement in different ways.

Second, our study shows that both sound and vibration feedback are linked to rewards during movement and to the sensation of force being exerted by the feedback guiding and helping to perform the squats; it is noteworthy that sound feedback is regarded as an internalized force while vibration feedback is regarded as an externalized force.

Finally, our study highlights the potential variation of effects according to the person's fitness level, with stronger effects in people with lower fitness. This presents new opportunities in supporting people who struggle the most with physical activity and contributes to the literature on factors that we identified as interacting with how body perception is built, adding physical activity level to the already identified personal values [95]. We discuss design opportunities to enhance body perception through wearable devices for supporting physical activity in the context of a strength workout that are made possible in light of our findings.

# 2 RELATED WORK

## 2.1 Auditory-Induced Changes in Body Representation

We interact with, think about, and perceive the world around us through our bodily senses [5, 32, 91]. The multisensory feedback generated by our actions is continuously used by our brains to adapt our mental body representations [14, 59]. Altering body-produced auditory feedback has been shown to be a particularly powerful way of changing how we perceive our bodies and, in turn, how we act in the world (for a review, see [87]). For instance, artificially lengthening the time taken to hear an object fall on the ground after being dropped from one's hand changes the internal estimates of body height; thus, people report feeling taller and behave as if their legs were longer [91]. Moreover, manipulating the auditory distance of tapping sounds when tapping one's hand on a surface can alter the mental representation of one's arm length, whereby increasing the auditory distance increases the perceived length [94, 96, 99]. This mental representation of a longer arm, in turn, affects the arm reaching movement in a way that is consistent with having a longer arm (i.e., lower reaching velocity) [14]. Additionally, manipulating cues (e.g., pitch, loudness) of sounds related to the level of applied strength when tapping a surface can result in changes in perceived tapping strength, one's own ability to tap and emotional state. It was found that participants felt more able to tap, more pleasant, and were less physiologically aroused as shown by their galvanic
skin response when the sound indicated high versus low levels of strength applied [87, 89]. Criti-
cally, evidence shows that feelings of agency and spatio-temporal congruency between action and
auditory manipulation are essential for these effects to emerge [63]. There is also related work
from the field of soma design on targeting sensory misalignment for explicitly disrupting sen-
sory perception to evoke estrangement or for disrupting the habitual as a path to design [101]).
Apart from these altered natural sounds, artificial sounds played in synchrony with a bodily ac-
tion, but which do not provide explicit feedback on the performed action, have been shown to
induce changes in body perception. For example, accompanying body movements with the sound
of a “creaky” door can make people feel stiffer [86], whereas accompanying joint movements with
prerecorded sounds and vibrations produced by a robotic arm can make people feel “robotised”
[52]. Another related work describes a wearable, *Squeaky/Pain*, that employed sounds of squeaky
wood accompanying bodily movement to create disturbing experiences that augment the wearer’s
somaesthetic awareness of one’s body as well as who was in control of it [23]. Such literature has
led researchers to explore the opportunities that the sonification of a movement offers in terms
of addressing psychological barriers to physical activity, such as perceived body weight, size, and
physical capabilities while engaged in physical activity. Our work aims to contribute to this body
of work by addressing two specific gaps highlighted in the next two subsections.

## 2.2 Changing Body Perception Through Sound to Facilitate Physical Activity

Existing technologies aim to facilitate and support physical activity through encouraging self-
tracking and goal-setting behaviour, providing quantitative feedback on physical activity (e.g.,
amount of activity, goals achieved) [16, 50]. However, only providing feedback on physical activ-
ity performance (e.g., Fitbit app) is often insufficient to ensure adherence and facilitate physical
activity, especially for those who struggle to engage [108]. An alternative approach is to tackle
the underlying psychological barriers or needs that prevent people from engaging in physical ac-
tivity (e.g., low self-esteem and motivation, negative body feelings, low self-efficacy, and related
affective states and moods due to such feelings) [75]. Instead of just monitoring the amount of
activity, a new line of research has been rethinking the design of physical activity technologies by
embedding these psychological factors into the design process [57, 68, 83].

Extensive research has focused on the use of music to promote physical activity adherence and
increase exercise participation, leading to behavioural changes [42] due to its ability to induce
positive emotional states [46, 102]. More generally, a positive influence of music on physical ac-
tivity has been largely reported. Different mechanisms have been pointed out, such as the role
of rhythmic entrainment [44]. For example, the influence of tempo has been studied on rowing
performance [74] and on treadmill exercise [25]. Furthermore, rhythm and musicality have been
investigated for their potential in helping to enhance physical fitness, including cardiorespiratory
endurance and muscular fitness, albeit with conflicting findings (for a review, see [36]). Music
has also been shown to influence perceived exertion [73]. In particular, the possibility to control
sound (called *musical agency*) has been demonstrated to play a key role in reducing perceived
exertion during strenuous physical performance [31] or beneficial for motivation, learning, and
performance in running [109].

Beyond addressing emotional states through music, several studies have explored the pre-
viously identified potential of auditory feedback to change mental body representations in the
context of facilitating physical activity in physically demanding and undemanding exercises for
active and inactive adults. Some earlier research has shown that altering the body-produced
auditory feedback can be an effective way of changing body perception and, in turn, related
affective state (feeling good about oneself and one's capability) and actual movement. Some of our
previous work [91, 95] shows that listening to one's own footstep sounds, which were modified in
real time in terms of their frequency spectrum, may change how people perceive their body size and weight. This, in turn, affects the way they walk and their emotional state in terms of valence and arousal. For instance, listening to high-frequency footstep sounds while walking has been shown to make people feel lighter and quicker, happier and aroused, and adapt their behaviour so that it is consistent with having a thinner body (i.e., a more dynamic swing and a shorter heel strike) as compared with the feelings and behaviours elicited by low-frequency footsteps (i.e., feeling heavier and slower) [91]. These positive effects of the high-frequency footstep have also been identified in more physically demanding exercises (i.e., gym steps, stair climbing), showing the potential of this approach to motivate and facilitate physical activity [15]. A similar setup has been tested and shown to be promising for physical rehabilitation of people with chronic pain [92] and stroke [34], and for studying anomalous body experiences in people with symptomatology of eating disorders [90]. However, rather than sonification of a movement, the approach used in these studies was to alter the characteristics of a natural action sound (the footstep sound) to look like the one of a heavy or light person. No artificial sonification of movement was explored.

In addition to altering the body-produced sounds, metaphorical movement sonification has potential to enhance body perception and facilitate physical activity. Artificial movement sonification, in which body movement is tracked and mapped into real-time auditory feedback, has often been used in the context of sports and physical rehabilitation to provide information on movement (e.g., on hand-water interaction while swimming [15]) and help movement execution and control (for recent reviews, see [3, 79]), and movement awareness in clinical conditions (e.g., physical activity for people with chronic pain [83] to overcome anxiety). Such studies, however, have not investigated effects on body size and physical capabilities perception. Subsequently, both quantitative [55] and qualitative studies [56] showed that movement sonification with natural metaphors (e.g., wind, water) enhances various aspects of body perception, emotional state, and movement in a variety of exercises, especially for physically inactive individuals. Recently, a quantitative study investigated the effects of ascending sounds when accompanying a simple arm raise movement [55]: the ascending sound eased the movement and enhanced several body feelings (i.e., lightness, speed, (less) tiredness, capability and motivation to perform the movement). This was a first attempt to investigate the effect of pitch sounds on proprioceptive awareness and bodily movement, which only explored the effect of dynamic pitch in a basic upward movement with relatively little bodily displacement.

However, most of these studies have focused on light exercises (e.g., arm raise) and only two studies have explored strength exercises. The quantitative study reported in [69] showed that movement sonification using a series of musical chords (rather than a continuous sound changing in pitch, such as the one used in this study) can motivate physically inactive people during a downward squat exercise being seen as rewarding the completed movement [69]. The focus of the study was on understanding the effect of expectation-related musical structures (e.g., stable vs. unstable cadence) embedded at the end of a movement sonification on the depth of a squat and on the repetition (time to return) in a sequence of squats. Effects on perception of one's body size and capability were not investigated. More recently, we used a qualitative-only approach in [57] in which we provide evidence of positive effects of ascending pitch sounds on inactive participants performing strength exercises (squats and step-ups). However, the study included only 7 inactive people and 7 very highly active people and used only qualitative methods. No quantitative measures of the effects on behaviour were gathered. While this small qualitative study provides some initial understanding of how sonification may impact body perception in strength exercises, an in-depth quantitative analysis of this effect is needed to inform the development of a sonification framework. In addition, rather than only looking at people at the two extreme levels of fitness
(very inactive vs. highly active), it is important to also consider low-but-active people as they represent a critical and large part of the population that need to be supported. In this study, we aim to address these gaps in strength exercises through a quantitative method and across inactive and low-active people.

## 2.3 Interaction between Movement Sonification and Sound Directions

Sound changing pitch leads to the perception that the sound is moving in a certain direction. Recent studies on body perception outside the context of exercise have looked at the effect of sounds changing in pitch on body perception. Drawing on the proposed metaphorical cross-modal correspondence between sound pitch and vertical movement (i.e., ascending pitch and upward movement, descending pitch and downward movement) [27], previous investigations focus on the effect of dynamic pitch on the mental representation of one's finger length when paired with a finger-pulling action [97]. In a controlled experiment, adult participants were asked to press and pull their finger up or down while presented with brief pure-tone sounds of rising, falling, or constant pitches. The study found that the pairing of the finger-pulling action with an ascending pitch sonification results in people perceiving their finger as significantly longer than with a descending or constant pitch regardless of the vertical pulling direction (i.e., pulling up or pulling down). This suggests that pitch direction rather than the cross-modal correspondence between pitch direction and vertical movement trajectory alters the perceived finger length. The authors of [67] investigated the same effect of ascending or descending sound while having their finger pressed or pulled but this time along the horizontal axis (i.e., pointing left/right) and found no effects; indeed, the cross-modal correspondence between dynamic pitch and horizontal movement has been shown to be less of an automatic mapping [65, 77]. Together, these findings suggest that the spatial metaphor of dynamic pitch is sufficient to alter mental body representations, yet only when paired with vertical movement.

Despite these findings, no study has addressed the interaction between sound direction and movement direction in the context of physical activity. Two differences exist with the finger-pulling situation. In finger pulling, the same physical force is in place, the other hand pulling the finger away from the attached hand. In the context of physical exercise instead, the force exerted by the body in the two vertical opposite directions requires very different muscle and movement activity that may interact differently with the direction of the sound. Hence, it remains to be investigated whether the positive effect of ascending pitch on body perception holds across both upward and downward movement directions when the extent of bodily displacement is increased (e.g., movement of the entire body as in squats vs. part of the body as in finger pulling or arm raises) and for a more demanding strength exercise. Our current study will precisely investigate this: in two experiments, we employ similar sounds, ascending and descending in pitch, to investigate their effects on body perception and movement behaviour when accompanying body movement during a physically demanding strength exercise composed of two directional phases, upward and downward, which both require great effort (i.e., squats).

## 3 EXPERIMENT 1: METHODS

Experiment 1 aimed to understand how the direction of the dynamic pitch sonification (i.e., ascending or descending pitch) with respect to the direction of the body movement (upward or downward) interact with the effects on body feelings, experience quality, and actual movement [55, 97] in the context of a strength exercise. The experiment was conducted remotely (i.e., in participants' homes) due to the COVID-19 restrictions at the time.

Experiment 1 was a mixed-methods experiment, which was approved by the University College of London (UCL) Research Ethics Committee (reference number 5095/001).
**3.1 Participants**

Participants were recruited through the University subject pool and social media. Experiment 1 was conducted entirely remotely. Inclusion criteria: 18 years or older; no known chronic, mobility, or hearing conditions; able to perform at least 5 consecutive unweighted squats; not pregnant. In Experiment 1, additional requirements were an Android phone (OS 6 or above) and ear/headphones for using the app. The study was remote via MSTeams; thus, participants needed a computer and webcam. A total of 22 adults participated (age: mean = 25.05 years, SD = 3.08, range = 22–34; 12 men, 10 women).

**3.2 Exercise Selection**

The squat exercise was chosen because it involves full-body displacement and both downward and upward movement, enabling the experimental manipulation of movement direction. It is one of the most effective and frequently used exercises to build muscle strength, enhance athletic performance, and minimise injury potential [28, 80]. It is linked to many everyday tasks (e.g., lifting objects) and is commonly used in rehabilitation after joint- or knee-related injury. The unweighted variation (i.e., body weight squat) was chosen to ensure participants’ safety regardless of physical activity level or access to equipment. While the focus was only on one exercise across the two studies, squats are interesting because of the strength challenges they pose in both vertical movement directions. They also complement previous work focused on arm movements. In addition, squats involve bending of body segments (legs in this case) rather than just a vertical displacement of a rigid arm, bringing complexity of the secondary horizontal displacements of individual body parts beyond the main vertical one. They also involve the use of a variety of muscles across the middle and lower body (lower back, glutes, various upper and lower leg muscles). As such, the learning may provide initial insights into the effect that sonification may have on a variety of exercises that present one or both directional demands in addition to the involvement of a variety of body parts that may be engaged in different secondary minor displacement (e.g., jump-squat, cube step-up, planks, pull-up bar exercises, arm raise with weights), and involve upper or lower body parts as well as bending of limbs. We call main displacement the largest one and the one perceived as characterizing the main direction of movement.

**3.3 Experiment Design**

The experimental design was a 2x2 within-subject remote experiment, with two independent variables and two levels each: the first independent variable was movement direction with levels upwards and downwards. The second independent variable was pitch direction of a simple musical sound with ascending and descending levels. We focused on the changes effected by the sound in terms of direction of movement, not on correct movement performance. The dependent variables included behavioural measures and quantitative and qualitative self-report data, and were selected in accordance with previous studies in this area [55, 57].

**Body feelings:** Self-reported changes in body feelings were quantified using the body feelings questionnaires used by related studies (e.g., [55, 56, 91, 95]). Our version comprised 12 items (7-point Likert-type response) about body perception (*Strength, Tiredness, Comfort, Capability*), movement perception (*Difficulty, Speed, Control, and Coordination*), feelings of oneself producing the sounds heard (*Agency*), and about motivation to do the exercise. Finally, there was an open-ended question to describe the perceived changes. The selected body-perception dimensions represent critical perceived barriers to engagement with exercises, including strength exercises such as squats. The movement perception dimension relates to important factors in exercise executions. Agency (feeling that one is driving the sound) is critical for sonification to have an effect [63] on body-perception alteration.
<figure><img src="image_4.png" /><figcaption>Fig. 2. Smartphone on thigh, calibrated between maximum standing and minimum comfortable squatting position; they correspond respectively to 0% and 100% of the total movement range (left). Movement angle thresholds to trigger the sound for upward and downward squats and for vibrotactile and sound conditions (middle). Audio energy profiles and spectrograms of the sound stimuli: Ascending and Descending sounds (right). Note that the mapping between percentage of movement and movement degrees varied for each participant since the range of movement varied for each participant and was determined during the calibration. For a participant able to squat 100 degrees, the correspondence between percentage of movement and degrees was one-to-one.</figcaption></figure>

**Affective experience:** The valence/pleasure and arousal subscales (9-point Likert-type response items) of the **self-assessment manikin (SAM)** [6] were used to assess emotional or affective responses as measured in related studies [55, 97]. Emotional responses here refer to a short-time process in response to an eliciting event, in this case the experimental condition, and which can be measured by looking at affective reports, physiological reactivity, and overt behavioural acts [6]. Here, we measure only one of the systems as we focus on affective reports. We note that changes in emotional or affective experience contribute to participants’ overall experience quality. Such changes may be due to changes in perception of oneself or in the overall exercise experience. Such distinctions, and other aspects contributing to the experience quality, are explored through the follow-up interviews.

**Movement behaviour:** This was measured using phone sensors (*accelerometer, gyroscope*). The data captured by the rotation vector (Y-axis) was normalized, visualized, and cleaned. Next, 8 movement variables were extracted for each squat repetition using MATLAB [55]. These were the peak and mean movement angle between start and end position; and time, velocity, and acceleration from minimum to end position (i.e., for the upwards movement) and from end to start position (i.e., for the downwards movement).

## 3.4 Materials

3.4.1 *Experimental Application.* A smartphone application was developed for the study based on [83]. The application used built-in phone movement sensors to track, measure, and sonify participants’ movements. The participants’ phone was attached to their thigh with an armband and a strap (Figure 2). The app was calibrated to participants’ maximum (i.e., standing) and their lowest comfortable squatting position (Figure 2, left and middle) using the device’s sensors. The maximum and minimum calibration points were considered as 0% and 100% of the total movement range, respectively. A change in movement angle triggered a continuous synchronous sound. Connected headphones were used to ensure good sound quality. We chose a sound stimulus that was already used in a previous study [55] on lift arm exercises. The sound appears as two percussive
notes (forming a fifth interval such as C-G, considered as consonant and stable in musical theory, and being predominant harmonics of a musical sound), with a short attack (peak at 0.2 s), decay (0.3 s), and sustained part of 0.5 s, and a release (intensity diminishing sharply during 0.3 s) with a reverberation effect. This sound envelope is shown in Figure 2 (right), where the audio energy is reported. The total sound duration was fixed to 1.3 s. As shown in Figure 2 (right, bottom), where the audio spectra are displayed, the sounds had a rich spectral content as they were initially created using marimba sampled (which give the percussive attack) and modified to ensure a longer sustain. For the experiment, the pitch direction of the musical sound was manipulated (using a phase vocoder) during the first 0.5 s for the sound to create a continuous ascending pitch change of one octave (frequencies being multiplied by two) or a continuous descending pitch change of one octave (frequencies being divided by two) or a continuous descending pitch change of one octave (frequencies being divided by two) based on [55] (Figure 2). The pitch variation is clearly visible in Figure 2 (right, bottom). Loudness was normalized across both sounds. To account for the difference in the recommended movement time between the squat ascent and descent [66], when squatting upward the sound was triggered as participants reached 10% of the movement angle and 20% when going down. The raw sensor values were stored on participants' smartphones and later shared with researchers.

## 3.5 Experiment Procedure

Before the experiment, participants completed a prescreening survey to check eligibility and installed the app. They gave informed consent for the study. A smartphone armband and strap were sent by post. At the start of the study, a researcher on MSTeams explained the experimental procedure to the participants and the participants prepared the materials (i.e., app, headphones/earphones, smartphone band). Next, the participants filled out the demographics questionnaire and watched instructional videos for attaching their phone to their thigh and performing the exercise. Each session was video recorded with consent. First, was the **practice phase**, in which the participants strapped their phone onto their thigh and calibrated the device, setting the maximum (i.e., starting) and minimum squatting position. They performed 3 squats to check whether the application worked correctly. Second, during the **experimental phase**, participants performed 4 sets of 5 squats with a short break (1-2 min) between each set. This phase had **full-squat sonification**, in which both downward and upward squat movement directions were accompanied by sound. Each set of squats consisted of either congruent movement-sound pairing (i.e., downward movement with descending pitch, upward movement with ascending pitch) or incongruent pairing (i.e., downward movement with ascending pitch, upward movement with descending pitch). Each participant performed four sets of squats, one for each possible pairing. To control for potential order effects, the order was fully randomized. The participants took a break after each set (reminded by the app by voice message). Movement data was collected throughout this phase. Next, participants performed 4 squats in the **self-report phase**, stopping after each squat to fill out the body feelings and SAM questionnaires. This phase had **half-squat sonification**, in which only one direction of the squatting movement (either up or down) was sonified, to assess the effect of each pitch direction on each movement direction. There were 4 fully randomised conditions (ascending sound-up movement, ascending sound-down movement, descending sound-up movement, descending sound-down movement). Movement data was not collected in this phase. This was because we had already collected the movement data in the previous phase. The focus in this phase was not on the movement; rather, it was on how participants perceived the sound and its effect on their movement. Thus, the focus was removed from the movement itself. Finally, at the end of the experiment, qualitative changes in bodily feelings and emotional
state were assessed using a short semi-structured interview (around 10 minutes). In the interviews, participants were asked to elaborate upon their experiences during the entire experiment related to their body feelings, movement perceptions, and experience qualities as well as the reasons behind these experiences. During the interview, participants were asked to think about all four conditions they experienced during the self-report phase. They were then asked to describe their experiences during the four different conditions. We asked them whether their experiences differed between conditions and in what way. We also asked participants how they would compare the experience of going up with the ascending pitch versus going up with descending pitch. We also asked them about the converse, i.e., how they would compare the experience of going down with the ascending pitch versus going down with the descending pitch. We then queried which sound (regardless of movement direction) they preferred the most and why. We further probed how the sound made them feel. Next, we asked how they felt about the movement of squatting down versus standing up from the squat and why they felt that way. We then asked participants about which sound-movement combination they preferred the most and why and how it made them feel (light, heavy, fast, slow, strong, tired, etc.). Finally, we asked participants whether any of the sounds or specific sound-movement combinations helped with their movement and in what way. Before we ended the interview, participants had the chance to add anything we may have missed. The interview was audio recorded. Participants emailed the movement files to the researchers.

## 3.6 Data Analysis

We collected quantitative data, including self-report and movement sensor data, as well as qualitative interview data. In this section, we describe the analysis approach for both.

Quantitative data analysis was done using the “ARTool” package in R [108] using 2x2 repeated measures analyses of variance (ANOVAs) on aligned rank transformed (ART) data with ‘movement direction’ and ‘pitch direction’ as within-subject factors. Separate ART ANOVAs were conducted for all self-report items. Movement data was obtained from sensors automatically and checked for outliers. Normality of model residuals was tested using the Shapiro-Wilk test. No outliers were detected but the model residuals for all dependent variables showed large normality violations. After testing various transformations unsuccessfully, we carried out non-parametric analysis for all movement variables: velocity, acceleration, and time measurements were submitted to separate 2x2 repeated measures ART ANOVAs with ‘movement direction’ and ‘pitch direction’ as within-subject factors. Since angle data (i.e., peak and mean movement angle) was measured for each squat rather than each movement direction, it was analysed using separate one-way repeated measures ART ANOVAs with ‘condition’ (congruent or incongruent) as the only within-subject factor. The significance level for all statistical tests was fixed at an alpha level of 0.05 and effect sizes $η_p^2$ were calculated [18].

Qualitative data analysis of each participant interview was done using thematic analysis [7] as follows. We transcribed interviews verbatim. Participants’ responses to open-ended questions from the self-report questionnaires were combined with their associated interview transcripts for analysis. Each transcript was then analysed following the Braun and Clarke approach to top-down thematic analysis [7]. The aim of the follow-up interview was to better understand participants’ experiences related to each of the sounds and sound-movement pairings throughout the experiment. Hence, the transcripts were analysed deductively, with codes based on the body feelings, movement perceptions, and experience qualities perceived during the different experimental and self-report conditions. Subsequently, the codes were grouped into larger themes and subthemes in relation to the research question.
<figure><img src="image_5.png" /><figcaption>Fig. 3. Boxplots for body feelings and experience qualities significantly influenced by pitch direction or movement direction (* indicates *p* < .05, ** indicates *p* < .01).</figcaption></figure>

# 4 EXPERIMENT 1: RESULTS

Three main themes emerged from the quantitative and qualitative data on the effects of movement sonification on (i) body feelings, (ii) movement behaviour, and (iii) experience qualities. Quantitative and qualitative results are reported together for completeness.

## 4.1 Effects of Movement Sonification on Body Feelings

Analysis of self-reported data shows a significant effect of movement direction on perceived movement ease. The effect of pitch direction on various body feelings was significant: when movement was accompanied by ascending sound, participants felt significantly lighter ($F(1, 21) = 13.21, p = 0.002, \eta p^2 = 0.39$), less tired ($F(1, 21) = 7.64, p = 0.012, \eta p^2 = 0.27$), more capable ($F(1, 21) = 6.30, p = 0.02, \eta p^2 = 0.23$), more motivated ($F(1, 21) = 13.63, p = 0.001, \eta p^2 = 0.39$), and perceived the movement to be easier ($F(1, 21) = 10.15, p = 0.004, \eta p^2 = 0.33$) (Figure 3). They also perceived the downward squat to be more difficult than upward ($F(1, 21) = 4.62, p = 0.043, \eta p^2 = 0.18$). There were no other significant effects or interactions.

Follow-up interviews showed that most participants discriminated between ascending/descending sounds, associating ascending pitch as positive and descending as negative regardless of the movement direction it sonified, based on their previous experiences of similar sounds. P3 said, "[...] in a video game a low-pitched sound is like you lost the game. That is what it sounded like when I heard the descending sound. A high-pitched sound means I successfully passed the level." Qualitative results for ascending pitch were in line with the quantitative self-report results mentioned above. Descending pitch made participants commonly feel heavier and less capable regardless of movement direction: P22 said, "descending sound was diffusing ... it would help me
sleep rather than exercise.” Ascending sounds were commonly associated with correct movement and descending sound with incorrect movement; the validation provided by the ascending sound acted as a motivation to continue the exercise. P2 explained, “Getting positive reinforcement makes you feel good. So you get that approval from a noise saying you did the movement correctly. This empowers you to do more”. Descending pitch had the opposite effect; its association with failure made participants reject the sound, like P20: “As the pitch was decreasing, it felt like I failed or did something wrong in the squat and I felt unmotivated to continue [...] that sound did not feel like it came from me, I mean no one wants to fail, right?” Ascending pitch with ascending movement (congruent sound–movement pairing), facilitated movement and amplified feelings of lightness associated with the sound. The ascending sonification reduced perceived difficulty of the upward movement and the ascending-up sound–movement combination enhanced capability, increasing perceived speed and perceived quality of movement. P19 said, “With the ascending sound, it felt like I went up really fast and that speed just carried me. It felt like I went up more than during other squats. Moreover, the ascending–up pairing was perceived as the most synergetic and in-sync with one’s movement, resulting in a higher perceived sense of agency over the sound. With this pairing, participants described feeling like a decompressing spring as they squatted up; this feeling intensified through the ascent and was the most salient at the top of the squat, facilitating not just the ascent but movement completion and progression onto the next squat. P20 said, “The rise in pitch made me feel like my ascent was much faster and springier. It made me feel like I could go for another round just to feel that springiness again.” About a quarter of the participants enjoyed the descending sound on the descent, perceiving their movement as controlled and less demanding. Feelings of heaviness associated with the descending sound enhanced the perceived speed, quality, and amount of the downward movement, as P7 said: “When the descending sound came as I was squatting down, it felt like I was pulled down by the sound and a bit heavier, faster and stronger, sinking deeper into the squat.” However, some other participants perceived the descending sound as negative and demotivating even on descent; one described it as a “battery losing its power” (P1). Moreover, the lack of upward sonification following the descending-down pairing in the self-report phase (only half of the squatting movement was sonified) impeded the subsequent upward movement, increasing the difficulty of the squat and reducing motivation to continue with the exercise. However, sonifying both squat directions increased the sense of agency and connection between the sounds and one’s movement. P11 said, “The descending sound when I am going down makes me feel like I am going to get stuck because I have no energy to go back up. But when followed by an ascending sound it worked because it balanced itself out. So, I am being pulled down and then pushed up.” Others felt accompanied by full-movement sonification: P11 felt that “When the sound went along with the movement, it felt like I was being pulled down and pulled up. So, it felt like I was not doing the exercise on my own.” We note here that some participants used the terminology of being pulled but this was not primed by the researchers.

## 4.2 Effects of Movement Sonification on Movement Behaviour

Results show a significant effect of movement direction: participants took significantly less time to complete the downward movement compared with the upward one regardless of pitch direction ($F(1, 21) = 8.41, p = 0.009, \eta p^2 = 0.29$), as expected (Figure 4). Movement velocity ($F(1, 21) = 7.62, p = 0.012, \eta p^2 = 0.27$) and maximum acceleration ($F(1, 21) = 24.41, p < 0.001, \eta p^2 = 0.54$) were significantly greater when squatting down compared to going up regardless of pitch direction. Pitch direction of the sound affected movement acceleration as participants had greater maximum acceleration with the descending compared to ascending pitch regardless of movement direction ($F(1, 21) = 6.20, p = 0.021, \eta p^2 = 0.23$) (Figure 4). There were no other significant effects.
<figure><img src="image_6.png" /><figcaption>Fig. 4. Means and standard errors for movement time, velocity, and maximum acceleration, indicating several significant main effects of movement direction and a significant main effect of pitch direction on maximum acceleration (* indicates *p* < .05, ** indicates *p* < .01, *** indicates *p* < .001).</figcaption></figure>

Behavioural measurements highlighted only acceleration effects of descending sound, but qualitative results were nuanced. Participants perceived ascending pitch as facilitating the squat, reducing perceived difficulty, and enhancing perceived speed and quality of movement. In contrast, most did not enjoy the descending pitch, as they felt it impeded the movement and reduced its quality. Further, pairing the “positive” ascending sound with upward movement of the squat seemed to enhance movement performance. P2 said: “*The hardest part of a squat is coming up so having the positive [ascending] sound on the way up made me feel better than when I went down with that sound.*” Thus, P2 did not find pairing ascending sound with the down movement (incongruent) as beneficial. Most participants found incongruent full-squat sonifications (pitch direction different from movement direction, e.g., ascending-down) unexpected and weird. For example, some participants described the sound as “*lacklustre and redundant*” (P15). Moreover, the misalignment between the sound and movement reduced the sense of agency for some. P13 said: “*It was a motivating sound, but did not fit the downward movement; I felt less connected to it. It was nicer than no sound but more neutral and awkward.*” Despite this, most participants reported little or no effect of congruency between sound and movement on actual or perceived body movement. Overwhelmingly, full-squat sonification was perceived as more beneficial to facilitating physical activity compared with half-squat sonification and people reported “*missing the full sonification*” (P9).

About one-fifth of the participants reported a slight or even a complete lack of effect of the full-squat sonification on their movement perception, body feelings, and experience quality. This seemed common with participants who were less used to squats and hence potentially too focused on their movements and the squat count, thus ignoring the sonification altogether. Some participants completed the sets of 5 squats too quickly, resulting in sensory overload and lowering their ability to distinguish between different sounds and pairings. P16 reported, “*Because I was doing the squats so quickly, the two sounds blurred into one. There were a lot of sounds and squats at the same time. [...] If I slowed down, maybe it would have had a different effect.*” Some participants suggested that the number of repetitions was too small for the movement sonification to elicit any salient effects. A few found it too easy. A low sense of agency over sound also often accompanied a complete lack of elicitation of effects on movement perception. A quarter of the participants had a hard time connecting the simple musical sound to their squatting movement. They argued the sounds were unfamiliar to how their body sounds, hence distancing themselves from the sonification. P14 said: “*[...] I strongly disagree with those questions about agency. I’ve had my body for a long time, and I’ve never heard it do anything of that kind.*”

## 4.3 Effects of Movement Sonification on Experience Quality

Emotional responses to the stimuli were measured to assess changes in the affective experience of participants, which contribute to the overall experience quality. Results showed
a significant effect of pitch direction both on self-reported valence and arousal: Participants felt more positive ($F(1, 21) = 8.56, p = 0.008, \eta p^2 = 0.29$) and more excited or aroused ($F(1, 21) = 4.69, p = 0.042, \eta p^2 = 0.18$) with the ascending compared to the descending pitch regardless of movement direction (Figure 3). There were no other significant effects of movement direction or any other interaction effects. Results were supported by interview findings. Participants preferred the ascending pitch as it was perceived as more energetic, stimulating, and motivating regardless of movement direction. Moreover, it was described as more awakening, energizing, evoking feelings of lightness, and useful when trying to be more “explosive” in performing the squat movement (i.e., high-intensity, short-duration movement), despite people showing stronger acceleration in the descending sound.

The descending sound with upward movement was the least preferred sound–movement pair due to perceived pitch and movement direction misalignment, making participants feel significantly heavier, demotivated, and less capable. The perceived ease of the ascent amplified the perceived incorrectness associated with descending sound regardless of movement direction. P8 said: “*The upward squat with the descending sound was definitely harder, I felt more muscle tension and heavier.*” The full-squat sonification (both up and down directions sonified) led to a feeling of bounciness or springiness, significantly increasing the perceived motivation, capability, and movement ease. Although this feeling was evoked by just the ascending sound on the ascent, it was enhanced by continuous two-way sonification. The full-squat sonification seemed to increase the perceived capability to complete the exercise. The evoked spring-like experience helped to maintain squatting pace, facilitating more squats. P20 said, “*When both movements were sonified, it enhanced the springy sensation and motivated me to do more reps. You saw I went down again for a few extra and that was due to the sound; I let the sound guide me and felt like I could keep going.*” This spring-like experience was reportedly enhanced when the pitch direction of the sonification matched movement direction (i.e., congruent sound–movement pairing). In fact, almost all participants described the congruent pairing as more natural and logical, mostly based on participants’ prior experiences of encountering such sound–movement pairings, like P7, “...we are used to these kinds of sound effects in films and media; I associate a rising sound with something going up like your body moving up. ... this association is intrinsic in our minds.” However, about two-thirds of the participants argued that while congruency (the alignment between pitch direction and movement direction) was logical, it was the contrast between the two sounds that was beneficial to facilitating the movement. The majority of participants agreed that the two movement directions of the squat should be sonified by a different sound for maximum benefit. P6 explained, “*It was more important than the two sounds are distinctive rather than which one is paired with which movement.*” P22 added, “*The two different sounds for two movement directions are good as it keeps the exercise interesting, exciting. If I had the same sound for five reps, it would be repetitive.*”

# 5 EXPERIMENT 2: METHODS

Results in Experiment 1 showed differences in effects for the ascending and descending sounds during squat exercises. Participants overall felt more positive, lighter, less tired, more capable, and motivated, as well as perceiving the movement to be easier with the ascending compared to the descending pitch sonification. Moreover, greater maximum acceleration was observed with the descending sound compared to the ascending sound regardless of movement direction. Further, our results showed that the participants disliked the incongruency between movement and sound direction and that the congruency between movement and sound direction enhanced the sound-induced body and emotional changes. The pairing of upward movement and sound seems to enhance the positive sensations associated with the upward sound. In contrast, the pairing of the downward movement and sound enhanced the feeling of heaviness. For some participants,
this was linked to feelings of “energy loss” but for other participants this feeling enhanced the perceived speed, quality, and amount of the strength downward movement.

Experiment 2 aimed to better understand the effects of the congruent full-squat sonification studied in Experiment 1 on body feelings, experience quality, and actual movement. Two control conditions were used to trigger comparative reflection in participants: no sensory feedback and haptic feedback. In this experiment, we focused on inactive or minimally active individuals (as defined in [37]), as previous works on movement sonification have shown that effects on some aspects of body perception, experience quality, and actual movement do not affect individuals with high physical activity levels [56]. Also, some participants in Experiment 1 suggested that the exercise was too easy for the sound to affect the movement. As a secondary aim, we investigated the effects according to the individual's actual physical activity level, as the literature shows that some factors such as personal values may trigger different effects of the feedback [99]. The study was conducted in person in a laboratory setting, which followed COVID-19 safety measures.

Experiment 2 was a mixed-method study with squat exercises accompanied by movement sonification, also approved by the UCL Research Ethics Committee (reference number 5095/001).

## 5.1 Participants

Experiment 2 was conducted face-to-face in a laboratory setting. There was no overlap between participants from Experiments 1 and 2. Participants in Experiments 1 and 2 were a completely different set of people due to time gaps and experimental scenarios. In Experiment 2, additional criteria were no known skin issues (as they had to wear a device on the skin) and self-identification as physically inactive. In addition, it was decided to recruit only women as participants. The reason was twofold: (1) to maximise the power of our findings given the physiological differences (muscle physiology) between the genders and the number of participants we could recruit [2, 21, 39] and (2) to ensure that the researcher collecting the data (a woman) and participants felt at ease during the placement of the electrodes on their upper body. Naturally, the findings from this second study cannot be generalized to a different gender, as we will discuss in the limitations and future directions. Finally, to verify physical activity level, participants completed the **International Physical Activity Questionnaire (IPAQ)** [37]. Half (N = 10) the participants were physically inactive, whereas others (N = 11) were minimally active. A total of 21 adults (age: mean = 24.24 years, SD = 7.09, range = 20–41; all women) participated.

## 5.2 Exercise Selection

Same as in Experiment 1.

## 5.3 Experiment Design

There were two independent variables in the experimental design: a within-subject independent variable, sensory feedback, with three levels (sound only, vibrotactile only, and a no-feedback-control), and a between-subjects independent variable, participants' physical activity level, with two levels (inactive and minimally active). As in Experiment 1, the measured dependent variables included behavioural measures and quantitative and qualitative self-report data. Some changes in these measures were introduced for a better understanding of people's sensations across the body.

**Body and movement feelings:** Body and movement feelings were captured using a contextual Body Map using an iPad (Figure 5). A Body Map is a very visual qualitative design and research tool featuring “a pictorial outline of the participant’s body” [17]. While they are used in multiple disciplines (including Health, Sociology, Anthropology, and Design Research) [1, 17, 33] differently and with slightly different variations, in body-focused design research, Body Maps typically involve an outline of the participant’s body onto which they draw their body experiences,
<figure><img src="image_7.png" /><figcaption>Fig. 5. (Left) A filled Contextual Body Map, showing effect on participant's body and movement feelings during exercise. (Right) Waist belt with vibration module. Five vibration motors were attached to the participant's back.</figcaption></figure>

including sensations and feelings (e.g., [1]). This allows participants to both reflect on and document their body sensations and feelings [1, 17, 33]. Like in other body-centered design research works (e.g., [57, 90]), we provided participants with “empty” outlines of bodies as templates to inscribe their sensations and feelings, which they did with color markers. Further, since Body Maps are visual representations of the participants’ physical form [17] and we were targeting women participants, we chose body outlines with identifiable female traits (Figure 5). Then, similar to [57, 93], our Body Maps were contextual in the sense that they represented key moments in the activity, such as standing and squatting. Like in [57, 90] as well, keywords next to the Body Maps were included to help participants map and illustrate body feelings relevant to the exercise. Then, like in [57], rather than to compare sensory modalities, we used Body Maps to support the participants’ reflection on sensations on the body in general and the different body parts related to the sensory feedback as well as on the impact of this feedback on their feelings about their body and movement. There was space to add notes for later discussion. Changes in movement feelings were quantified using 5-point Likert-type response items assessing participants’ feelings of being in control of their movement, of movement guidance, and of goal motivation. These subscales were presented with the SAM questionnaire and with the same pictorial format [6].

**Experience qualities:** The valence/happiness, arousal/excitation, and dominance subscales (5-point Likert-type response items) of the SAM questionnaire [6] were used to assess changes in feelings. Follow-up interviews explored in more detail such feelings and their possible cause.

**Movement behaviour:** This was measured using an **Inertial Motion Unit (IMU)** (see Section 5.4) attached to the participant's right thigh. Data treatment and extracted parameters were the same as in Experiment 1.

## 5.4 Materials

5.4.1 *Experiment Device and Feedback.* A wearable device (Figure 1) was developed to deliver sound and vibrotactile feedback in response to participants' movements and to measure movement behaviour. The device was based on [55, 56]. It integrated a wearable band with a wireless emitter (BITalino R-IoT embedding a 9-axis IMU digitized at 16 bits). The band wirelessly transmits movement data through WiFi using the OSC protocol to a laptop running Max/MSP (Cycling'74). The laptop stores raw movement data. A pair of wireless headphones (Sony WH-CH510) delivered sound feedback; a vibration module incorporated with the wearable device delivered vibrotactile feedback. The vibration module, based on [93, 98], consisted of five vibration motors
(10 mm in diameter) attached to the person's back, forming a vertical line along the spine (see Figure 1); motors were connected to a microcontroller with an adjustable waist belt (see Figure 5). The sensor was calibrated as in Experiment 1. Following this, the laptop detected and triggered sounds or vibrotactile feedback to accompany movement. To avoid extreme sensitivity, the threshold of movement angle to trigger the sound for both upward and downward squats was 20% in sound conditions. However, for the vibrotactile condition, the threshold was 5% for upward squats and 10% for downward squats to counterbalance the existing delay of activating vibration. Feedback sounds were the same as Experiment 1. We focused on full-squat sonifications with congruent movement-sound pairing (downward movement-descending pitch, upward movement-ascending pitch) for both sensory feedback conditions. For vibrotactile feedback, the 5 motors were activated in top-down or bottom-up sequences to generate motion [43] consistent with squat directions, based on [93]. Each vibrator was sequentially activated and vibrated for 434 ms, with 217 ms overlap between activations: the total duration of the vibrotactile sequence was 1.3 s, same as for the sound feedback. Vibration intensity for each motor in the sequence varied to emulate "bounciness" or "springiness" described for the sound feedback. The intensity was 135 for the first motor in the sequence, then 165 and 195, respectively, for the second and third motors, and 255 for the last two motors (intensity values are related to the vibration amplitude set in the microcontroller).

## 5.5 Experiment Procedure

The experiment was conducted in a lab. Participants gave informed consent and completed the IPAQ. Once instructed on the experimental procedure, the vibration motors were attached to the participant's back along the spine by placing the first motor under the bone of the neck and the last motor at the horizontal line of the hip bone, ensuring that all motors were equidistant. Next, the participants calibrated the device to set the maximum (i.e., starting) and minimum squatting positions, and practiced the exercise. Next, during the *experimental phase*, participants performed 3 sets of 5 squats. Movement data was continuously recorded during this phase, except during breaks between sets. Each set differed in sensory feedback received while performing the exercise (sound feedback, vibrotactile feedback, and no feedback). The order of conditions was fully randomized. During the short break between each set, participants completed the body map and questionnaires assessing experience quality and body and movement feelings. Finally, a short (10 minutes) follow-up semi-structured interview focused on participants' salient feelings about their body and movement behaviours and the comparison of body feelings across the three conditions in a combination of their self-reported surveys. Here, the body maps and the questionnaires filled out by participants were used to ask questions and maintain a discussion with the participants about their experience on the exercise and the sensory feedback. The interview was audio recorded with consent. At the end, participants were debriefed with contact information in case of inquiry.

## 5.6 Data Analysis

*Quantitative data analysis* of questionnaire data was done, as in Experiment 1, using 3x2 repeated-measures ART ANOVAs with feedback condition ('sound', 'vibration', 'no feedback') as within-subject factors and participants' PA level ('inactive', 'minimally active') as between-subject factors. For significant interactions, separate ART ANOVAs were conducted for the two groups of participants. In case of significance, these were followed by paired t-tests corrected for multiple comparisons. While movement data was non-normal, deviations from normality were within an acceptable range as the data showed a linear pattern. Therefore, repeated-measures ANOVAs were used to check interaction effects between feedback condition and PA level. Wilcoxon tests followed significant effects.
*Qualitative data analysis:* Body maps were analysed by linking the words reported by participants with the body parts they indicated by using colors (Figure 5). The analysis was done by providing an overview of differences in perceived body feelings with comparisons between the three sensory conditions with clustered bar charts. Findings were combined to better interpret the interview scripts. Interview audio recordings were transcribed verbatim. Transcripts were analysed using a combination of top-down and bottom-up thematic analyses [7]. Top-down codes used were changes in experience qualities, changes in perceived feelings, motivations, emotional states, perceived velocity, perceived movement behaviour, guiding effect of sound, distracting effect, and body control. These codes were mainly generated from the research goals on the difference between body perceptions and feelings, experience qualities, motivation, and physical activity behaviours during different feedback conditions. The analysis of Experiment 2 was led by the second author, who was present at the study and transcribed and analysed the data to identify themes. The resulting themes were discussed with the other authors before finalising and writing up. Most participants (N=20) expressed preferences for squats with sound or vibration feedback; thus, we focus on these conditions in various domains in results.

# 6 EXPERIMENT 2: RESULTS

## 6.1 Effects of Sensory Feedback on Body Feelings

Body maps yielded 163 records of body feelings for the three feedback conditions. Analysis suggests that sound is more capable of evoking body feelings during squats (N1 = 57) and associated with a greater number of body parts (N2 = 10) compared with the vibration condition (N1 = 55 and N2 = 9) and no-feedback condition (N1 = 54, N2 = 8). With vibration feedback, the most prominent feelings were of being flexible, fluid, and fast, respectively linked with the back (flexible and fluid) and whole body (fast). A notable finding for sound feedback was that a greater number of feelings targeted on the whole body were reported rather than single body parts. For instance, one-quarter of participants reported feeling flexibility and speed over the whole body. Thus, sound stimuli could lead to larger impacts on body sensation across the body, instead of body parts, around the signal inputs. In the no-feedback condition, participants reported more negative body feelings (e.g., feeling heavy, not flexible, or slow) than positive ones; more feelings associated with body parts from the middle body (e.g., upper thigh and hip), often associated with feeling heavy and strong.

The self-reported data shows a significant effect of the feedback condition on felt movement guidance ($F(2, 38) = 6.18, p = 0.004, \eta p^2 = 0.245$) and control over one's body ($F(2, 38) = 4.07, p = 0.025, \eta p^2 = 0.175$). Follow-up paired comparisons showed that participants felt less control over their body with vibration than with the no-feedback condition without sensory input ($F(60) = 2.53, p = 0.036$; Figure 6). However, participants reported more movement guidance in the vibration compared with the no-feedback condition ($F(60) = -2.42, p = 0.047$; see Figure 6). There were no significant differences between the sound and other two conditions. Further, there was a significant interaction effect between feedback condition and participant's PA level in terms of reported movement guidance ($F(2, 38) = 4.54, p = 0.016, \eta p^2 = 0.193$). Separate ART ANOVAs were conducted for the two groups of participants, which showed a significant effect only for the “minimally active” participants ($F(2, 20) = 11.68, p < 0.001, \eta p^2 = 0.538$). For those participants, both the sound ($F(30) = -3.02, p = 0.013$) and the vibration feedback ($F(30) = -3.97, p = 0.001$) provided a larger sense of movement guidance than the no-feedback condition. There were no other significant effects.

Follow-up interviews confirmed that both sound and vibration feedback affected participants' body feelings, but differently. Like in Experiment 1, participants related changes in sound pitch
<figure><img src="image_8.png" /><figcaption>Fig. 6. Boxplots for body feelings and experience qualities significantly influenced by the sensory feedback in Experiment 2 (* indicates $p < .05$, ** indicates $p < .01$, *** indicates $p < .001$).</figcaption></figure>

to dynamic objects such as a pinball or spring, and synchronous generation along with squatting direction enhanced such experiences in flexibility. Half of the participants reported feelings of lightness and flexibility over the whole body: P01 reported, “I felt a kind of bounce feeling on my knee or probably over the body, because its sound effect is very like a spring, and it makes my body feel lighter anyway.” Such feelings were most prominent at the beginning of the movement, as many participants highlighted that the intensity of sound faded away at the end positions, reducing the impact of the sound. P05 said, “I noticed there’s a change in sound when I go up and down. [...] it’s very in sync with my squats, and has a fade-away effect like strong at the beginning and weak at the end. ... I think it’s a whole, but the strong intensity at the beginning might influence my body most.” Two-thirds of participants enjoyed the vibration feedback, seeing it as helpful and eliciting feelings of flexibility and strength. P17 reported: “The vibration was quite enjoyable. I have a bit of back pain and I felt that it went away with the vibration. So it made squats easier. When going up and down, my back felt more flexible and stronger. It’s like adding fuel to my body.... I felt stronger going up and down with my legs.” For sound feedback, elicited body feelings depended on the interaction between pitch change (i.e., ascending and descending sound) and movement trajectory. Few participants reported that the sound effect in the downward versus the upward part of the squats was stronger and they perceived it as heaviness mainly on their upper thigh. The descending pitch generated a sensation of a dropping force when they moved down. This resonates with findings of Experiment 1, in which participants reported feeling pulled down by the sound. P15 reflected, “... the sound when I move down, I feel a dropping force to sort of help my movement by following the force created by sound when I went down.” This sensation of a force being exerted by the sound feedback was regarded by participants as an internalized force whereas the effect of vibration feedback was regarded as an externalized force, helping them to perform squats. Participants linked sounds to dynamic daily objects, such as springs or pinballs, and perceived themselves as those objects, with their properties. P19 said, “The sound [...] made me feel like I was a spring. ... I imagine a spring being very fluid, flexible. And because I had that image in my mind, I acted like the spring.” Conversely, vibration feedback was often described using metaphors such as an “add-on zip” or “hand rubbing” the back of participants and associated with a helpful force exerted on their back. P12 said, “It felt like someone had put a zip on my back, but it wasn’t uncomfortable; instead, it was fun. I feel the vibrations quite smooth by going up and down, like someone is pushing you up or pushing you down, so smooth.” As with quantitative self-report results above, most participants described the force being exerted by the sound and vibration feedback as beneficial in guiding squatting exercise. P8 said, “For the condition with no sound or vibration is like driving a boat on the sea, but for sound and vibration, it’s like you stop on the river but you can float by the waves from the water, and it’s external driven force.” Although their movement angles triggered the feedback, most participants
<figure><img src="image_9.png" /><figcaption>Fig. 7. Means and standard errors for movement angle, velocity and time indicating significant main effect of sensory feedback and a significant interaction of sensory feedback with participant's PA level on these movement parameters.</figcaption></figure>

regarded the sound and vibration signals as reminders of when to move upwards or downwards. P10 said, “...it's like a reminder, like setting an alarm clock at the time I need to go up and down, so it pushes me to do the next movement.” Both sound and vibration feedback were reported to make participants focus attention on detecting changes in the feedback away from their working bodies. The same effect was interpreted differently by different participants, leading to two perspectives. Some participants regarded it as an interference to fully perceive the sensations over the body, adversely affecting their correct position for performing squats, such as P19: “Vibration distracted me, so instead of keeping my back straight and squatting down backward, I thought I was squatting forward.” Others reported that this sensory distraction could provide a more relaxing experience, such as P13: “With the vibration, this time I focused all on my back instead of feeling my legs, so I felt much lighter on my legs and so relaxed so I could easily do the 5 squats.” P10 added, “The sound can just distract attention — it takes focus away from the limbs and thigh and tiredness is spread out.”

## 6.2 Effects of Sensory Feedback on Movement Behaviour

The feedback condition significantly affected peak angle (F(2, 206) = 4.71, p = 0.010, ηp2 = 0.044), mean angle (F(2, 208) = 5.27, p = 0.006, ηp2 = 0.048), upward velocity (F(2, 208) = 10.55, p < 0.001, ηp2 = 0.092) and downward velocity (F(2, 208) = 7.35, p = 0.001, ηp2 = 0.066). Participants squatted farther in the sound than vibration condition (Z = -2.41, p = 0.016). However, this comparison should be approached cautiously, as the triggering angle differed between these conditions. Participants' movements were faster with no feedback than with feedback, both for upwards (no feedback vs. sound: Z = -3.39, p = 0.001; no feedback vs. vibration: Z = -3.83, p < 0.001) and downwards (no feedback vs. sound: Z = -4.02, p < 0.001; no feedback vs. vibration: Z = -3.10, p = 0.002). When accounting for participants' PA level, differences in squatting movement quality were more prominent for the inactive compared with the minimally active group. A significant interaction effect was found between feedback conditions and PA level for 4 movement parameters (Figure 7): peak angle (F(2, 206) = 8.96, p < 0.001, ηp2 = 0.08), mean angle (F(2, 206) = 3.78, p = 0.024, ηp2 = 0.035), downward velocity (F(2, 206) = 13.57, p < 0.001, ηp2 = 0.116) and downward time (F(2, 206) = 6.27, p = 0.002, ηp2 = 0.057). Follow-up tests ran separately for inactive and minimally active groups showed significant differences only for physically inactive participants, who displayed greater squatting peak angles (F(2, 98) = 7.68, p = 0.001, ηp2 = 0.14) and mean
angles ($F(2, 98) = 6.41, p = 0.002, \eta p^2 = 0.116$) with sound than with vibration (peak angle: $Z = -2.36, p = 0.018$; mean angle: $Z = -2.54, p = 0.010$); slower downwards movement ($F(2, 138) = 13.92, p < 0.001, \eta p^2 = 0.167$) with vibration than no feedback ($Z = -4.88, p < 0.001$); and longer downwards time ($F(2, 98) = 4.58, p = 0.013, \eta p^2 = 0.085$) with vibration ($Z = -4.56, p < 0.001$) and sound ($Z = -4.12, p < 0.001$) than with no feedback.

Qualitative results showed that while both sensory feedbacks affected participants' behaviour in terms of guidance, the different nature of signals was associated with different aspects of guidance on squat behaviours. Some people perceived sound feedback as company that helped them combat the loneliness of self-exercise, similar to a voice chatbot. Indeed, sound feedback was often connected with the human voice, providing a feeling of being coached or accompanied, leading to a greater motivation to adapt behaviour in terms of squat quality. P8 elaborated, "*I feel like I'm being monitored [...] So that pushed me to do squats in a more standard way. Probably because the sounds remind me of the process of a child's growth, we always listen to the guide from teachers or parents*". Conversely, participants described vibration stimulation as intuitively guiding them. P10 explained, "*I felt some strength at my back, and then the vibration gave me the feeling like a flow behind me, that helped me to straighten my back or, more accurately, it reminds me to keep my back straighter*". The vibration was mainly amplifying the sensations on the back and the vertical vibration trajectory provided directional guidance. Time delays adversely interrupted the effect from sensory feedback. Most participants reported that sound and vibration stimuli were slower than the time required to complete the squats but chose to adapt their behaviour to follow the feedback after the first one or two squats. It affected the vibration condition more with two reports of a severe time delay for vibration and, consequently, a low sense of agency. P9 said, "*when I did squats, I realized that it didn't come with me so I didn't take the signal. My expectation was that I would squat and get up and down and the vibe would immediately follow, but I realized that it didn't, so I discarded that signal.*"

## 6.3 Effects of Sensory Feedback on Experience Quality

Results showed a significant effect of feedback on self-reported arousal ($F(2, 38) = 9.612, p < 0.001, \eta p^2 = 0.335$; see Figure 6, right). Participants felt more excited or aroused with sound feedback compared with no feedback ($F(60) = -3.06, p = 0.009$) and also with vibration feedback compared with no feedback ($F(60) = -3.198, p = 0.006$). There were no significant differences between sound and vibration conditions or effects on valence. In the interviews, participants reported feelings of excitement associated with sound feedback. Over half enjoyed congruent sound pitch with squatting movements, reporting elevated arousal and positive states, as in Experiment 1. P5 said, "*In the case of sound, I became EXCITED, because I hear sounds that are like a pinball. It reminds me of how I used to play pinball with my sis and it's interesting, less boring, and mood goes up, I want to hear it again.*" Almost two-thirds of the participants enjoyed the vibration feedback and described it as "massage", associated with relaxed feelings on the back or the whole body. They reported more perceived strength over the body and becoming more athletic in ongoing squats. Participants associated sensory feedback with rewards for movement. Almost all participants could identify the changes in sounds with ascending and descending pitch that were congruent with movement direction and associated pitch changes with dynamic objects with rewards. P12 said, "*... it was like game tones, it had that rewarding feeling that I crouched down at the correct position and there's a kind of 'bingo'. ... like you got a gold coin for crouching down a bit'*". Two-fifths of participants reported such incentive-based reward effects in the vibration condition. P8 explained, "*It's like when you play some drum games. ... how your bounces fit the vibration and I enjoy it, like a reward for completing a full squat*". The reward effect led to increased motivation in both sensory conditions in terms of quality and quantity. Nearly half associated sound feedback with increased motivation in terms
of velocity or number of squats, which contrasted with behavioural results showing that the sensory feedback slowed down movements. Wanting more dynamic sounds for positive experience resulted in higher motivation to perform more squats quicker. P10 said, “...I know that two sound tones were different because my movement changed, so I want to continue doing the movement to hear it. I love this sound.” Vibration being self-triggered by movement also increased the motivation to squat but less than in sound conditions and fully concentrated around the stimulated body part. In contrast to sound, vibration feedback increased the motivation to improve the quality, rather than the velocity or number, of squats. The pattern of vibration direction was perceived more clearly compared with sound, and the congruency between vibration and squatting direction encouraged participants to perform deeper to align with the vibration position. P13 said, “It feels like a massage while moving, like playing a virtual figure in a video game. ... the vibrations have a process from the bottom to the top and then top to the bottom, because I also have such a process of movement. I try to follow the rhythms to reach the deep position when going down.”

# 7 DISCUSSION

The first question that our study explored was whether vertical asymmetries in the effect of sonification, found in simple exercises such as arm raising [55], would also occur in the context of strength exercise. This point is important to assess to provide valid sonic interaction design recommendations in this case that concerns a large set of mobile applications for sport training. The fact that results found in arm raising might not simply be transferred to strength exercise is due to the attention focus during exercise being largely dependent on exercise intensity [41]. Thus, we cannot infer a priori that people would attend to the sound rather than their physiological sensations. Both our quantitative and qualitative findings confirm the asymmetries in the sonification effects for ascending and descending sounds during squat exercises. Participants overall felt lighter, less tired, more capable, and motivated. Further, they perceived the movement to be easier and felt more physically active with the ascending compared with the descending pitch sonification. Experiment 2 showed that the feelings of lightness and flexibility induced by sound spread over the whole body (as opposed to a specific body part, such as in the case of vibration feedback). These effects may be linked to effects on proprioception and, ultimately, to the facilitation and enhancement of physical activity. In addition, we have extended the previous literature by investigating downward sonification accompanying not only upward movements but also downward movements. Indeed, no single study had previously measured the effects of the descending pitch sound in relation to a downwards movement and, based on previous studies [95, 97], it could not be assumed that the direction of the sound would have (or not have) an effect on changes in body perceptions (we expand on this point in the next sections). Asymmetries in the association of music parameters with movements have been found by the authors of [26], who report that “listeners who associate a musical stimulus with a particular kinetic quality often do not associate the inverse stimulus with the opposite kinetic quality”. Asymmetry in loudness perception variations for tones increasing or decreasing in pitch was also found by the authors of [88]. First, our results showed that participants disliked the incongruency between movement and sound direction and that the congruency between movement and sound direction enhanced the sound-induced body perception and experience quality. That is, the pairing of upward movement and sound was enhancing the positive sensation of lightness, speed, energy, and positive mood induced by the upward sound. It also increased the perceived speed and the perceived sense of agency over the sound, and it was perceived as helping in completing the squat. In contrast, the pairing of the downward movement and sound was enhancing the feeling of heaviness, slowness, and lack of energy. However, our results show also that what may be perceived as a negative body feeling may indeed be helpful in a strength downward movement. The feeling of being heavier provided a sense of facilitation in our
<table><caption>Table 1. Body-Effect Sonification Design Framework with Design Implications Based on the Effects of Directional Sensory Feedback Observed in This Study</caption><thead><tr><th></th><th>Ascending sound with upwards movement</th><th>Descending sound with downwards movement</th><th>Ascending sound with downwards movement</th><th>Descending sound with upwards movement</th><th>Vibration congruent with movement direction</th></tr></thead><tbody><tr><th>Experience quality</th><td>Energetic, excited, <strong>positive</strong>, correct</td><td>Negative, incorrect, enjoyed by some</td><td>Energetic, excited, <strong>positive</strong>, correct</td><td><strong>Negative</strong>, incorrect</td><td>Excited</td></tr><tr><th>Body feelings</th><td><strong>Light, less tired</strong>, <strong>capable, motivated</strong>, physically active</td><td>Heavy, tired, less <strong>capable</strong>, less <strong>motivated</strong></td><td>Light, less tired, <strong>capable</strong>, <strong>motivated</strong></td><td><strong>Heavy, tired, less capable</strong>, less <strong>motivated</strong></td><td>Flexible, strong</td></tr><tr><th>Body part affected</th><td>Whole body</td><td>-</td><td>-</td><td>-</td><td>Stimulated body part (here, trunk)</td></tr><tr><th>Feelings of movement and proprioception</th><td><strong>Fast, easy</strong>, useful</td><td>Slow, no energy</td><td>Fast, easy, useful</td><td><strong>Slow</strong></td><td>Fast, fluid</td></tr><tr><th>Behaviour</th><td><strong>Lower speed than no sound</strong></td><td>Increased acceleration <strong>Lower speed than no sound</strong></td><td>-</td><td><strong>Increased acceleration</strong></td><td><strong>Lower speed than no sound</strong></td></tr><tr><th>Guiding force</th><td>Lifted up, agency over sound, internal (self) force</td><td>Pushed down, internal (self) force</td><td>Reduced agency over sound</td><td>Reduced agency over sound</td><td>External force</td></tr><tr><th>Physical activity facilitation</th><td>Positive, facilitating</td><td>Facilitating</td><td>Positive, facilitating, but not preferred</td><td>Inhibiting, least preferred</td><td>-</td></tr></tbody></table>

Color indicates if effects on strength exercises were congruent (Green) or incongruent (Blue) with those observed in light exercises [55]. Gray background shows that the question had not been previously investigated.

participants as if the sound operated as a gravitational force. This is interesting for any strength downward movement in which there is a conflict between the slowness of the movement due to movement control and muscle tension and the need for sense of progress (e.g., returning from a pull-up bar movement, or downward phase of a plank exercise, or return of weighted arm raised exercise). Still, the effects generated by the interaction between the direction of the sound and of the movement are more complex, as discussed in the following sections. Table 1 summarizes the design implications based on the effects observed in this study; we compare the effects with those observed for a light exercise, as reported in [55], as for that study the same sounds were used and movement behaviour, body movement, and emotion feelings were also measured.

## 7.1 Ascending Pitch Is Perceived as More Positive than Descending Pitch Even in Strength Exercises

The results from this research show that the ascending sound led to feelings of “springiness or bounciness” or being “pulled up”, which seem to be more prominent at the beginning of the movement and align with the qualitative reports of this sound in [57]. These observed effects relate to the effects of the cross-modal correspondence between pitch and vertical movement (e.g., [26, 51, 58, 104]) by which an ascending sound is associated with motion upwards and may also affect the perceived body position, as suggested by [55, 97]. Such associations of tonal sounds rising in pitch have been also found in gestural depictions of sounds [53]. Moreover, the feelings of lightness and other PA-facilitation body feelings found for the ascending sounds also link to the bottom-up multisensory body-perception mechanisms by which enhancing the high frequency of action sounds (i.e., footstep sounds) lead to a perceived lighter body weight [91] and make people feel more capable of exercising [95]. Like the ascending pitch squat sonification, the ‘high-frequency’ footstep sounds whilst walking also significantly enhance emotional state, whereby the listeners feel more aroused and positive [91, 95]. Moreover, a recent study investigating the effect of manipulating the static pitch of classical piano excerpts found a positive association between pitch
height and reported valence [45]. Thus, unlike the difference in the effect of static compared with
dynamic pitch on physical size [27], the effect of static and dynamic pitch on valence and arousal
seems to be rather similar. Previous research found that sound in the lower-frequency range can be
perceived as less pleasant than sound in the higher-frequency range in specific contexts [45, 110].
This contrasts with our results, suggesting that the “metaphorical effect” of pitch change over-
rule other direct effects. The reports of the qualitative experiences reveal positive–negative and
success–failure contrasts between the two sonifications: participants commonly perceived the as-
cending pitch as the positive and the descending pitch as the negative sound. They also associated
the ascending pitch with correctly performed movement and the negative descending sound with
incorrect movement. Moreover, they perceived their body and movement more positively when
their squat was accompanied by the ascending pitch, which led to positive reinforcement and en-
couragement to continue with the exercise, and more negatively with the descending pitch, which
triggered feelings of failure and incorrectness and resulted in participants rejecting the sound and
not taking ownership of it. In existing psychological literature, there is evidence of a metaphor-
ical linkage between spatial elevation and various dimensions of experience and cognition. This
includes the use of the ‘up’ and ‘down’ contrast to denote distinctions in positive and negative
affect, as well as dominance and power dynamics [24, 62]. In prior works, the connection between
high/low pitch and high/low locations has also be explained partly through a more fundamen-
tal metaphorical association, in which ‘down’ represents negativity and ‘up’ signifies positivity
[22, 45]. Our study findings align with this metaphorical association, demonstrating that sounds
descending in pitch (interpreted as descending to lower positions) were associated with failure
or incorrectness, denoting negative contexts. Conversely, sounds ascending in pitch (interpreted
as ascending to higher positions) were linked to positive contexts. Nevertheless, a more complex
analysis is certainly needed. First, it remains to be established whether the triggered change in feel-
ings and experience relate to the absolute frequency range of the sound ending or more specifically
to the direction of the pitch change, and the interaction between the loudness and pitch profiles
that suggest different metaphors. Further investigations are needed to establish which sound pa-
rameters drive the association with certain metaphors, and whether expectancy mechanisms cre-
ated by the sound dynamics (pitch and/or loudness profiles) induce or favor specific movements
[68, 69].

## 7.2 Physical Demand of Exercise Shifts Attention to the Proprioceptive Feelings Triggered by the Directional Sound

Our current results suggest that sonification directions (referring to increases/decreases in pitch) may have different effects in demanding exercises from those observed for light exercise. They showed that the descending sonification increased the maximum acceleration of both the upward and downward movement directions. This contrasts with the reports from participants, who described the ascending sound as being, in addition to positive (see Section 7.1), energetic, empowering, stimulating, and awakening, energizing, and useful when trying to be more “explosive” (i.e., high-intensity, short-duration movement). Further, our results also contrast with those from previous literature on a simple upward arm raise movement [55]: in that study [55], higher acceleration was observed for the ascending versus the descending sound. It is possible that the difference lies in the level of demand of the movement. For easy movements, the attentional focus may be on enjoying the experience, and then the positive and energetic feelings triggered by the ascending sound may have led to overdoing and speeding up the movement, whereas the negative feelings triggered by the descending sound may have reduced the motivation to perform the movement. Instead, the physical demand of the squat (or similar downward demanding movements) may have shifted the attention to the body, and then the proprioceptive feelings of slowness and heaviness
triggered by the descending sound may have invoked a movement counteraction. In other words, the descending sound makes people feel heavier and slower; then, people may feel the need to push their movements further to compensate for it and increase the speed and energy. This potential counteraction could explain the observed increase in maximum acceleration in both upward and downward movement directions in response to the descending sonification, which can be beneficial for some exercises, such as explosive strength training. In supporting such an interpretation, in the descending-down pairing, while the feelings of heaviness were evoked, these were surprisingly seen as positively impacting the exercise, as this heaviness enhanced the perceived speed, quality, and amount of the downward movement, as if participants were being “pulled down” by the sound. This result is interesting, as it shows how the effect of sonification on PA, modulated through dynamic body perceptions, not only depends on the form of the sound feedback but also on other internal factors that are critical in the specific activity or context. Previous literature had shown that the direction of changes in body size perception and their physical activity facilitating/inhibiting effect, which were triggered by changes in frequency of footstep sounds, were affected by body size-related personal values. Those wanting to be heavier appreciated feeling heavier in response to the low-frequency sound, whereas those wanting to be thin appreciated feeling lighter in response to the high-frequency sound [95]. Our work adds to this literature by showing that not just internal body values but also demand of physical activity (with the shift in attention it may trigger: from emotional to proprioceptive sensation) modulates the responses to the same perceived body change. This interaction is also supported by Experiment 2 results suggesting that the effects may vary according to the level of fitness of the person and appear stronger in people with a low fitness level. In particular, we found that while the minimally active group was more aware of the movement guidance provided by the sensory cues, such cues had a larger influence in the squatting movement quality (movement angle, time, and velocity for the downward movement) of the inactive group. This interaction between focus of attention and effect of sound on behaviour may also be supported by the Experiment 2 results, in which haptic feedback was used as a second control condition in addition to a more standard no-sound condition. The results of Experiment 2 show that while haptic feedback is perceived as an external force that helps the movement, sound is perceived as an internal force and possibly more easily associated with oneself and other internal processes. Indeed, sound was often considered more inviting to shift the attention from the body itself, whereas the haptic feedback seems to bring back the attention to the specific body part stimulated. This is also clear from the fact that, in Experiment 2, people associate haptic feedback with changes and sensations on the trunk, whereas the sound was associated with a variety of body parts.

## 7.3 Novelty and Contributions

Our work presents significant contributions and novel insights regarding the impact of sound on movement facilitation and the associated sensory processes leading to changes in body perceptions, differentiating it from previous work in five key aspects.

(1) **Nua**nceded Understanding of the Facilitation Effects of the Sound and Verticality Association on a Vertical Strength Exercise: While the connection between sounds changing in pitch and the perception of verticality has been acknowledged in prior works [22, 27] and exploited in artistic practices [61] and rehabilitation [70], our research offers a deeper understanding in relation to exercise facilitation. We reveal a preference for ascending sounds concerning perceived success during exercise. Moreover, we highlight the unique benefits of both ascending and descending pitch sounds, contributing valuable insights for a broader understanding of their impact on body perceptions and on movement qualities during exercise performance.
(2) **Comparative Analysis of Sound and Haptic Feedback:** While haptic feedback was included for comparative purposes, our study unveils distinctive sensory and motor processes associated with sound and vibration. We challenge the prevailing trend of utilizing vibration in commercial devices [19, 107], perhaps due to its social acceptability [20], suggesting that auditory feedback may offer advantages in sensory-motor learning [3], as sound is perceived as an internal force. This raises crucial questions regarding the extent to which haptic feedback truly facilitates motor learning compared with auditory feedback (see, for instance, works on the use of haptic feedback [111] or audiomotor feedback [30, 35] for motor learning). Our results suggest that audio might be more beneficial for self-connection during exercise, while haptic feedback may be more conducive to engaging with the external space. These findings shed new light on the design and utilization of feedback modalities for optimizing exercise experiences and motor learning.

(3) **Fitness Level-Dependent Effects and Body Perception:** A distinctive contribution of our study is the observation that the effects of sound on movement facilitation may vary based on an individual's fitness level, exhibiting a greater impact in individuals with lower fitness levels. This was already suggested by a prior qualitative study [57], but our study provides experimental evidence for it. This insight introduces new prospects for effectively supporting individuals who face challenges in engaging with physical activity. Moreover, this contribution enriches the existing literature on the multifaceted factors that influence body perception, incorporating physical activity level as a significant determinant alongside already recognized personal values [95]. By delineating how fitness levels interact with sound-induced effects on movement, our research enhances the understanding of the complex interplay between body perception and physical activity engagement. Further, while not a novelty originating from this study specifically but rather stemming from recent research, we wish to emphasize our **perception-centric approach to exercise**. This diverges from prior research exploiting cognitive behavioural therapies [40, 76, 108]. Rather than centering on reward-based or goal-oriented behavioural strategies, our study aligns with a complementary approach proposed in recent studies [57, 91], which focuses on how individuals perceive themselves and their bodily movements **during the moment of exercise**.

(4) **Use of Sound for Enhancing Virtual Reality (VR) and Game Experience:** Our findings contribute to other domains by providing further understanding on how sound could be exploited to alter people's experience of their body and movement. For example, the findings from our studies open new opportunities for game scenarios that involve vertical body movements. Large or demanding vertical body displacement in games has been addressed by physically lifting a person using ropes attached to large infrastructures, which require large spaces [49, 103], or using heavy wearable propellers [78, 100], limiting the person's movement. A more viable solution has been recently proposed by the authors of [71]. They alter the perception of the height of a vertical displacement by moving a weight up and down along the user's back. The movement of the weight alters the perceived jump momentum, creating the sensation of a more accelerated or decelerated vertical jump. In addition, they show that such an approach reinforces the visual illusion when used in VR. Unfortunately, this solution requires a person to wear a large backpack. Our findings suggest that this illusion could be modulated by simply altering the sonification of a person's vertical displacement. Furthermore, as shown in [71], the combination of downward and upward sonification congruent or incongruent to the vertical displacement could enable the creation of various modulations, making the vertical displacement feel easier or more demanding (creating an illusion of resistance) as required. The impact itself could also be
manipulated by altering the ending of the sonification (e.g., slowing it down or interrupting it abruptly) to alter the feeling of the terrain material [8] and its interaction with the body weight. However, it remains to investigate the amount of illusion displacement that the sound would be able to generate alone or in combination with other congruent sensory feedback.

(5) **Leveraging Sonification for Strategic Congruence and Incongruence to Shape Body-Movement Associations:** In addition, this work contributes to the body of literature on sensory incongruence [63, 101]. A synthesis of the literature on the use of sensory alignment from various disciplines in the area of entertainment technology [60] suggested that audio interacting with kinaesthetic experience appears to only enhance the sense of body rotation. Our findings suggest that carefully designed movement sonification could offer additional opportunities to enhance or alter the VR kinaesthetic experience by altering the perceived body weight and augmenting the sense of success or failure. In terms of incongruence between sensory stimuli, Kim and Lee [48] found that semantic incongruence between audio and visual feedback did not lead to a negative effect. Instead, it created a novelty effect. The authors argue that this difference from previous studies on sensory incongruence was due to the audio representing aspects of the environment only and not of the person- avatar. In our study, the incongruence introduced by sound was spatial and related to one's body. A possible explanation for the acceptance of the illusion could be that the sonification was not a natural sensory experience that one would normally encounter when moving (such as it would be for footstep sounds [96]). Hence, we argue that people were able to attribute meaning to the incongruence between the sound and the different aspects of their body and movement, allowing them to make sense of it in that way. They attributed the incongruence in direction between the movement and the sound to the weight of the body rather than the movement direction, using this link to guide their meaning-making process and evaluation of the experience. This was enabled by the fact that the temporal alignment was maintained to ensure agency [63], enabling spatial and metaphorical/semantic associations to take place. One could argue, therefore, that the congruence and incongruence of sonification with the movement could be carefully designed to trigger associations with the movement or other aspects of the body that may influence movement to reach the desired experiential outcome.

## 7.4 Limitations and Future Research

7.4.1 *Generalizability.* A limitation regarding generalizability of this study, like many others in the field (e.g., [55, 68, 69, 81, 83]), is the focus on one particular exercise (in our case, a strength exercise: squats). While each individual focused study like ours is not able to make generalizable claims about physical activity in general, they add to the earlier body of work, putting to test, but also building on and extending designs (e.g., different sonifications), application (e.g., different physical exercises), and insights in those earlier studies. Hence, the growing body of focused studies like ours allow for a growing understanding of the interaction between sound and movement behaviour, body perception, and experience quality in physical activity. Yet, further work is needed to test our findings with other body movements and types of exercises in order to generalize our findings. In particular, future squat sonification designs should be evaluated both with regard to their short- and long-term effects in the context of physical activity participation. Further, as the focus of our article was on building on understanding the experience quality of participants, we measured emotional responses with affective reports; body perceptions were measured in terms of body feelings and impact on movement behaviour. Because emotional and body feelings are subjective, to have a full understanding of how emotional state and body-perception were affected
by the use of our device, future studies could investigate the effects on physiological reactivity and other overt behavioural acts apart from the body movement features explored in this study.

Finally, a decision was made to solely include women as participants in order to mitigate potential confounding factors arising from physiological differences. Consequently, our findings from this study can only be extrapolated to this specific population and are not completely generalizable. As a next course of action, we plan to widen the recruitment criteria and delve deeper into the impact and interactions of these variables in relation to sonification effects.

### 7.4.2 Methodology.
Technical issues were exacerbated due to methodological complications imposed by COVID-19. In particular, the experimental application used in Experiment 1 needed to be deployed remotely in different environments, on different smartphones and software versions with no researcher support to position the phone on the body or troubleshoot the application. In some smartphone models, participants reported a slight sonification lag, especially during the upward movement. This technical issue is likely due to the audio latency of specific smartphone models. Delays between action and sensory feedback can disrupt the agency and diminish the saliency of the sensory-induced bodily illusions [96], as reported in some cases in this study. This adverse delay was minimised in Experiment 2, which ran in a lab setting with the support of a collocated researcher. Still, some participants reported delays in sound and vibration in line with their movement, which was due to a rapid launch of squats (i.e., in those cases, the reported issue was not due to latency but rather to the duration of sound and vibration stimuli being longer than the time required to complete the squats). Experiment 2 only included women as participants. Future work would require including more diverse participants with regards to gender as well as other aspects (e.g., fitness level, age, body ideals). With respect to tools used in this experimental design, in retrospect we realized that the body outlines used in this study portrayed an exaggerated and far from common idealized female body, which could potentially have an opposite effect in the population than the tool intended, i.e., feeling not well represented or even excluded [85]. Hence, we encourage researchers employing Body Maps to employ more inclusive empty outlines better representing the plurality of bodies that the design or research targets [85].

### 7.4.3 Sonification Design.
Also related to a methodological limitation, in Experiment 2, we chose the haptics modality as the second control condition to trigger comparative reflections in people and help them make explicit their experiences. However, a direct comparison of the effects was not intended. Nevertheless, the vibration feedback emerged to be interesting on its own. Recent works have shown the potential of haptic metaphors delivered to the body to elicit changes in bodily feelings (feeling heavy, strong) [98] and facilitate physical activity through such body-perception changes [52, 93]. These results together invite more research on haptics for body perception changes in physical activity. Moreover, the present study used sounds and vibrotactile sequences of a fixed duration (i.e., 1300 ms). We found that the temporal alignment between the squat sonification onset as well as its progression and the actual movement is important to ensure a sense of agency and saliency of evoked bodily experiences, for instance, to avoid misalignment between stimulus and movement in the case of rapid squats. Because of this, an anchoring approach [76] whereby small pieces of the sonification are played at each movement stage (i.e., discrete sonification) could strengthen the mental association between the movement and the sound. Given that the most salient and enjoyable experience evoked by the squatting movement sonification was the spring-like or bouncy-ball sensation, future squat sonifications could explore more metaphorically aligned sound-movement pairings [56]. Hence, using spring-like or bouncy-ball sounds instead of the musical sounds for the squatting movement sonification might enhance the reported facilitatory spring-like sensations as well as the saliency of other auditory bodily illusions. These sounds could still incorporate the pitch change and by doing so not only facilitate the reported
changes in body feelings but also enhance the sense of agency. As shown by qualitative reports, the ascending-up pairing was perceived as the most synergetic and in sync with one's movement, resulting in a higher perceived sense of agency over the sound. In contrast, participants reported that the descending sound led them to reject it as being produced by them, in order to minimise the negative association between this sound and feelings of failure and incorrectness.

## 8 CONCLUSION

Previous works have shown that sounds ascending in pitch accompanying a simple movement create a sense of lightness and ease, accelerate movement, and trigger positive feelings, whereas sounds descending in pitch create a sense of heaviness and slowness. Here, we investigated whether these effects extend to an exercise requiring larger effort and tested for the first time the interaction between sound direction and movement direction. Our study provides valuable and unique insights into the influence of sound on movement facilitation and body perceptions during exercise, distinguishing itself from prior research by addressing three critical aspects: a nuanced understanding of sound and verticality association, a comparative analysis of sound and haptic feedback, and the identification of fitness level-dependent effects on body perception during physical activity. Our results show that independent of the movement direction, the ascending pitch sound triggers feelings of being lighter, less tired, and more capable, whereas the descending pitch triggers feelings of being heavier, even in a more demanding exercise. Extending previous results, we show that the congruence between the direction of the movement and the direction of the sound amplifies the related effects. In addition, our work shows that the different elicited body and experience qualities by both ascending and descending sounds support movement and physical activity in different ways, in particular because the increase in the physical demand of exercise seems to shift the focus of attention from the positive or negative feelings to the proprioceptive feelings triggered by the sound. Finally, we show that sound feedback generates an internalized force that guides and helps the performance of squats, especially in inactive people. Our results open opportunities for supporting physical activity in demanding situations through wearable devices incorporating sonification of movement using musical sounds changing in pitch. While we have focused on one exercise only, squatting, its physical demand, complexity in terms of demand on various body parts, and its complementarity with literature on an arm raise exercise provide insights into possible effects on a variety of vertical displacement strength exercises. Transfer of effects to other strength vertical exercises (e.g., jump-squat, cube-step, weight lifting, plank, pull-up bar) is also supported by the similarity in results observed on the facilitation and positive effects of the ascending sound on the ascending direction of the movement despite the strength demand.

## ACKNOWLEDGMENTS

This work was supported by the Spanish Agencia Estatal de Investigación project grant "MAGIC outFIT" I + D + i /PID2019-105579RB-I00, funded by the Spanish Agencia Estatal de Investigación (MCIN/AEI/10.13039/501100011033/), and by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 101002711). J.L. was partially supported by the doctoral training grant BES-2017-080471.

## REFERENCES

[1] Karen Anne Cochrane, Kristina Mah, Anna Ståhl, Claudia Núñez Pacheco, Madeline Balaam, Naseem Ahmad-pour, and Lian Loke. 2022. Body maps: A generative tool for soma-based design. In *16th International Conference on Tangible, Embedded, and Embodied Interaction (TEI '22)*. ACM, New York, NY, Article 38, 14 pages. DOI: https://doi.org/10.1145/3490149.3502262
[2] Sandro Bartolomei, Giuseppe Grillone, Rocco Di Michele, and Matteo Cortesi. 2021. A comparison between male and female athletes in relative strength and power performances. *Journal of Functional Morphology and Kinesiology* 6, 1 (2021). DOI: https://doi.org/10.3390/jfmk6010017

[3] Frédéric Bevilacqua, Eric O. Boyer, Jules Françoise, Olivier Houix, Patrick Susini, Agnès Roby-Brami, and Sylvain Hanneton. 2016. Sensori-motor learning with movement sonification: Perspectives from recent interdisciplinary studies. *Frontiers in Neuroscience* 10 (8 2016), 385. Issue AUG. DOI: https://doi.org/10.3389/fnins.2016.00385

[4] Stuart Biddle and Nanette Mutrie. 2007. *Psychology of Physical Activity*. Routledge, London. DOI: https://doi.org/10.4324/9780203019320

[5] Matthew Botvinick and Jonathan Cohen. 1998. Rubber hands 'feel' touch that eyes see. *Nature* 391 (2 1998), 756-756. Issue 6669. DOI: https://doi.org/10.1038/35784

[6] Margaret M. Bradley and Peter J. Lang. 1994. Measuring emotion: The self-assessment manikin and the semantic differential. *Journal of Behavior Therapy and Experimental Psychiatry* 25 (3 1994), 49-59. Issue 1. DOI: https://doi.org/10.1016/0005-7916(94)90063-9

[7] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. *Qualitative Research in Psychology* 3, 2 (2006), 77-101. DOI: https://doi.org/10.1191/1478088706qp063oa

[8] Roberto Bresin, Anna de Witt, Stefano Papetti, Marco Civolani, and Federico Fontana. 2010. Expressive sonification of footstep sounds. https://apisemanticscholar.org/CorpusID:41479908

[9] Stephen Brewster and Lorna M. Brown. 2004. Tactons: Structured tactile messages for non-visual information display. In *Proceedings of the 5th Conference on Australasian User Interface—Volume 28 (AUIC '04)*. Australian Computer Society, Inc., AUS, 15-23.

[10] Stephen Brewster and Ashley Walker. 2000. Non-visual interfaces for wearable computers. In *IET Conference Proceedings*. IET Digital Library, London, 6-6. Issue 145. DOI: https://doi.org/10.1049/ic:20000511

[11] Giada Brianza, Ana Tajadura-Jiménez, Emanuela Maggioni, Dario Pittera, Nadia Bianchi-Berthouze, and Marianna Obrist. 2019. As light as your scent: Effects of smell and sound on body image perception. In *Human-Computer Interaction – INTERACT 2019*. David Lamas, Fernando Loizides, Lennart Nacke, Helen Petrie, Marco Winckler, and Panayiotis Zaphiris (Eds.). Springer International Publishing, Cham, 179-202. DOI: https://doi.org/10.1007/978-3-030-29390-1_10

[12] Kendall J. Burdick, Seiver K. Jorgensen, Megan O. Holmberg, Samantha P. Kultgen, Taylor N. Combs, and Joseph J. Schlesinger. 2018. Benefits of sonification and haptic displays with physiologic variables to improve patient safety. *Proceedings of Meetings on Acoustics* 35. Issue 1. DOI: https://doi.org/10.1121/2.0000941

[13] Nihal Büker, Raziye Şavkın, Akin Süzer, and Nuray Akkaya. 2021. Effect of eccentric and concentric squat exercise on quadriceps thickness and lower extremity performance in healthy young males. *Acta Gymnica* 51 (2021). DOI: https://doi.org/10.5507/ag.2021.015

[14] Lucilla Cardinali, Francesca Frassinetti, Claudio Brozzoli, Christian Urquizar, Alice C. Roy, and Alessandro Farnè. 2009. Tool-use induces morphological updating of the body schema. *Current Biology* 19, 12 (2009), R478-R479. Issue 12. DOI: https://doi.org/10.1016/j.cub.2009.05.009

[15] Daniel Cesarini, Davide Calvaresi, Chiara Farnesi, Diego Taddei, Stefano Frediani, Bodo E. Ungerechts, and Thomas Hermann. 2016. MEDIATION: An eMbEddeD system for auditory feedback of hand-water InterAction while swimming. *Procedia Engineering* 147 (1 2016), 324-329. DOI: https://doi.org/10.1016/j.proeng.2016.06.301

[16] Fabio Ciravegna, Jie Gao, Neil Ireson, Robert Copeland, Joe Walsh, and Vitaveska Lanfranchi. 2019. Active 10: Brisk walking to support regular physical activity. In *Proceedings of the 13th EAI International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth'19)*. ACM, New York, NY, 11-20. DOI: https://doi.org/10.1145/3329189.3329208

[17] Bronwyne Coetzee, Rizwana Roomaney, Nicola Willis, and Ashraf Kagee. 2019. Body mapping in research. In *Handbook of Research Methods in Health Social Sciences*. Pranee Liamputtong (Ed.). Springer Singapore, Singapore, 1237-1254. DOI: https://doi.org/10.1007/978-981-10-5251-4_3

[18] Jacob Cohen. 2013. *Statistical Power Analysis for the Behavioral Sciences*. Routledge, New York. DOI: https://doi.org/10.4324/9780203771587

[19] *CuteCircuit*. 2018. The Hug Shirt. Retrieved from http://cutecircuit.com/the-hug-shirt/

[20] Ella Dagan, Elena Márquez Segura, Ferran Altarriba Bertran, Miguel Flores, Robb Mitchell, and Katherine Isbister. 2019. Design framework for social wearables. In *Proceedings of the 2019 on Designing Interactive Systems Conference (DIS '19)*. ACM, New York, NY, 1001-1015. DOI: https://doi.org/10.1145/3322276.3322291

[21] Michael J. Del Bel, Teresa E. Flaxman, Kenneth B. Smale, Tine Alkjaer, Erik B. Simonsen, Michael R. Krogsgaard, and Daniel L. Benoit. 2018. A hierarchy in functional muscle roles at the knee is influenced by sex and anterior cruciate ligament deficiency. *Clinical Biomechanics* 57 (2018), 129-136. DOI: https://doi.org/10.1016/j.clinbiomech.2018.06.014

[22] Ophelia Deroy, Irune Fernandez-Prieto, Jordi Navarra, and Charles Spence. 2018. Unraveling the paradox of spatial pitch. In *Spatial Biases in Perception and Cognition*. Cambridge University Press, Cambridge, UK, 77-93. DOI: https://doi.org/10.1017/9781316651247.006
[23] Arife Dila Demir, Nithikul Nimkulrat, and Kristi Kuusk. 2022. "Squeaky/Pain": Cultivando experiencias perturbadoras y la transición de perspectivas para las interacciones somaestéticas. *Revista Diseña* 20 (2022). DOI: https://doi.org/10.7764/disena.20 firman

[24] Sarah Dolscheid and Daniel Casasanto. 2015. Spatial congruity effects reveal metaphorical thinking, not polarity correspondence. *Frontiers in Psychology* 6 (Nov 2015). DOI: https://doi.org/10.3389/fpsyg.2015.01836

[25] Judy Edworthy and Hannah Waring. 2006. The effects of music tempo and loudness level on treadmill exercise. *Ergonomics* 49, 15 (2006), 1597-1610.

[26] Zohar Eitan and Roni Y. Granot. 2006. How music moves: Musical parameters and listeners images of motion. *Music Perception* 23 (2 2006), 221-248. Issue 3. DOI: https://doi.org/10.1525/mp.2006.23.3.221

[27] Zohar Eitan, Asi Schupak, Alex Gotler, and Lawrence E. Marks. 2014. Lower pitch is larger, yet falling pitches shrink. https://doi.org/10.1027/1618-3169/a000246 61 (1 2014), 273-284. Issue 4. DOI: https://doi.org/10.1027/1618-3169/A000246

[28] R. F. Escamilla. 2001. Knee biomechanics of the dynamic squat exercise. *Medicine and Science in Sports and Exercise* 33 (2001), 127–141. Issue 1. DOI: https://doi.org/10.1097/00005768-200101000-00020

[29] Jamie Ferguson, John Williamson, and Stephen Brewster. 2018. Evaluating mapping designs for conveying data through tactons. In *Proceedings of the 10th Nordic Conference on Human-Computer Interaction (NordiCHI '18)*. ACM, New York, NY, 215-223. DOI: https://doi.org/10.1145/3240167.3240175

[30] Emma Frid, Jonas Moll, Roberto Bresin, and Eva Lotta Sallnäs Pysander. 2019. Haptic feedback combined with movement sonification using a friction sound improves task performance in a virtual throwing task. *Journal on Multimodal User Interfaces* 13 (12 2019), 279-290. Issue 4. DOI: https://doi.org/10.1007/s12193-018-0264-4

[31] Thomas Hans Fritz, Samyogita Hardikar, Matthias Demoucron, Margot Niessen, Michiel Demey, Olivier Giot, Yongming Li, John-Dylan Haynes, Arno Villringer, and Marc Leman. 2013. Musical agency reduces perceived exertion during strenuous physical performance. *Proceedings of the National Academy of Sciences* 110, 44 (2013), 17784-17789.

[32] Shaun Gallagher. 2006. How the body shapes the mind. *Minds and Machines* 18 (9 2006), 413-415. Issue 3. DOI: https://doi.org/10.1007/s11023-008-9108-4

[33] Denise Gastaldo, Natalia Rivas-Quarneti, and Lilian Magalhães. 2018. Body-map storytelling as a health research methodology: Blurred lines creating clear pictures. *Forum Qualitative Sozialforschung* 19, 2 (2018). DOI: https://doi.org/10.17169/fqs-19.2.2858

[34] Alba Gomez-Andres, Jennifer Grau-Sánchez, Esther Duarte, Antoni Rodriguez-Fornells, and Ana Tajadura-Jiménez. 2020. Enriching footsteps sounds in gait rehabilitation in chronic stroke patients: A pilot study. *Annals of the New York Academy of Sciences* 1467 (5 2020), 48-59. Issue 1. DOI: https://doi.org/10.1111/nyas.14276

[35] Brianna L. Grant, Paul C. Yelder, Tracey A. Patrick, Bill Kapralos, Michael Williams-Bell, and Bernadette A. Murphy. 2019. Audiohaptic feedback enhances motor performance in a low-fidelity simulated drilling task. *Brain Sciences* 10, 1 (Dec 2019), 21. DOI: https://doi.org/10.3390/brainsci10010021

[36] Francesca Greco, Elisa Grazioli, Loretta Francesca Cosco, Attilio Parisi, Maurizio Bertollo, and Gian Pietro Emerenziani. 2022. The effects of music on cardiorespiratory endurance and muscular fitness in recreationally active individuals: A narrative review. *PeerJ* 10 (Apr 2022), e13332. DOI: https://doi.org/10.7717/peerj.13332

[37] Maria Hagströmer, Pekka Oja, and Michael Sjöström. 2006. The International Physical Activity Questionnaire (IPAQ): A study of concurrent and construct validity. *Public Health Nutrition* 9 (9 2006), 755-762. Issue 06. DOI: https://doi.org/10.1079/PHN2005898

[38] Kelly S. Hale and Kay M. Stanney. 2004. Deriving haptic design guidelines from human physiological, psychophysical, and neurological foundations. *IEEE Computer Graphics and Applications* 24, 2 (2004), 33-39. Issue 2. DOI: https://doi.org/10.1109/MCG.2004.1274059

[39] Rena Hale, Jerome G. Hausselle, and Roger V. Gonzalez. 2014. A preliminary study on the differences in male and female muscle force distribution patterns during squatting and lunging maneuvers. *Computers in Biology and Medicine* 52 (2014), 57-65. DOI: https://doi.org/10.1016/j.compbiomed.2014.06.010

[40] Daniel Harrison, Paul Marshall, Nadia Berthouze, and Jon Bird. 2014. Tracking physical activity: Problems related to running longitudinal studies with commercial devices. In *Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '14 Adjunct)*. ACM, New York, NY, 699-702. DOI: https://doi.org/10.1145/2638728.2641320

[41] Jasmin C. Hutchinson and Gershon Tenenbaum. 2007. Attention focus during physical effort: The mediating role of task intensity. *Psychology of Sport and Exercise* 8 (2007). Issue 2. DOI: https://doi.org/10.1016/j.psychsport.2006.03.006

[42] Felicity A. Baker, Imogen N. Clark, and Nicholas F. Taylor. 2016. The modulating effects of music listening on health-related exercise and physical activity in adults: A systematic review and narrative synthesis. *Nordic Journal of Music Therapy* 25, 1 (2016), 76-104. DOI: https://doi.org/10.1080/08098131.2015.1008558

[43] Ali Israr and Ivan Poupyrev. 2011. Tactile brush: Drawing on skin with a tactile grid display. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11)*. ACM, New York, NY, 2019-2028. DOI: https://doi.org/10.1145/1978942.1979235
[44] W. J. Trost, C. Labbé, and D. Grandjean. 2017. Rhythmic entrainment as a musical affect induction mechanism. *Neuropsychologia* 96 (2017), 96–110. DOI: https://doi.org/10.1016/j.neuropsychologia.2017.01.004

[45] Lucas Jaquet, Brigitta Danuser, and Patrick Gomez. 2014. Music and felt emotions: How systematic pitch level variations affect the experience of pleasantness and arousal. *Psychology of Music* 42 (2014). Issue 1. DOI: https://doi.org/10.1177/0305735612456583

[46] Patrik N. Juslin and Daniel Västfjäll. 2008. Emotional responses to music: The need to consider underlying mechanisms. *Behavioral and Brain Sciences* 31, 5 (2008), 559–621. DOI: https://doi.org/10.1017/S0140525X08005293

[47] Pavel Karpashevich, Pedro Sanches, Rachael Garrett, Yoav Luft, Kelsey Cotton, Vasiliki Tsaknaki, and Kristina Höök. 2022. Touching our breathing through shape-change: Monster, organic other, or twisted mirror. *ACM Transactions on Computer-Human Interaction* 29, 3 (Feb. 2022), 22:1–22:40. DOI: https://doi.org/10.1145/3490498

[48] Hayeon Kim and In-Kwon Lee. 2022. Studying the effects of congruence of auditory and visual stimuli on virtual reality experiences. *IEEE Transactions on Visualization and Computer Graphics* 28, 5 (2022), 2080–2090. DOI: https://doi.org/10.1109/TVCG.2022.3150514

[49] MyoungGon Kim, SungIk Cho, Tanh Quang Tran, Seong-Pil Kim, Ohung Kwon, and JungHyun Han. 2017. Scaled jump in gravity-reduced virtual environments. *IEEE Transactions on Visualization and Computer Graphics* 23, 4 (2017), 1360–1368. DOI: https://doi.org/10.1109/TVCG.2017.2657139

[50] Michel C. A. Klein, Adnan Manzoor, and Julia S. Mollee. 2017. Active2Gether: A personalized m-health intervention to encourage physical activity. *Sensors* 17 (2017). Issue 6. DOI: https://doi.org/10.3390/s17061436

[51] Dafna Kohn and Zohar Eitan. 2016. Moving music: Correspondences of musical parameters and movement dimensions in children's motion and verbal responses. *Music Perception* 34 (2016). Issue 1. DOI: https://doi.org/10.1525/mp.2016.34.1.40

[52] Yosuke Kurihara, Taku Hachisu, Katherine J. Kuchenbecker, and Hiroyuki Kajimoto. 2013. Virtual robotization of the human body via data-driven vibrotactile feedback. *Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)* 8253 LNCS. DOI: https://doi.org/10.1007/978-3-319-03161-3_8

[53] Guillaume Lemaitre, Hugo Scurto, Jules Françoise, Frédéric Bevilacqua, Olivier Houix, and Patrick Susini. 2017. Rising tones and rustling noises: Metaphors in gestural depictions of sounds. *PLoS ONE* 12 (2017). Issue 7. DOI: https://doi.org/10.1371/journal.pone.0181786

[54] Alice Wai Yi Leung, Ruth Suk Mei Chan, Mandy Man Mei Sea, and Jean Woo. 2019. Identifying psychological predictors of adherence to a community-based lifestyle modification program for weight loss among Chinese overweight and obese adults. *Nutrition Research and Practice* 13 (2019). Issue 5. DOI: https://doi.org/10.4162/nrp.2019.13.5.415

[55] Judith Ley-Flores, Eslam Alshami, Aaneesha Singh, Frédéric Bevilacqua, Nadia Bianchi-Berthouze, Ophelia Deroy, and Ana Tajadura-Jiménez. 2022. Effects of pitch and musical sounds on body-representations when moving with sound. *Scientific Reports* 12 (2022), 2676. Issue 1. DOI: https://doi.org/10.1038/s41598-022-06210-x

[56] Judith Ley-Flores, Frederic Bevilacqua, Nadia Bianchi-Berthouze, and Ana Taiadura-Jimenez. 2019. Altering body perception and emotion in physically inactive people through movement sonification. In *2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII '19)*. IEEE, Cambridge, UK, 1–7. DOI: https://doi.org/10.1109/ACII.2019.8925432

[57] Judith Ley-Flores, Laia Turmo Vidal, Nadia Berthouze, Aaneesha Singh, Frédéric Bevilacqua, and Ana Tajadura-Jiménez. 2021. SoniBand: Understanding the effects of metaphorical movement sonifications on body perception and physical activity. In *Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI '21)*. ACM, New York, NY, Article 521, 16 pages. DOI: https://doi.org/10.1145/3411764.3445558

[58] Fumiko Maeda, Ryota Kanai, and Shinsuke Shimojo. 2004. Changing pitch induced visual motion illusion. *Current Biology* 14 (2004). Issue 23. DOI: https://doi.org/10.1016/j.cub.2004.11.018

[59] Angelo Maravita and Atsushi Iriki. 2004. Tools for the body (schema). *Trends in Cognitive Sciences* 8 (2 2004), 79–86. Issue 2. DOI: https://doi.org/10.1016/j.TICS.2003.12.008

[60] Joe Marshall, Steve Benford, Richard Byrne, and Paul Tennent. 2019. Sensory alignment in immersive entertainment. In *Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19)*. ACM, New York, NY, 1–13. DOI: https://doi.org/10.1145/3290605.3300930

[61] Paul H. Mason. 2012. Music, dance and the total art work: Choreomusicology in theory and practice. *Research in Dance Education* 13, 1 (Apr 2012), 5–24. DOI: https://doi.org/10.1080/14647893.2011.651116

[62] Brian P. Meier and Michael D. Robinson. 2004. Why the sunny side is up: Associations between affect and vertical position. *Psychological Science* 15, 4 (Apr 2004), 243–247. DOI: https://doi.org/10.1111/j.0956-7976.2004.00659.x

[63] Fritz Menzer, Anna Brooks, Pär Halje, Christof Faller, Martin Vetterli, and Olaf Blanke. 2010. Feeling in control of your footsteps: Conscious gait monitoring and the auditory consequences of footsteps. *Cognitive Neuroscience* 1 (2010), 184–192. Issue 3. DOI: https://doi.org/10.1080/17588921003743581
[64] James Minogue and M. Gail Jones. 2006. Haptics in education: Exploring an untapped sensory modality. *Review of Educational Research* 76 (2006). Issue 3. DOI: https://doi.org/10.3102/00346543076003317

[65] S. A. Mudd. 1963. Spatial stereotypes of four dimensions of pure tone. *Journal of Experimental Psychology* 66 (1963). Issue 4. DOI: https://doi.org/10.1037/h0040045

[66] Gregory D. Myer, Adam M. Kushner, Jensen L. Brent, Brad J. Schoenfeld, Jason Hugentobler, Rhodri S. Lloyd, Al Vermeil, Donald A. Chu, Jason Harbin, and Stuart M. McGill. 2014. The back squat: A proposed assessment of functional deficits and technical factors that limit performance. *Strength and Conditioning Journal* 36 (2014). Issue 6. DOI: https://doi.org/10.1519/SSC.0000000000000103

[67] Elena Nava and Ana Tajadura-Jiménez. 2020. Auditory-induced body distortions in children and adults. *Scientific Reports* 2020 10:1 10 (2 2020), 1–14. Issue 1. DOI: https://doi.org/10.1038/s41598-020-59979-0

[68] Joseph Newbold, Nicolas E. Gold, and Nadia Bianchi-Berthouze. 2020. Movement sonification expectancy model: Leveraging musical expectancy theory to create movement-altering sonifications. *Journal on Multimodal User Interfaces* 14 (2020), 153–166. DOI: https://doi.org/10.1007/s12193-020-00322-2

[69] Joseph W. Newbold, Nadia Bianchi-Berthouze, and Nicolas E. Gold. 2017. Musical expectancy in squat sonification for people who struggle with physical activity. *Proceedings of the 23rd International Conference on Auditory Display—ICAD 2017*, 65–72. DOI: https://doi.org/10.21785/icad2017.008

[70] Nikou Nikmaram, Daniel S. Scholz, Michael Großbach, Simone B. Schmidt, Jakob Spogis, Paolo Belardinelli, Florian Müller-Dahlhaus, Jörg Remy, Ulf Ziemann, Jens D. Rollnik, and Eckart Altenmüller. 2019. Musical sonification of arm movements in stroke rehabilitation yields limited benefits. *Frontiers in Neuroscience* 13 (Dec 2019). DOI: https://doi.org/10.3389/fnins.2019.01378

[71] Romain Nith, Jacob Serfaty, Samuel G. Shatzkin, Alan Shen, and Pedro Lopes. 2023. JumpMod: Haptic backpack that modifies users’ perceived jump. In *Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23)*. ACM, New York, NY, Article 82, 15 pages. DOI: https://doi.org/10.1145/3544548.3580764

[72] World Health Organization. 2010. *Global Recommendations on Physical Activity for Health*. Retrieved from https://apps.who.int/iris/bitstream/handle/10665/44399/9789241599979_eng.pdf?sequence=1 http://www.ncbi.nlm.nih.gov/pubmed/26180873

[73] Jeffrey A. Potteiger, Jan M. Schroeder, and Kristin L. Goff. 2000. Influence of music on ratings of perceived exertion during 20 minutes of moderate intensity exercise. *Perceptual and Motor Skills* 91, 3 (2000), 848–854.

[74] Mária Rendi, Attila Szabo, and Tamás Szabó. 2008. Performance enhancement with music in rowing sprint. *The Sport Psychologist* 22, 2 (2008), 175–182. DOI: https://doi.org/10.1123/tsp.22.2.175

[75] Patricia Rick, Milagrosa Sánchez-Martín, Aneesha Singh, Sergio Navas-León, Mercedes Borda-Mas, Nadia Bianchi-Berthouze, and Ana Tajadura-Jiménez. 2022. Investigating psychological variables for technologies promoting physical activity. *Digital Health* 8 (2022). DOI: https://doi.org/10.1177/20552076221116559

[76] John Rooksby, Mattias Rost, Alistair Morrison, and Matthew Chalmers. 2014. Personal tracking as lived informatics. *Conference on Human Factors in Computing Systems — Proceedings*. DOI: https://doi.org/10.1145/2556288.2557039

[77] Elena Rusconi, Bonnie Kwan, Bruno L. Giordano, Carlo Umiltà, and Brian Butterworth. 2006. Spatial representation of pitch height: The SMARC effect. *Cognition* 99 (2006). Issue 2. DOI: https://doi.org/10.1016/j.cognition.2005.01.004

[78] Tomoya Sasaki, Kao-Hua Liu, Taiki Hasegawa, Atsushi Hiyama, and Masahiko Inami. 2019. Virtual super-leaping: Immersive extreme jumping in VR. *Proceedings of the 10th Augmented Human International Conference 2019* (2019). https://apisemanticscholar.org/CorpusID:71715388

[79] Nina Schaffert, Thenille Braun Janzen, Klaus Mattes, and Michael H. Thaut. 2019. A review on the relationship between sound and movement in sports and rehabilitation. *Frontiers in Psychology* 10 (2 2019). Issue FEB. DOI: https://doi.org/10.3389/fpsyg.2019.00244

[80] Brad J. Schoenfeld. 2010. Squatting kinematics and kinetics and their application to exercise performance. *Journal of Strength and Conditioning Research* 24 (12 2010), 3497–3506. Issue 12. DOI: https://doi.org/10.1519/JSC.0b013e3181bac2d7

[81] Roland Sigrist, Georg Rauter, Laura Marchal-Crespo, Robert Riener, and Peter Wolf. 2014. Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning. *Experimental Brain Research* 233 (2014), 909–925. Issue 3. DOI: https://doi.org/10.1007/s00221-014-4167-7

[82] Roland Sigrist, Georg Rauter, Robert Riener, and Peter Wolf. 2013. Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review. *Psychonomic Bulletin and Review* 20 (2013). Issue 1. DOI: https://doi.org/10.3758/s13423-012-0333-8

[83] Aneesha Singh, Stefano Piana, Davide Pollarolo, Gualtiero Volpe, Giovanna Varni, Ana Tajadura-Jiménez, Amanda CdeC Williams, Antonio Camurri, and Nadia Bianchi-Berthouze. 2016. Go-with-the-flow: Tracking, analysis and sonification of movement and breathing to build confidence in activity despite chronic pain. *Human-Computer Interaction* 31 (7 2016), 335–383. Issue 3-4. DOI: https://doi.org/10.1080/07370024.2015.1085310
[84] Daniel Spelmezan, Anke Hilgers, and Jan Borchers. 2009. A language of tactile motion instructions. MobileHCI09 — The 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. DOI: https://doi.org/10.1145/1613858.1613896

[85] Katta Spiel. 2021. The bodies of TEI-investigating norms and assumptions in the design of embodied interaction. TEI 2021 — Proceedings of the 15th International Conference on Tangible, Embedded, and Embodied Interaction. DOI: https://doi.org/10.1145/3430524.3440651

[86] Tasha R. Stanton, G. Lorimer Moseley, Arnold Y. L. Wong, and Gregory N. Kawchuk. 2017. Feeling stiffness in the back: A protective perceptual inference in chronic back pain. Scientific Reports 7 (2017). Issue 1. DOI: https://doi.org/10.1038/s41598-017-09429-1

[87] Tasha R. Stanton and Charles Spence. 2020. The influence of auditory cues on bodily and movement perception. Frontiers in Psychology 10 (1 2020). DOI: https://doi.org/10.3389/fpsyg.2019.03001

[88] Patrick Susini, Stephen McAdams, and Bennett K. Smith. 2007. Loudness asymmetries for tones with increasing and decreasing levels using continuous and global ratings. Acta Acustica United with Acustica 93 (2007). Issue 4.

[89] Ana Tajadura-Jimenez, Nadia Bianchi-Berthouze, Enrico Furfaro, and Frederic Bevilacqua. 2015. Sonification of surface tapping changes behavior, surface perception, and emotion. IEEE MultiMedia 22 (1 2015), 48–57. Issue 1. DOI: https://doi.org/10.1109/MMUL.2015.14

[90] Ana Tajadura-Jiménez, Laura Crucianelli, Rebecca Zheng, Chloe Cheng, Judith Ley-Flores, Mercedes Borda-Más, Nadia Bianchi-Berthouze, and Aikaterini Fotopoulou. 2022. Body weight distortions in an auditory-driven body illusion in subclinical and clinical eating disorders. Scientific Reports 2022 12:12, 1 (11 2022), 1–17. DOI: https://doi.org/10.1038/s41598-022-24452-7

[91] Ana Tajadura-Jiménez, Maria Basia, Ophelia Deroy, Merle Fairhurst, Nicolai Marquardt, and Nadia Bianchi-Berthouze. 2015. As light as your footsteps: Altering walking sounds to change perceived body weight, emotional state and gait. Conference on Human Factors in Computing Systems — Proceedings 2015-April (4 2015), 2943–2952. DOI: https://doi.org/10.1145/2702123.2702374

[92] Ana Tajadura-Jiménez, Helen Cohen, and Nadia Bianchi-Berthouze. 2017. Bodily sensory inputs and anomalous bodily experiences in complex regional pain syndrome: Evaluation of the potential effects of sound feedback. Frontiers in Human Neuroscience (2017). DOI: https://doi.org/10.3389/fnhum.2017.00379

[93] Ana Tajadura-Jiménez, Judith Ley-Flores, Omar Valdiviezo-Hernández, Aneesha Singh, Milagrosa Sánchez-Martín, Joaquín Díaz Duran, and Elena Márquez Segura. 2022. Exploring the design space for body transformation wearables to support physical activity through sensitizing and bodystorming. ACM International Conference Proceeding Series Par F180475 (6 2022). DOI: https://doi.org/10.1145/3537972.3538001

[94] Ana Tajadura-Jiménez, Torsten Marquardt, David Swapp, Norimichi Kitagawa, and Nadia Bianchi-Berthouze. 2016. Action sounds modulate arm reaching movements. Frontiers in Psychology 7 (9 2016). DOI: https://doi.org/10.3389/fpsyg.2016.01391

[95] Ana Tajadura-Jiménez, Joseph Newbold, Linge Zhang, Patricia Rick, and Nadia Bianchi-Berthouze. 2019. As light as you aspire to be: Changing body perception with sound to support physical activity. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–14. DOI: https://doi.org/10.1145/3290605.3300888

[96] Ana Tajadura-Jiménez, Manos Tsakiris, Torsten Marquardt, and Nadia Bianchi-Berthouze. 2015. Action sounds update the mental representation of arm dimension: Contributions of kinaesthesia and agency. Frontiers in Psychology 6 (5 2015). DOI: https://doi.org/10.3389/fpsyg.2015.00689

[97] Ana Tajadura-Jiménez, Maria Vakali, Merle T. Fairhurst, Alisa Mandrigin, Nadia Bianchi-Berthouze, and Ophelia Deroy. 2017. Contingent sounds change the mental representation of one’s finger length. Scientific Reports 7 (12 2017), 5748. Issue 1. DOI: https://doi.org/10.1038/s41598-017-05870-4

[98] Ana Tajadura-Jiménez, Aleksander Väljamäe, and Kristi Kuusk. 2020. Altering one’s body-perception through E-Textiles and haptic metaphors. Frontiers in Robotics and AI 7 (2020). DOI: https://doi.org/10.3389/frobt.2020.00007

[99] Ana Tajadura-Jiménez, Aleksander Väljamäe, Iwaki Toshima, Toshitaka Kimura, Manos Tsakiris, and Norimichi Kitagawa. 2012. Action sounds recalibrate perceived tactile distance. Current Biology 22 (7 2012), R516–R517. Issue 13. DOI: https://doi.org/10.1016/j.cub.2012.04.028

[100] Takumi Takahashi, Keisuke Shiro, Akira Matsuda, Ryo Komiyama, Hayato Nishioka, Kazunori Hori, Yoshio Ishiguro, Takashi Miyaki, and Jun Rekimoto. 2018. Augmented jump: A backpack multirotor system for jumping ability augmentation(ISWC '18). ACM, New York, NY, 230–231. DOI: https://doi.org/10.1145/3267242.3267270

[101] Paul Tennent, Joe Marshall, Vasiliki Tsaknaki, Charles Windlin, Kristina Höök, and Miquel Alfaras. 2020. Soma design and sensory misalignment. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). ACM, New York, NY, 1–12. DOI: https://doi.org/10.1145/3313831.3376812

[102] Peter C. Terry, Costas I. Karageorghis, Michelle L. Curran, Olwenn V. Martin, and Renée L. Parsons-Smith. 2020. Effects of music in exercise and sport: A meta-analytic review. Psychological Bulletin 146, 2 (2020), 91.
[103] Marcel Tiator, Okan Köse, Roman Wiche, Christian Geiger, and Fritz Dorn. 2018. Trampoline jumping with a head-mounted display in virtual reality entertainment. In *Intelligent Technologies for Interactive Entertainment*, Yoram Chisik, Jussi Holopainen, Rilla Khaled, José Luis Silva, and Paula Alexandra Silva (Eds.). Springer International Publishing, Cham, 105–119.

[104] Otis C. Trimble. 1934. Localization of sound in the anterior-posterior and vertical dimensions of “auditory” space. *British Journal of Psychology. General Section* 24 (1 1934), 320–334. Issue 3. DOI: https://doi.org/10.1111/j.2044-8295.1934.tb00706.x

[105] Manos Tsakiris and Patrick Haggard. 2005. Experimenting with the acting self. *Cognitive Neuropsychology* 22 (2005). Issue 3-4. DOI: https://doi.org/10.1080/02643290442000158

[106] Vasiliki Tsaknaki. 2021. The breathing wings: An autobiographical soma design exploration of touch qualities through shape-change materials. In *Designing Interactive Systems Conference 2021 (DIS '21)*. ACM, New York, NY, 1266–1279. DOI: https://doi.org/10.1145/3461778.3462054

[107] UPRIGHT. 2023. *UPRIGHT Posture Training Device*. Retrieved from https://www.uprightpose.com

[108] Elisabeth T. Kersten van Dijk, Joyce H. D. M. Westerink, Femke Beute, and Wijnand A. IJsselsteijn. 2017. Personal informatics, self-insight, and behavior change: A critical review of current literature. *Human-Computer Interaction* 32 (11 2017), 268–296. Issue 5-6. DOI: https://doi.org/10.1080/07370024.2016.1276456

[109] Bas Van Hooren, Jos Goudsmit, Juan Restrepo, and Steven Vos. 2020. Real-time feedback by wearables in running: Current approaches, challenges and suggestions for improvements. *Journal of Sports Sciences* 38, 2 (2020), 214–230.

[110] Daniel Västfjäll. 2002. Emotion in product sound design. *ResearchGate* (2002). Issue January.

[111] Camille K. Williams, Victrine Tseung, and Heather Carnahan. 2017. Self-control of haptic assistance for motor learning: Influences of frequency and opinion of utility. *Frontiers in Psychology* 8 (2017). DOI: https://doi.org/10.3389/fpsyg.2017.02082

Received 9 March 2023; revised 18 October 2023; accepted 10 November 2023