<figure><img src="image_14.png" /></figure>

Because these analyses of coreference relations arise both in concrete visual representations such as comics but also in abstract pictorial narratives such as that in (2), recent work has investigated whether coreference can be found in other visual representations beyond pictures. In particular, Patel-Grosz et al. (2018) has argued that there is coreference and disjoint reference in Bharatanatyam, a narrative Indian dance form. The authors found that reference to a same character involved in two different activities was marked by a fluid transition between the two positions referring to those activities. By contrast, disjoint reference was marked by a more structured sequence of intermediate dance positions marking the difference from the referent of the first dance position to the referent of the second dance position. Coreference seems, therefore, also visually encoded in abstract forms such as dance.

## 2.2 Coreference in music

If coreference relations are not exclusive to natural languages but also exist in sequences of pictures and sequences of dance positions, it becomes plausible that such relations can also be expressed beyond the visual modality, and in particular in audition.

Several examples from Schlenker (2022) suggest that genuine referential ambiguities can be found in music, and that conveying coreference can be achieved through different means, notably through orchestrations of pieces originally written for a single instrument, and through choreographies. For instance, in Chopin's Mazurka op. 33 n°2,<fnref n="3" /> a same melodic pattern of the form AB is repeated in A'B' with A = A' and B = B'.

Based solely on this structure, and under the hypothesis that two objects are here represented,<fnref n="4" /> at least two interpretations are possible: one object corresponds to AB and another one to A'B'; or one object corresponds to A and A', and another one to B and B'. But once one notices Chopin's dynamics indication on the score (circled in red) that A and B

<footnote n="3">³See for instance *this interpretation* by Arthur Rubinstein</footnote>

<footnote n="4">⁴As noted by Schlenker, this need not be the case.</footnote>
<figure><img src="image_15.png" /><figcaption>Figure 1: First bars of Chopin's mazurka Op. 33 n°2
Simplified version taken from the violin part of the arrangement for piano and violin by G. Adolfson (1870)
Retrieved from imslp.org</figcaption></figure>

have to be played (**forte**) (i.e. quite loud) and that A' and B' have to be played (**pianissimo**) (i.e. very soft), the second interpretation becomes unlikely.

In many orchestrations of this piece,<fnref n="5" /> AB and A'B' are assigned different groups of musical instruments; hence each passage is heard with a different timbre, reinforcing the disjoint reference between AB and A'B'. As discussed in Schlenker (2022), this disjoint reference was also made salient by visual means in Russian choreographer Michel Fokine's choreography on an orchestration of Chopin's Mazurka named *Les Sylphides* in which parts AB and A'B' involve different groups of dancers.

Together, these cases suggest that reference to objects can be found in music. After establishing how pitch and dynamics are semantically interpreted when paired to objects in section 3, we will then explore whether they can trigger referential effects in section 4 before introducing the special case of timbre in section 5.

<footnote n="5">⁵See Schlenker (2022) for a list of works involving similar orchestration choices, among which Benjamin Britten's, Roy Douglas's, Maurice Keller's, and Gordon Jacob's.</footnote>
# 3 Testing the semantics of pitch and dynamics

This section explores whether hypotheses about the semantic interpretation of pitch and dynamics as energy levels and distance of objects as posited in Schlenker (2017) can be experimentally confirmed. We present these hypotheses in the next subsections.

## 3.1 Theoretical framework

*Dynamics*. In Schlenker’s model, dynamics (referred to as ‘loudness’ in the work cited<fnref n="6" />) are argued to be an ambiguous musical property. Dynamics can either be interpreted as distance to an object from a perspectival center (**the louder the music, the closer the object**), or about its level of energy (**the louder the music, the more energy the object has**). These two assumptions are argued to come from normal (i.e. non-musical) auditory cognition. In our environment, a sound produced by an object is naturally perceived as louder when the object gets closer; and sound-producing objects that have high energy levels tend to produce louder sounds. In the first block of our experiment, we tested whether dynamics could indeed be semantically interpreted as either energy or distance.

*Pitch*. In the same model, when the nature of an object denoted by music is given, pitch is interpreted as the level of energy of that object: **the higher the pitch, the higher the level of energy of the denoted object**. Such a semantics for pitch is argued to be lifted from normal auditory cognition as well: generally speaking, things tend to produce higher-pitched or higher-frequency sounds when excited (Schlenker, 2017). Similarly, animal vocalizations and human voice seem to rely on high pitch to convey high level of emotional arousal or energy (Bachorowski, J. A., 2008; Seyfarth and Cheney, 2003). Although many other hypotheses on the interpretation of pitch have been experimentally tested in the literature,<fnref n="7" /> this specific interpretation in terms of energy has not. Finally, besides this interpretation of pitch as energy, and in order to have a symmetric paradigm involving as many stimuli manipulating pitch as stimuli manipulating loudness, we ran an exploratory test of the hypothesis according to which pitch could also be interpreted as distance, though not backed up by previous

<footnote n="6">⁶We prefer the term ‘dynamics’ to that of ‘loudness’ to get closer to music terminology: although strictly speaking, the main property co-varying with dynamics in music is indeed loudness (or volume), dynamics might have extra-dimensions to them (Olsen, n.d.)</footnote>

<footnote n="7">⁷For instance, Eitan (2013); Eitan and Granot (2004) on the mapping of pitch to a vertical position in space.</footnote>
theoretical work. The rule tested here was that **the higher the pitch, the closer the object.**

Note that in both cases, and although these inferential mechanisms might have been
lifted from normal auditory cognition, they seem to apply to silent objects just as much. For
instance, we still expect a louder sound to represent a silent object closer to us compared to
a softer one.

**3.2 Experimental design**

In this section, we first provide some experimental details that were common to all three
blocks of the experiment, before digging into the design of the first one.

**3.2.1 General experimental procedure**

The elements mentioned in this subsection hold for all three blocks of the experiment, which
was run on a pool of 60 participants recruited through the online platform Prolific©. All
participants first went through a few slides presenting the experiment and testing the audio
level to be fixed for the rest of the experiment. All of them gave online informed consent before
starting. The experiment had three blocks, the first of which was always the block testing for
the semantics of pitch and dynamics presented in this section. The order of the two subsequent
blocks, testing for pitch and loudness, and for timbre respectively, was randomized.<fnref n="8" /> The
experiment took between about 15 to 25 minutes to complete. Participants were paid through
the platform at an hourly rate of $9, based on an expected mean experimental time of 20
minutes.

An attention check was inserted after the experimental block 2 to make sure participants
were still listening and focused. It consisted of a video picked at random among all stimuli
with a voice instead of a musical stimulus indicating which response to give, and to type a
given word in a text box appearing on the following screen. Among all 60 participants, 5
failed the attention check and were therefore removed from the data. This criterion, along
with all hypotheses to be tested, the design of the experiment and the statistical analyses

<footnote n="8">⁸A post-hoc linear mixed-effect model revealed no significant effect of order on responses from block 2, be it for stimuli involving a change in dynamics (χ² = 0.4192, p = 0.5173)) or a change in pitch (χ² = 0.8188, p = 0.3655))</footnote>
scripts were preregistered prior to data collection. All files were archived as open data at this link.<fnref n="9" /> The preregistration is available here. In this paper, we present the three blocks separately for clarity.

All stimuli were created from a combination of visual and audio stimuli and available here. Audio stimuli were generated with Garageband© (version 10.3.5). The corresponding Garageband folder containing information on loudness levels and changes, timbre, duration and tempo is accessible in the same folder. Visual stimuli were generated with Open source software Blender© (version 3.4.1 2022-12-20) using the scripting interface allowing for programming the visual animations and easily changing its main properties. Each script used to generate the visual animations and to incorporate the audio stimuli is available in the corresponding folders for each block of the experiment.<fnref n="10" />

The following subsection provides details on the design and stimuli from the first experimental block testing for the semantics of pitch and dynamics.

### 3.2.2 Design and stimuli of block 1

The purpose of the first block of the experiment was to test whether participants endorsed the expected semantic interpretations derived from hypotheses in section 3.1. To that end, we created audiovisual stimuli<fnref n="11" /> in which six musical stimuli (a crescendo and a decrescendo - i.e. an increase and decrease in loudness, respectively -, a rise and a fall in pitch along a

<footnote n="9">⁹Any modification to the preregistered statistics, mainly pertaining to matters of (i) visualization, (ii) data presentation or (iii) the investigation of the statistical significance of some contrasts not predicted by the theory, which could not be anticipated from pilot experiments, are justified on the scripts.</footnote>

<footnote n="10">¹⁰While all information is available in the aforementioned documents, we here provide justification for some of the choices made in the design of the stimuli.</footnote>

<footnote>(i) *On the audio stimuli:* For the two 1st blocks of the experiment, in which timbre had to be constant, we used the timbre 'Soft Square Lead' available in the Garageband instruments library, which sounds close to a pure sound although created through square waves instead of sine waves. Two main reasons motivated this choice. First, although initially, we were planning on using pure sounds, pilots on informants suggested that it seemed to alter the naturality of the task, as the pure sounds were distracting. Second, we still wanted to work with a rare synthesized timbre to avoid personal or cultural associations made from well-known instruments timbres, and to avoid trivial associations with the musical instruments themselves rather than with external objects. Because it is reasonable to posit that participants were either never or only very rarely exposed to that very specific timbre, the peripheral associations with it were arguably interacting with the task less.</footnote>

<footnote>(ii) *On the audiovisual combinations:* One will note that while our visual stimuli were continuous scenes, our audio stimuli were discrete: the changes in dynamics or in pitch occurred between discrete notes, except for the 3rd block of the experiment. The reason for this was that it is arguably a good approximation of how musical meaning works: music is indeed a sequence of discrete musical events, while the events in the world that it represents are continuous.</footnote>

<footnote n="11">¹¹We were here inspired by several audiovisual tasks from previous research, which have been very fruitful to establish some relations between music and world properties in work such as Blumstein et al. (2012) where authors paired soundtracks with videos to test for the effect of musical non-linearities (such as abrupt frequency drops), or Sievers et al. (2019) who developed a computer program from which participants could use cursors to create a musical track from the video of a ball and vice versa.</footnote>
<figure><img src="image_16.png" /><figcaption>Table 1: Distribution of the stimuli across conditions in experimental block 1
(All stimuli can be accessed by clicking on the corresponding cell.)</figcaption></figure>

scale in major mode and in minor mode) were paired with either a congruent or incongruent visual among four visual animations (an object moving away, moving closer, gaining energy or losing energy). For instance, under the assumption that dynamics are interpreted as distance, and given a crescendo, the congruent stimulus involves an object moving closer, and the incongruent version would involve an object moving away.

**The congruent stimuli testing for loudness** involved either (i) a crescendo paired with an object approaching or gaining energy, or (ii) a decrescendo paired with an object either moving away or losing energy. **The incongruent stimuli testing for loudness** involved either (i) a crescendo paired with an object moving away or losing energy, or (ii) a decrescendo paired with an object either moving closer or gaining energy. **The congruent stimuli testing for pitch** involved either (i) an ascending scale in major or minor mode paired with an object gaining energy, or (ii) a descending scale in major or minor mode paired with an object losing energy. **The incongruent stimuli testing for pitch** involved the same scales paired with a change in energy in the opposite direction. Energy gain/loss was visually indicated through a shaking movement of the target object with a linearly increasing/decreasing amplitude. Although the stimuli used for illustration here only involve cubes, each stimulus also had a cylinder version in order to make sure that the evidenced effects, if any, were not specific to a specific 3D shape and to introduce variability in the stimuli. A summary of the design with accessible stimuli is available in Table 1.

For each trial, participants were presented with a pair of videos containing the same musical sound. One video was congruent with the predicted interpretation, and one was
incongruent with that interpretation. Participants were then asked to click on a button under the one they preferred,<fnref n="12" /> as shown in the screenshot of the experimental interface in Figure 2. The position of the congruent video on the screen was randomized. Trial order was fully randomized as well. The choice was binary so that participants had to choose one stimulus or the other.<fnref n="13" /> Our prediction was that the congruent video would be systematically preferred.

<figure><img src="image_17.png" /><figcaption>Figure 2: Experimental task - 1st experimental block</figcaption></figure>

## 3.3 Results

The results in Figure 3 confirm that the predicted (congruent) interpretations of both pitch and dynamics were significantly preferred compared to incongruent interpretations.<fnref n="14" /> An increase (or decrease, respectively) in dynamics was systematically judged as being more

<footnote n="12">¹²For a review of methods and a justification of the use of preference tasks in linguistics, which are particularly relevant to highlight contrasts compared to interpretation judgment tasks, see Ionin and Zyzik (2014)</footnote>

<footnote n="13">¹³While it could be argued that this could lead to possible biases, forcing a binary choice would in any case not show any preference for any of the two videos if there were none, as the choice would then be random.</footnote>

<footnote n="14">¹⁴For each musical property (pitch and dynamics), and each interpretation (as energy level or distance), we wanted to check whether participants chose the congruent interpretation significantly more than the incongruent one, i.e. that for each participant, the proportion of congruent stimuli was significantly higher than 50%. We therefore ran 4 one-sample t-tests. (i) For the distance interpretation of loudness: *t* = 4.3501, *df* = 104, *p* = 3.186e-05, (ii) for the energy interpretation of loudness, *t* = 3.5158, *df* = 107, *p*-value = 0.0006442, (iii) for the distance interpretation of pitch, *t* = 2.1689, *df* = 213, *p*-value = 0.03119 and (iv) for the energy interpretation of pitch, *t* = 6.2841, *df* = 214, *p*-value = 1.82e-09. We also ran linear mixed-effects model on the data for both properties separately: (i) for loudness, neither object type (χ² = 0.368, *p* = 0.5441)) nor the visual interpretation (χ² = 0.368, *p* = 0.5441)) had a significant effect on the mean endorsement of congruent stimuli picked, and (ii) for pitch, neither object type (χ² = 0.8526, *p* = 0.3558)) nor mode (major or minor) (χ² = 2.2412, *p* = 0.3261)) had a significant effect on the responses. However, the visual interpretation had a significant effect (χ² = 6.6422, *p* = 0.009959)) on the responses, indicating that the interpretation of pitch as energy was more easily accessible to participants than that as distance.</footnote>
congruent with an object moving closer (moving away, respectively), and of an increase (or decrease, respectively) in energy. This confirms that dynamics had indeed at least two available interpretations as both distance and energy level. Similarly, the results from pitch confirm our predictions: a rise in pitch (or fall, respectively) was systematically more associated with the object gaining energy (or losing energy, respectively). Neither the object type (cube/cylinder) nor the mode of the scale (major/minor) had any significant effect on the responses.

Interestingly, the exploratory test on the interpretation of pitch as distance, which was not backed up by theory, revealed a significant preference for a mapping from a rise in pitch to a decrease in object distance: the higher the pitch, the closer the associated object. Although this slight preference was found, the results finally show a significant interaction between the two available interpretations of pitch with a significantly higher endorsement rate for the interpretation as energy than for the interpretation as distance, suggesting that the interpretation of pitch as energy was more easily accessible.

<figure><img src="image_18.png" /><figcaption>Figure 3: Results from experimental block 1 on the interpretation of pitch and dynamics as distance and energy</figcaption></figure>
# 4 Testing coreference with pitch and dynamics

Having established the experimental reality of semantic interpretations of pitch and dynamics, we will now ask whether they can help establish reference to objects.

## 4.1 Experimental paradigm

The goal of this second block of the experiment was to determine whether dynamics and pitch could establish reference relations to objects. The theory tested here, referred to as the target theory hereafter, could be phrased as follows:

**Target theory:** Musical properties are associated with objects as soon as these objects satisfy the interpretation of these properties.

In other words, when one musical sound and two objects are present, reference is established between the musical sound and an object if that object satisfies the interpretation of the property.

### 4.1.1 Target stimuli

To clarify the design of the stimuli, let us derive the predictions from this target theory for this particular stimulus, which can be schematically and discretely represented as follows:

<table><caption>Figure 4: A discrete representation of an audiovisual stimulus</caption><thead><tr><th rowspan="2">Music</th><th colspan="2">Crescendo</th></tr><tr><th>Loudness -</th><th>Loudness +</th></tr></thead><tbody><tr><td>Visual</td><td><img src="target_object.png" alt="Target object: a white cube with an arrow pointing towards it, next to a control object: a white cube with an arrow circling it."></td><td><img src="control_object.png" alt="Control object: a white cube. The arrow in the control object's image points towards the white cube in the target object's image."></td></tr></tbody></table>

This audiovisual stimulus involves a sound with a crescendo (increasing loudness), constant pitch and constant timbre. Visually, it shows two white cubes that are identical in all respects, except for their position and movement. One, on the left, is moving closer to the observer at
constant speed and thus satisfies the crescendo (referred to as the target object hereafter),
and the other one, on the right, is spinning on itself (referred to as the control object).
The reason for having both objects move was to make sure that participants do not merely
associate the sound with the moving object, regardless of whether that object satisfies the
musical property.

Participants were asked to pick the object they thought the sound referred to. The
experimental interface looked like the screenshot displayed in Figure 5. In our example, the
prediction was that the cube on the left, moving closer and matching the crescendo, would
be preferably selected. All stimuli used for this second block of the experiment, accessible in
Tables 2 and 3, were designed in the same way as the example discussed above. The stimuli
testing for loudness involved either (i) a crescendo paired with a target object approaching
or gaining energy, or (ii) a decrescendo paired with a target object either moving away or
losing energy. The stimuli testing for pitch involved either (i) an ascending scale in major or
minor mode paired with a target object gaining energy, or (ii) a descending scale in major or
minor mode paired with a target object losing energy.<fnref n="15" /> Just as in the first block, energy
gain/loss was visually indicated through a shaking movement of the target object with a
linearly increasing/decreasing amplitude. The control object was always spinning. Just as for
the first block, each stimulus had a cube version and a cylinder version in order to make sure
that the evidenced effects, if any, were not specific to a specific 3D object and to introduce
variability in the stimuli. The position of the target object was randomized.

**4.1.2 Control stimuli**

In the event that, as predicted, a preference for the target object (the one satisfying the
musical property, here the cube on the left) was found, it would still be possible that this
preference is found merely due to (i) a salience effect (the cube moving closer is more salient
in the visual scene than the cube spinning) or (ii) a general preference for the behavior of
that object (maybe people just prefer objects that move towards them). This is the reason
why two conditions were introduced. Stimuli in the **target condition** consisted of scenes

<footnote n="15">¹⁵Note that in this second block, we did not test for the ability of pitch to establish reference to the target object through its interpretation as distance, for we considered that not enough theoretical background justified predictions in this regard</footnote>
<figure><img src="image_19.png" /><figcaption>Figure 5: Experimental task - 2nd experimental block</figcaption></figure>

with a musical sound changing with respect to one of the two tested properties (dynamics
or pitch): the stimulus represented in Figure 4 falls under this condition. Stimuli in the
**control condition** consisted of ambiguous scenes with a constant discrete musical sound
instead (neither changing in dynamics, nor in pitch).<fnref n="16" /> The experimental design for this
second experimental block with accessible stimuli is provided in Tables 2 and 3.

If our target theory is true, we predicted a greater preference for the target object in
the target condition than in the control condition. If not, then the prediction was that the
preference level for the target object should be the same in the target and in the control
conditions.

**4.2 Results**

As predicted, the results in Figure 6 reveal a significantly higher preference for the target
object in the target condition than in the control condition, for both interpretations of

<footnote n="16">¹⁶Note that the target and control *conditions* are different from the target and control *objects*: the difference between the conditions lies in the *sound* (changing/constant), while the difference in the objects lies in their movement (energy or distance change/spinning). All stimuli, in both the target and control conditions, involved a target and a control object.</footnote>
<figure><img src="image_20.png" /><figcaption>Table 2: Distribution of the stimuli across conditions in experimental block 2 - DYNAMICS (All stimuli can be accessed by clicking on the corresponding cell.)</figcaption></figure>

<figure><img src="image_21.png" /><figcaption>Table 3: Distribution of the stimuli across conditions in experimental block 2 - PITCH (All stimuli can be accessed by clicking on the corresponding cell.)</figcaption></figure>

dynamics (as distance and as energy).<fnref n="17" /> Although participants tended to prefer the target object even when a constant sound was heard, they picked the target object significantly more when it satisfied the musical sound.

<footnote n="17">¹⁷For loudness, a linear mixed-effect model revealed a significant effect of stimulus type (control vs target) ($\chi^2 = 73.928, p < 2.2e-16$)), and similarly for pitch ($\chi^2 = 13.658, p = 0.0002193$). As mentioned in the scripts, the simplification of the models made due to the preregistered models failing did not allow for testing the significance of the interaction between stimulus type (target/control) and visual interpretation (distance/energy) in the case of dynamics. Qualitatively, at least, this interaction is not visible.</footnote>
<figure><img src="image_22.png" /><figcaption>Figure 6: Results from experimental block 2 - dynamics</figcaption></figure>

The same pattern was found for pitch as seen in Figure 7: we found a significantly higher preference for the target object in the target condition, compared to the control condition.

<figure><img src="image_23.png" /><figcaption>Figure 7: Results from experimental block 2 - pitch</figcaption></figure>

Together, these results confirm our target theory that both pitch and dynamics establish reference relations to objects. In the target condition, one object only satisfied the semantics
of the sound, and that object was systematically judged as being the one the sound was
about. Importantly, this pattern can neither be explained by a mere salience effect of the
object, nor by the fact that it was the only one moving in the scene. Hence, we can conclude
that the visual representation of the target object and the musical sound are coreferential:
they both refer to a same object.

# 5 Testing coreference with timbre

This final section investigates how timbre behaves with regards to reference to objects. We will here argue that (i) timbre encodes reference to objects just as pitch and loudness, and that (ii) timbre makes it possible to show that reference relations hold between musical properties and *objects* (and not only between musical properties and *scenes* involving those objects, a distinction that the paradigm in Section 4 was unable to make).

## 5.1 Motivation

Although the results in section 4 confirmed our target theory, they seem to be unable to reject an alternative theory which would yield the same predictions. We will first present this alternative theory, and then explain how manipulations of timbre can be used to reject it.

**Alternative theory**

Musical properties are associated with *scenes* as soon as some objects within these scenes satisfy the interpretation of these properties.

This alternative theory predicts that as long as *some* part of the visual scene satisfies the sound property, the scene is considered felicitous. This does not entail a proper binding of musical properties and objects, but merely an association of musical properties to a weak notion of objects defined as parts of scenes. It is therefore impossible to assert from the previous results that pitch and loudness establish reference to objects strictly speaking, but only that they refer to scenes involving objects.

From an experimental standpoint, it was however not possible to reject the alternative theory using the paradigm described in 4 because the task was specifically to answer a question about reference to an object ('Which object does the tune refer to?'). But if our
alternative theory is true, and music rather refers to situations than objects, then it is possible
that participants simply responded by disregarding whatever in the scene did not match the
music (the control object), and hence reconstructed a notion of object from the scene. If
that is the case, then despite the contrasts found in 4, the results do not show that musical
properties referred to objects.

What was needed then was to design stimuli so that no part of the scene could be
disregarded. One way of doing so is to have both objects satisfy the musical property, so
that neither can be ignored. But if both objects satisfy the musical property, then both can
be denoted by that property (by definition), and the reference relation is lost: it would be
impossible to know which of the two objects is being referred to.<fnref n="18" /> Since it is not possible
to test reference to objects rather than to scenes with just one property and two objects
satisfying it, we reasoned that introducing a *second* property makes this possible, for it allows
both objects to satisfy a first property, and only one to satisfy a second property so as to
preserve reference.

This is where timbre comes into play. Although theoretically, any combination of two
musical properties should allow for different predictions of the two theories (such as dynamics
and pitch, for instance), timbre seemed like a particularly good candidate second property,
for it has been argued to be responsible for object identification (i.e. interpreted as the
nature of an object (Schlenker, 2022)). Our target theory applied to timbre therefore states
that timbre is associated with an object as soon as this object satisfies the interpretation of
timbre. Given that the interpretation of timbre is precisely the identity of the object it is
associated with, satisfying the interpretation amounts to having this association consistent
throughout the scene, and the prediction is that *Timbre is associated with an object as soon
as this association is consistent*. By contrast, the alternative theory states that timbre is
associated with a *scene* as soon as some object within that scene satisfies the interpretation
of timbre, which by the same reasoning makes the prediction that *Timbre is associated with
a scene as soon as some object is consistently present in that scene*.

<footnote n="18">¹⁸The same problem arises if we think about both objects satisfying the musical property at different times.</footnote>
Note that the main difference in the predictions of the two theories is that the target theory requires that the timbre/object association be consistent, while the alternative theory does not; it only requires that the scene consistently contains some object. We explain how we designed our stimuli based on this difference in the following section.

## 5.2 Experimental design

Each stimulus was designed as a scene involving two objects, each of which was assigned a given timbre during a pairing phase. The reason for this pairing phase was that to create differences in the timbre/object association consistency, it was first needed to create that association. It was established through the repetition of co-occurrences of forward movements satisfying a crescendo and paired with corresponding timbres. Each object was either paired with a cello timbre or an organ timbre (available in Garageband© instruments library). This choice was made so as to guarantee a sharp contrast and a clear identification of each timbre. The resulting scene is discretely represented in Figure 8.

<table><caption>Figure 8: Design and stimuli - 3rd experimental block</caption><thead><tr><th colspan="2">Audiovisual events</th><th>E<sub>1</sub></th><th>E<sub>2</sub></th><th>E<sub>3</sub></th><th>E<sub>4</sub></th><th>E<sub>5</sub></th></tr></thead><tbody><tr><td colspan="2">Visual</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td rowspan="2">Sound<br>Crescendo</td><td>match</td><td>Timbre 1</td><td>Timbre 2</td><td>Timbre 1</td><td>Timbre 2</td><td>Timbre 1</td></tr><tr><td>mismatch</td><td>Timbre 1</td><td>Timbre 2</td><td>Timbre 1</td><td>Timbre 2</td><td>#Timbre 2</td></tr><tr><td colspan="2"></td><td colspan="4">PAIRING PHASE</td><td>FINAL PHASE</td></tr></tbody></table>

At any point in time during the pairing phase (in events $E_1$ to $E_4$), one of the two objects satisfies the crescendo. In the final phase (event $E_5$), the last object to move forward always satisfies the crescendo as well. However, this object only satisfies the condition on timbre established during the pairing phase (consistency in the 'match' condition (an example is accessible here)). In the 'mismatch' condition, the last object to move was paired with a sound whose timbre had previously been associated with another object (inconsistency) (the corresponding stimulus violating the timbre/object association is accessible here).
If the alternative theory is true, i.e. if musical properties are associated with the scene as soon as some object in that scene satisfies these properties, then we predict that both kinds of stimulus should be equally good and that no preference should be found for either one: in both conditions, an object satisfies the final crescendo. The alternative theory does not make any prediction as to which of the two objects should satisfy this final crescendo because it does not require that timbre/object associations be consistent.

By contrast, if the target theory is true and the alternative theory is false, then stimuli in the match condition should be preferred because the object that satisfies the final crescendo also has to satisfy the condition on timbre. In other words, if timbre combined with dynamics does establish reference to objects and not merely to scenes, then the stimuli with coreference holding between both properties should be preferred.

Participants were presented with pairs of videos, one from the match condition, and the corresponding one from the mismatch condition. Audiovisual stimuli were identical in every respect except for the last event in the scene, which was paired with different timbres as shown in 8. Participants had to click on the video they preferred.<fnref n="19" /> The appearance of the experimental interface was very much similar to that shown in Figure 2. The position of each audiovisual stimulus was fully randomized; as was the position of the object moving first within the videos themselves. The stimuli were designed so that half of them involved the repetition of a same timbre in events *E*₄ and *E*₅ and half of them did not, to avoid biases from the repetition of a sound with a same timbre directly associated with another object. The order of stimuli was fully randomized as well.

## 5.3 Results

As predicted, the results in Figure 9 show a statistically significant preference for the coreferential stimuli over the non-coreferential ones.<fnref n="20" />. In other words, participants tended to prefer the interpretation under which the object/timbre pair remained constant over

<footnote n="19">¹⁹Note that the task had to change to test for reference to objects indirectly, because as mentioned earlier, asking participants about which object refers to already presupposes reference to object and does not allow for a dissociation in predictions between a music/scene and a music/object pairing.</footnote>

<footnote n="20">²⁰Just as for the first experimental block, we just performed a one-sample t-t-test to check whether coreferential stimuli were preferred significantly more than 50% of the time, which it did: *t* = 4.1087, *df* = 49, *p*-value = 0.0001508</footnote>
the interpretation under which that pairing changed. These results allow us to reject the
alternative theory: it is not the case that timbre is merely associated with scenes containing
objects that satisfy them; timbre *and* loudness are in this case associated with the objects
themselves. Therefore, the associations from music to objects evidenced in this paper are
more than just correlations to properties of scenes: they are reference relations to *objects*.
And because the musical sound referred to the same object as the object represented by the
visual object in the scene, we can conclude that both representations (musical and visual)
were coreferential.

<figure><img src="image_24.png" /><figcaption>Figure 9: Results from experimental block 3
Preference for coreferential over non-coreferential interpretation</figcaption></figure>

6 Conclusion

We can draw three conclusions from the previous results. First, it is indeed possible for pitch
and dynamics to be interpreted as energy levels and/or distance to the observer. We now
have experimental evidence that the three following rules hold:
1. The higher the pitch, the higher the level of energy of the object.

2. The louder the dynamics, the higher the level of energy of the object.

3. The louder the dynamics, the closer the object to the observer.

We also gathered significant data in favor of the following rule, which will yet have to be
further investigated, both theoretically and experimentally:

4. The higher the pitch, the closer the object to the observer.<fnref n="21" />

Second, we confirmed that it is possible for pitch and dynamics to determine reference to objects: both properties can help decide which object is being referred to by the sound.<fnref n="22" />

Third, we confirmed that timbre is able to establish reference as well when combined with other properties such as dynamics. Timbre also helped establish that reference relations hold between musical properties and *objects*, and not only between musical properties and *scenes* involving these objects.

In our experimental paradigm, we worked with minimal audiovisual stimuli that involved
very simplified visual scenes and very minimal musical stimuli. This choice was deliberate
as it allowed for greater simplicity in the stimuli and for better control. However, this begs
the question of (i) whether similar effects are found in music without visual animations, and

<footnote n="21">²¹Although this finding did was not backed up by theory, it is possible to speculate on a few reasons why we found a pitch height/distance mapping. The experimental results on associations between pitch height and depth or distance has mainly been inconsistent in the literature. While, contrary to our results, Eitan and Granot (2006) found that rises were associated with *increasing distance* (i.e. objects moving away), Eitan and Timmers (2010) did not find any correlation between high-pitch music and judgments of distance in listeners. One must however note that Eitan and Timmers (2010) did not test pitch changes but rather associations between pitch *range* and judgments of distance, while in our case, a crucial property of the sounds we tested was that they were changing in pitch. Our results are however consistent with other theories (discussed in Eitan (2013)) on the pitch/distance association modeled after the Doppler effect, which does predict that a rise in pitch should be associated with an increase in distance, which happens naturally when a sound source gets closer to a listener. They are also consistent with findings from Ghazanfar and Maier (2009), which established increased attention in Rhesus Monkeys when exposed to sounds rising in pitch, suggesting that a rise in pitch might be an important cue for detecting objects approaching in this species. Similar mechanisms seem to indirectly exist in humans: for instance, Ilie and Thompson (2006) show that low pitches are judged as more pleasant and trigger less arousal in listeners compared to high pitches. It might be that watching an object approach triggers some kind of alert system associated with higher energy arousal: the higher the pitch, the more threatening the object. From an evolutionary perspective, it might thus be that pitch is interpreted as a cue to object detection. However, one must note that the pitch/distance association was the weaker we found in the results. Although significant, this calls for caution regarding their interpretation. Further work will have to develop more fine-grained theories of pitch height/depth mappings to confirm whether our findings can be replicated, or whether this association might just be a byproduct of a more fundamental law instantiated in another space than the 3D space.</footnote>

<footnote n="22">²²Although we used an audiovisual paradigm throughout this experiment, one must note the main motivation for doing so was to limit noise: even though listeners do get spontaneous mental representations from music, music semantics is so abstract that many such representations are licensed given one musical passage or piece. This fundamental property of music semantics thus makes it hard to investigate subtle effects in which the number of possible associations with objects should be fixed. It is however likely that the role the visual modality played here can be filled by other modalities, or else by the auditory modality and other musical information just as much (for instance providing context, or limiting the number of possible denotations).</footnote>
(ii) whether our results generalize to cases of real music, which is far more complex than the minimal stimuli used in our experiment.

(i) Our use of visual animations paired with musical stimuli was motivated by the fact that music often licenses many mental scenes: to a same musical excerpt correspond many situations involving many different objects. Formal accounts of musical meaning have claimed that there are many situations that a same musical excerpt *is true of* (Schlenker, 2017, 2022). This results in musical meaning being very abstract: it is of course possible that different listeners spontaneously represent music in different ways, i.e. with different corresponding situations. But a central claim has been that all these situations do share some of their properties, and using visual animations is a way of making these situations *less* abstract: it is a methodological reason that motivates work with audiovisual scenes. Let us take an example from the stimuli used in this experiment: a crescendo. We have seen that a crescendo can be interpreted as an object approaching. We have already seen from the data that the results are insensitive to whether the object was a cube or a cylinder. Arguably, they would remain the same if it were any other object.

It is however another question whether they would be preserved without any visual cue. In the case of the first block, they probably would. If the assumption that listeners interpret music as being *about* something else than music itself is true, then it is likely that listeners have the ability to represent objects even in the absence of a visual anchor. In the case of the second block, the contribution of the visual scene was to force the representation of a scene where two objects were present instead of one. But if participants could consistently decide which of the two objects was being referred to, then it is highly likely that the same results would have been found whether three, four or more objects were present, as long as one object only satisfied the musical property. And if this happens with any number of objects, then it follows that regardless of the number of objects one posits, the reference to one object is preserved as long as that object matches the musical properties. In the third block, the contribution of the visual scene was to force an interpretation under which two objects were present. The results from this block showed that the stimuli where the timbre/object relation was consistent were preferred, which suggests that timbre might well
be interpreted as the identity of an object. This is consistent with the idea that there are
as many objects represented as there are timbres, which is in line with a principle stated
in Schlenker (2022) that ‘When two musical events have different timbre, they tend to be
not coreferential.’ It is therefore plausible that in the absence of any visual information,
listeners would still represent two objects when two different timbres are heard. However, the
point about the reference to objects could not have been made without a visual contribution
because music alone is more permissive than audiovisual scenes. Using audiovisual scenes
can thus help reveal mechanisms that (probably) exist in music alone (such as reference to
objects rather than scenes), but that are either hard or impossible to test in the absence of
any interaction with another representational system.

(ii) Under the assumption that our results hold for music alone, a final question as to
whether they generalize to real music remains. It is true that we worked with very simplified
music which hardly approximates the level of complexity of real music. However, we argue
that if such effects can be found in minimal stimuli, then they should be at least partially
preserved in more sophisticated music that makes use of similar patterns. Crescendos and rises
in pitch are very frequent in music, and although they might interact with other properties
that this experiment did not explore (such as harmonic structure, tempo, subtle articulations
or interpretative adjustments), there is no reason to believe that the semantic effects we
gave evidence for here should disappear. However, if one wanted to get closer to real music,
one way to do it is to reconstruct a musical passage through similar rules as the ones we
tested in Section 3: each combination of properties should come with its own set of rules. It
is possible that in real music, many more than two objects are often represented; and it is
also possible that some of the tendencies and rules we discussed here do not hold in certain
cases. Although the *scenes* that real music can be true of certainly have a greater level of
complexity, this is arguably independent from the fact that musical properties are mapped
onto objects and not merely onto scenes. This fundamental property of music to refer to
objects should therefore not be lost as music gets more complex; but further work should
examine whether this claim really holds in real music.
## Acknowledgments

This research received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 788077, Orisem, PI: Schlenker). Research was conducted at Institut d'Etudes Cognitives, Ecole Normale Supérieure - PSL University. Institut d'Etudes Cognitives is supported by grant FrontCog ANR-17-EURE-0017.

## Ethics

This research is part of the project review and approved by the ethics committee of Inserm, the Institutional Review Board (IRB00003888, IORG0003254, FWA00005831) of the French Institute of medical research and Health under Opinion number 20-733.
Bibliography

Abusch, D. (2013), ‘Applying Discourse Semantics and Pragmatics to Co-reference in Picture Sequences’, *Proceedings of Sinn und Bedeutung 17*, 9–25.

Abusch, D. (2017), The formal semantics of free perception in pictorial narratives., in 'Proceedings of the 21st Amsterdam Colloquium', a. cremers, t. van gessel, & f. roelofsen edn, pp. 85–96.

Abusch, D. (2020), Possible-Worlds Semantics for Pictures, in 'The Wiley Blackwell Companion to Semantics', John Wiley & Sons, Ltd, pp. 1–31. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781118788516.sem003.

Bachorowski, J. A. (2008), Vocal expressions of emotion., in 'Handbook of emotions, 3', pp. 196–210.

Blumstein, D. T., Bryant, G. A. and Kaye, P. (2012), ‘The sound of arousal in music is context-dependent’, *Biology Letters* 8(5), 744–747.

Clarke, E. (2001), ‘Meaning and the Specification of Motion in Music’, *Musicae Scientiae* 5(2), 213–234.

Cross, I. and Woodruff, G. E. (2009), Music as a communicative medium, in R. Botha and C. Knight, eds, ‘The Prehistory of Language’, Oxford University Press, pp. 77–98.

Eitan, Z. (2013), How pitch and loudness shape musical space and motion, in 'The psychology of music in multimedia', Oxford University Press, New York, NY, US, pp. 165–191.

Eitan, Z. and Granot, R. Y. (2004), Musical Parameters and Spatio-Kinetic Imagery, in 'ICMPC8 : proceedings of the 8th international conference on music perception & cognition: August 3-7, 2004 : Evanston, Illinois, USA', Causal Productions, pp. 57–63.

Eitan, Z. and Granot, R. Y. (2006), ‘How Music Moves’, *Music Perception* 23(3), 221–248.

Eitan, Z. and Timmers, R. (2010), ‘Beethoven’s last piano sonata and those who follow crocodiles: Cross-domain mappings of auditory pitch in a musical context’, *Cognition* 114, 405–422. Place: Netherlands Publisher: Elsevier Science.

Gabrielsson, A. and Lindström, E. (2010), The role of structure in the musical expression of emotions, in 'Handbook of music and emotion: Theory, research, applications', Series in affective science, Oxford University Press, New York, NY, US, pp. 367–400.

Ghazanfar, A. A. and Maier, J. X. (2009), ‘Rhesus monkeys (Macaca mulatta) hear rising frequency sounds as looming.’, *Behavioral Neuroscience* 123(4), 822–827.

Godøy, R. I. and Leman, M., eds (2010), *Musical gestures: sound, movement, and meaning*, Routledge, New York. OCLC: ocn298781501.

Ilie, G. and Thompson, W. F. (2006), ‘A Comparison of Acoustic Cues in Music and Speech for Three Dimensions of Affect’, *Music Perception* 23, 319–329. Place: US Publisher: University of California Press.

Ionin, T. and Zyzik, E. (2014), ‘Judgment and Interpretation Tasks in Second Language Research’, *Annual Review of Applied Linguistics* 34, 37–64. Publisher: Cambridge University Press.

Juslin, P. N. and Laukka, P. (2003), ‘Communication of emotions in vocal expression and music performance: Different channels, same code?’, *Psychological Bulletin* 129(5), 770–814.

Karttunen, L. (1969), Pronouns and Variables., R. I. Binnick, A. Davison, G. M. Green, and J. L. Morgan, Chicago: Department of Linguistics, University of Chicago. Reprinted in J.
D. McCawley, ed., *Syntax and Semantics 7: Notes from the Linguistic Underground*. New York: Academic Press, 1976.

Koelsch, S. (2012), *Brain and music*, Wiley-Blackwell, Chichester, West Sussex; Hoboken, NJ. OCLC: ocn767563922.

Larson, S. (2012), *Musical forces: motion, metaphor, and meaning in music*, Musical meaning & interpretation, Indiana University Press, Bloomington. OCLC: ocn707212791.

Montague, R. (1973), *The Proper Treatment of Quantification in Ordinary English*, in 'Approaches to Natural Language.', P. Suppes, J. Moravcsik, and J. Hintikka, *Reidel: Dordrecht*.

Olsen, K. N. (n.d.), 'Intensity dynamics and loudness change: a review of methods and perceptual processes', *Acoustics Australia*.

Patel-Grosz, P., Grosz, P. G., Kelkar, T. and Jensenius, A. R. (2018), 'Coreference and disjoint reference in the semantics of narrative dance', *Proceedings of Sinn und Bedeutung 22*(2), 199–216. Number: 2.

Schlenker, P. (2017), 'Outline of Music Semantics', *Music Perception* 35(1), 3–37.

Schlenker, P. (2022), 'Musical meaning within Super Semantics', *Linguistics and Philosophy* 45(4), 795–872.

Schlenker, P., Bonnet, M., Lamberton, J., Lamberton, J., Chemla, E., Santoro, M. and Geraci, C. (2023), 'Iconic Syntax: Sign Language Classifier Predicates and Gesture Sequences'. *LingBuzz Published In: To appear in Linguistics & Philosophy*.

Seyfarth, R. M. and Cheney, D. L. (2003), 'Meaning and Emotion in Animal Vocalizations', *Annals of the New York Academy of Sciences* 1000(1), 32–55. _eprint_:
https://onlinelibrary.wiley.com/doi/pdf/10.1196/annals.1280.004.

Sievers, B., Lee, C., Haslett, W. and Wheatley, T. (2019), 'A multi-sensory code for emotional arousal', *Proceedings of the Royal Society B: Biological Sciences* 286(1906), 20190513.

Sievers, B., Polansky, L., Casey, M. and Wheatley, T. (2013), 'Music and movement share a dynamic structure that supports universal expressions of emotion', *Proceedings of the National Academy of Sciences* 110(1), 70–75.
# Chapter 4

## Meaning from music alone

### Purpose

In the previous chapter, we saw that investigating the interaction between musical sounds and visual perception reveals interesting properties of music semantics. In this chapter, we move away from paradigms using music in combination with language or visual perception, and explore a pure version of musical meaning through a case study of music evoking a particular instance of body motion: walking. Our findings show that investigating such specific case studies help reveal some further properties of music semantics.

This chapter contains two articles. The first one, which was published in the Proceedings of the 22nd Amsterdam Colloquium (2019) [reference provided in the paper], explores theoretical implications of walk-denoting music. The second article presents the detailed results from an experiment aiming at testing the predictions from the theoretical paper. Most of the theoretical content from the first paper is explained in the second paper as well.
Léo Migotti

Institut Jean Nicod
ENS-EHESS-CNRS

Léo Zaradzki

Laboratoire de Linguistique Formelle
Université Paris Diderot

ABSTRACT

Music has recently been argued to have a referential semantics (Schlenker, 2017, 2019), i.e. to trigger inferences about an extra-musical reality. In this view, because the set of possible denotations is often very large, the meaning of music is often very abstract. Here we consider a very particular kind of musical sequences, which we call walk-denoting as they strongly evoke walking-situations — namely situations in which at least one character is walking. We show that the current model for music semantics is doubly insufficient. First, it makes wrong predictions with respect to the considered musical snippets. Using the method of minimal pairs, we come up with an enhanced model that accounts for inferential differences that the previous one left aside. Second, it relies on the non-trivial assumption that all notes are interpreted as events, while alternative theories seem to be just as plausible. Because a rewriting of our prototypical musical snippet adding a quaver did not seem to affect the denotation, the possibility that some musical events denote nothing needs to be investigated. Finally, we sketch the overall theoretical landscape through two main theories, which either consider that all musical events are interpreted, or that some of them might not be.

<footnote>*This paper was published in the Proceedings of the 22nd Amsterdam Colloquium:
Migotti, L. Zazadzki, L. (2019). *Walk-denoting music: refining music semantics*. In Julian J. Schlöder, J.J., McHugh,
D. Roelofsen, F. (eds), Proceedings of the 22nd Amsterdam Colloquium, pp. 593-602. [link]</footnote>
# Contents

<table><tr><td>1</td><td>Introduction</td><td>116</td></tr><tr><td>2</td><td>A formal model for music semantics</td><td>116</td></tr><tr><td>3</td><td>Walk-denoting music and walking-situations</td><td>118</td></tr><tr><td>4</td><td>Theoretical refinements</td><td>121</td></tr><tr><td>4.1</td><td>Hierarchical structures</td><td>121</td></tr><tr><td>4.2</td><td>Do all notes have the same semantic status?</td><td>124</td></tr><tr><td>4.3</td><td>The meaning of uninterpreted notes</td><td>127</td></tr><tr><td>4.4</td><td>Is every event musically represented?</td><td>128</td></tr><tr><td>5</td><td>Conclusion</td><td>128</td></tr><tr><td></td><td>Bibliography</td><td>130</td></tr></table>
# 1 Introduction

Recent investigations about the application of formal linguistics methods to non-linguistic objects such as music strongly suggest that music can convey information about the world through semantic rules that bridge the characteristics of music and the ones of what it can evoke or represent. While evidence has been provided regarding the interpretation of some musical features from a purely semantic perspective (Schlenker, 2017, 2019), little attention has been given to the systematic link that exists between the internal structure of music, i.e. its syntax, and the information it conveys, i.e. its semantics. Yet, we know that both music and the situations evoked can be represented hierarchically (Jackendoff, 2009; Schlenker, 2019). It then seems to make much intuitive sense to posit that if music has a semantics, its syntax, in relation with that of the denoted situation, has to play a role as well. We first present Schlenker's theory of musical semantics. We discuss a case-study about musical snippets evoking walking-situations, and we then highlight some of the limitations of Schlenker's model. We argue that Schlenker's theory lacks conditions on the rules linking music and situations structures. We finally present two possible theoretical accounts of this relationship. We will keep the choice to be made for further theoretical and experimental research.<fnref n="2" />

# 2 A formal model for music semantics

Because music can evoke or make us think about certain events, be they real or not, and describe some situations better than others, Schlenker (2017, 2019) argued that it must have a semantics. Indeed, music does not only convey information about its form and its internal structure, it also conveys information about an extra-musical reality: some music evoke sad or happy situations, others might well describe a landscape or an animal. The set of all the hypothetical situations a music can appropriately describe is therefore taken to be its

<footnote n="2">²We would like to deeply thank Philippe Schlenker for providing us with reliable introspective judgments as well as crucial theoretical insights, and for reviewing this article. We also thank Emmanuel Chemla who helped us clarifying our theoretical hypotheses. Thank you also to all our informants for taking the time to describe the detailed spontaneous inferences they drew from our musical stimuli.</footnote>
meaning. The following section draws from Schlenker (2017, 2019) and presents the concepts,
terminology and notation that are needed for our theoretical proposals in section 4.

The first core idea we rely on is that music is able to convey information about the world
because certain musical parameters such as timber, pitch, loudness or harmonic stability
(among many others) are semantically interpreted: each of them bears some of the meaning of
music (Schlenker, 2017, 2019). For instance, pitch might well provide information regarding
the size of a character involved in a scene that is depicted by the music. From now on we
will talk about *inferences* to refer to this information music provides and listeners get. Also,
we will call what these inferences are about *virtual sources*<fnref n="3" />.

Because music shares many features with sound, but cannot be reduced to it, the musical
parameters responsible for the triggering of inferences can be split into two main categories.
The first category gathers the parameters music and sound have in common, which makes it
possible to derive semantic rules from normal auditory cognition. Loudness is of this sort:
as any sound in nature has a certain level of loudness, we know from our world experience
that loudness can be linked to certain properties of the actual source of the sound, arguably
either its distance to the listener, or its level of energy. Applying this to music, we get a
semantic rule on loudness interpretation, according to which the loudness of any musical
event is ambiguous and either interpreted in terms of distance or energy of the *virtual* source.
The second category gathers parameters that have no trivial counterpart in the non-musical
world, and are intrinsically linked to tonal and harmonic properties of music. For instance,
the harmonic stability of a given chord in a given key follows from tonal rules. As we do not
experience harmonic stability of non-musical events, we cannot derive a semantic rule from
auditory cognition; rather, Schlenker argues that this parameter is interpreted as the *actual*
stability of the source, or that of the emotional state in which the listener is put.

The second core idea is that our music semantics needs to state rules linking the musical
parameters to their semantic interpretation. From now on, we will use ‘musical event’ to refer
to any note or chord, and ‘denoted situation’ to refer to any complex situation pertaining

<footnote n="3">³Their naming must not, however, confuse us about their nature: virtual sources are not actual sources of the sound; they are virtual objects that may or may not produce sounds — if they do, the music does not even need to match this sounds — involved in the denotation of music.</footnote>
to the set of situations a musical snippet can evoke. Formally, we define, just as Schlenker, a musical snippet as an *n*-tuple $M = (m_1, ..., m_n)$, and a possible situation *S* as an *n*-tuple *S* = (*e*₁, ..., *e*ₙ), where (*m*₁, ..., *m*ₙ) is the succession of notes, each of which represents the corresponding event carrying the same index in the situation.

In formal linguistics, the common view is that the meaning of a sentence is the set of all situations of which the sentence is true. Transposing this to music, the meaning of *M* is the set of all situations which *M* is true of. We thus need a notion of musical truth. We say that *M* denotes *S* (*M* ⊨ *S*) if *S* is one — among many others — possible denotations for *M*. The final step is therefore to find rules to compute the truth-value of a music, given a specific situation it might denote. Schlenker posits that those rules are order-preservation rules: for *M* to denote *S*, musical parameters involved in each event in *M* need to be ordered in the same way as their interpretation in the denotation. For instance, loudness levels and corresponding levels of energy or distances from the listener must be ordered in the same way.

Although the above model makes clear intuitive sense, we argue that it makes wrong predictions regarding the possible denotations of some musical snippets. Specifically, we claim that:

1. It makes incorrect predictions regarding the possible denotations of some specific musical snippets we think trigger strong inferences about a virtual source walking, as shown in next section.

2. It relies on the assumption that each musical event is systematically interpreted, regardless of its structural role, while it seems reasonable to posit that some musical events are more important than others.

**3 Walk-denoting music and walking-situations**

As stated, the above theory fails to account for some strong inferences we believe to be
triggered by the prototypical musical snippet about walking-events in Figure 1⁴.

<footnote n="4">⁴audio file: https://www.youtube.com/watch?v=BWbqZ1BiRzI&feature=youtu.be
All scores are directly clickable to access the audio file.</footnote>
A walking-situation is a situation in which at least one of the virtual sources is walking. A walk-denoting music is a musical snippet that can denote a walking-situation. For levels of stability to match, and a music to denote a walk, we thus need to have a musical event to walking-event matching as shown in Figure 1: bass notes represent footsteps, while second and fourth chords represent the ‘bounces’ occurring during the transition from one foot to the other.

Based on our own introspective judgments, as well as on that of informants, we argue that the music contained in Figure 1 triggers very strong inferences about walking-situations. One might argue that listeners get these inferences because they are constantly experiencing walking-situations. This argument does not explain, however, the existence of contrastive judgments between musical snippets which, based on Schlenker’s model, should all be able to denote a walking-situation, while the inferential judgments we got from our informants do not match the theoretical predictions from the model. Let us consider the score in Figure 2⁵.

In order to compute the possible denotations of this piece, we first need to understand what the meaningful parameters are, both in the music itself, and in the virtual walking-situation. Intuitively, we argue that the most prominent parameter which is involved in a walking-situation and varies throughout it is stability: each footstep appears to be a quite stable event, while the transition from one foot to the other are relatively less stable, be it only because a foot is lifted in the air. Thus, the corresponding musical parameter *in music* must be harmonic stability. Based on rules of preservation of ordering, the theory predicts that music in Figure 2 must be able to denote a walking-situation, which it does not, according to our introspective judgments. Thus, we need to refine the formal theory that made this incorrect prediction possible.

<figure><img src="image_25.png" /><figcaption>Figure 1: Prototypical walk-denoting music</figcaption></figure>

<footnote n="5">⁵audio file: https://www.youtube.com/watch?v=SvmXWX_xAeU&feature=youtu.be</footnote>
<figure><img src="image_26.png" /><figcaption>Figure 2: A music failing to denote a walking-situation</figcaption></figure>

<figure><img src="image_27.png" /><figcaption>Figure 3: Violating alternation condition</figcaption></figure>

<figure><img src="image_28.png" /><figcaption>Figure 4: Prototype with extra quavers</figcaption></figure>

In order to do so, we varied a whole set of musical parameters once at a time, and selected the most relevant ones according to a few informants. This led us to hypothesise that, for a musical snippet to denote a walking-situation, it has to involve the steady repetition of two different chords, that are both intrinsically stable, and sufficiently close to each other in the tonal space.

A way to test whether this set of conditions is accurate was then to build minimal pairs, i.e. couples of stimuli made of the above prototype and a composed musical snippet based on this very prototype but violating one of the five above conditions. Our prediction is thus that violating any of the conditions would trigger an inferential preference for the prototypical snippet, that satisfies all conditions<fnref n="6" />.

From a theoretical perspective, it appears that these five parameters can be classified in two groups that make cognitive sense, and that can be derived from theoretical considerations. As a walk itself is defined as the alternation of two footsteps, a first natural class of conditions follows from the fact that any music that denotes a walking-situation must also be composed of exactly two events. The second and third conditions on repetition and regularity can be derived from the same physical fact: a normal, stereotypical walk is necessarily the repetition of footsteps (which explains why the musical events must themselves be repeated), and that repetition needs to be approximately symmetric (which explains why the repetition of musical events shall never be broken and remain steady). A second class of conditions appears to be linked to the fact that the right footstep is necessarily different from the left one, but that

<footnote n="6">⁶In order to check our theoretical intuitions, we are currently running an experiment which aims at checking whether these conditions actually play a role in the triggering of inferences about walking-situations in listeners, by presenting participants the minimal pairs that are available at https://www.youtube.com/watch?v=04Puddu3wXQ&feature=youtu.be.</footnote>
both events are not so different and are also both rather stable, although one might be a
little bit more stable than the other; thus, the corresponding two musical events must be
minimally different as well.

These conditions being stated, we were however concerned with the music snippet in
Figure 4⁷. Indeed, introspective judgments given by our informants as well as our owns
suggested that the denotation of this rewritten version of the prototype, in which a quaver
was added on each offbeat, was not affected; or that is was affected in a very subtle way, that
did not correspond to a radical change in the denotation or a new event. We thus wondered
how this could be the case, from a theoretical perspective.

# 4 Theoretical refinements

In addition to showing that Schlenker's interpretive rules are incorrect or at least underspecified, walk-denoting music examples raise one fundamental question. The example in Figure 4 suggests that some notes may not be interpreted as concrete events in the denoted situation<fnref n="8" />.

If this is so, then we have to answer two new questions:

1. If a note is not interpreted as a concrete event, what is its semantic role?

2. How do we determine which notes must be interpreted and which may remain uninterpreted?

In the rest of the article, we discuss theoretical issues related to these questions and present
competing theories of musical meaning.

## 4.1 Hierarchical structures

We know from Lerdahl and Jackendoff (1983) that there are ways to assign hierarchical
structures — *e.g.* tree-like structures — to musical pieces. Many different views address the

<footnote n="7">⁷audio file: https://www.youtube.com/watch?v=gR-wHm4nFYk&feature=youtu.be</footnote>

<footnote n="8">⁸While this is a constructed example, examples exist in music. To give only a famous one, in the opening of *Peter and the Wolf* by S. Prokofiev, after the first occurrence of Peter's theme in first violins, it is played again by the seconds, while the firsts play high-pitched offbeats. We believe that it does not add any event to the denoted situation (e.g. Peter gamboling through the meadows) but only gives it a more carefree character. The example can be heard here: https://www.youtube.com/watch?v=uwKgH8QH_mc&t=2m39s.</footnote>
question of which structure is the best to account for musical hierarchy and dependencies,
such as time-span reduction or prolongational reduction from Lerdahl and Jackendoff (1983).
From a different perspective, Harasim, Rohrmeier and O'Donnell argue that the internal and
harmonic structure of any given tonal piece can be accounted for with the three notions of
prolongation, preparation and substitution (Harasim et al., 2018). On the other hand, it has
been proposed that events too feature a hierarchical structure that can be represented as a
tree (Jackendoff, 2009).

In a famous series of experiments where they asked subjects to segment taped common
situations in sub-units, Zacks, Tversy and Iyer showed that people conceive situations as
partonomic hierarchies (Zacks et al., 2001). In particular, people form mental groups of events
that recursively embed into one another, often in a goal-directed fashion. On the musical
side, Lerdahl and Jackendoff gave formal rules derived from Gestalt principles to determine a
so-called *grouping structure* for musical pieces. This grouping structure is exactly of the same
partonomic kind as those evidenced by Zacks and colleagues. For example, Figure 5 (taken
from Lerdahl and Jackendoff (1983)) shows the grouping structure that the system derives,
in accordance with listeners' common intuitions, for the opening theme of Mozart's 40th
Symphony. As shown by the curly brackets, the first three notes form a first group together,
and the next three form another group. These two groups are then grouped together at a
higher level to form a new group, and so on.

Elaborating on ideas already suggested in Schlenker's work, we posit that for a musical snippet *M* to denote a situation *S*, the grouping structures of both must match, in the following sense: a group on the situation side must not contain a group boundary on the musical side. More precisely, if *e*₁, ..., *e*ₖ and *a* are events of *S* associated in *M* with musical events *m*₁, ..., *m*ₖ and *b*, and if there is a group to which *e*₁, ..., *e*ₖ belong but *a* does not belong, then there must be no musical group to which *b* together with some but not all the *m*ᵢ

<figure><img src="image_29.png" /><figcaption>Figure 5: A possible grouping structure for the opening theme of Mozart's 40th Symphony.</figcaption></figure>
<figure><img src="image_30.png" /><figcaption>Figure 6: Example of grouping mismatch.</figcaption></figure>

<figure><img src="image_31.png" /><figcaption>Figure 7: Example of tree embedding</figcaption></figure>

belongs. As an illustration of this phenomenon, let us take the opening motif of Beethoven’s 5th Symphony. This consists in four notes: G G G E. According to Lerdahl and Jackendoff’s rules<fnref n="9" /> they are grouped as [[G, G, G], E]. In the corresponding section of *Fantasia 2000*<fnref n="10" /> each note is illustrated by a coloured lightning: the Gs are interpreted as blue lightnings, and the E as an orange one. It seems to us, though, that image and sound fit far less well if we colour, say, the third lightning in orange too. As shown in Figure 6, this is because the orange group would then contain the G/E boundary.

While we believe that preserving the hierarchical structures is a rather natural requirement for the interpretive rules, we will not argue further here, and leave this study for future research. We here assume the following strong version of preservation. First, we posit that musical snippets are associated with a hierarchical structure which is mathematically implemented as a directed rooted tree, possibly with vertices labeled as heads at each level<fnref n="11" />. Such a modeling is compatible with many approaches to musical syntax, either based on grouping structures or rather on harmony. Thus, we do not need to commit to any particular formal system here. Second, in line with Jackendoff (2009), we posit that situations too are associated with a hierarchical structure implemented as a directed rooted tree with heads. While this is a stronger assumption than what had been experimentally proved by Zacks et al., we take it as a working hypothesis. Third, we claim that a necessary condition for a musical snippet *M* to denote a situation *S* is that the tree of *S* can be embedded into the tree of *M*. Formally, if *M* is a tree (*V*, *E*) and *S* is a tree (*V'*, *E'*), where *V* and *V'*

<footnote n="9">⁹and also to general Gestalt principles</footnote>

<footnote n="10">¹⁰an animated film where image is intended to be a denotation of the music; see: https://www.youtube.com/watch?v=nMnlxYkZKaU</footnote>

<footnote n="11">¹¹We discuss this notion of heads in more details below. At first glance, let us say that heads are musical events that are more prominent or structurally more important than non-heads.</footnote>
represent the sets of vertices and *E* and *E'* represent the sets of edges<fnref n="12" />, a necessary condition for *M* |= *S* is the existence of an injective root-preserving function *f*: *V'* → *V* such that: ∀*x*,*y* ∈ *V*' , (*x*,*y*) ∈ *E'* ⇒ *f*(*x*) ∼ *f*(*y*), where *f*(*x*) ∼ *f*(*y*) means that there exists a path in *M* from *f*(*x*) to *f*(*y*). What *f* does is that it takes any event of *S* and maps it onto a musical event meant to be its musical representation, in an injective way. Moreover, if an event of *S* is subordinated to another, then the same subordination relationship should hold between their musical representations. Figure 7 shows an example of such a function.

4.2 Do all notes have the same semantic status?

We now turn to the question of uninterpreted notes. The discussion from the end of section 3
suggested that all notes of a musical snippet are not necessarily interpreted as events in the
denoted situation. While we do not have data to decide whether or not this can be, we will
here present two competing positions about it.

In Figure 4 we saw that in the walk-denoting case, bass notes were interpreted by steps
and other beats by bounces; as for the additional offbeats, it was not clear. What we can say
is that most important musical events are bass notes, the other beats are musical ‘bounces’
of these bass notes, and offbeats are kind of squared bounces (bounces’ bounces). Thus,
most important musical events are interpreted as important events in the situation too. This
suggests that the musical salience of notes plays a role. This is why we added heads to our
tree-implementations: heads are special vertices meant to represent the most salient events
of a musical snippet or a situation. We will examine below alternative ways of implementing
salience and how it can be computed. Now we sketch two informal opposite theories, to be
refined below:

* **Theory A** (strong¹³ theory): Every note is necessarily interpreted in the situation — though maybe by a very abstract event — that is, every note matches to an event in the situation (in this regard, this is the closest position to Schlenker’s). Moreover, more important musical events should correspond to more important events in the situation¹⁴.

<footnote n="12"><sup>12</sup>Since the tree is directed, edges are ordered pairs of vertices.</footnote>
<figure><img src="image_32.png" /><figcaption>Figure 8: An illustration of Theory B on the example from Figure 4.</figcaption></figure>

* **Theory B** (weak<sup>13</sup> theory): Each note is not necessarily interpreted as an event in the situation. This does not mean that these uninterpreted musical events are semantically vacuous, but they do not refer to a concrete event, rather they modify the denoted situation as some adverb would in linguistics. Yet the musical heads always need to be interpreted — though not necessarily by head events. Moreover, more important interpreted musical events should correspond to more important events in the situation<sup>14</sup>.

As an illustration, Figure 8 shows possible trees associated with the music of Figure 4 and with the corresponding walking-situation, and an embedding of *S* into *M* as described by Theory B (for the sake of readability, we drew the arrows only for the first branch of *S*, but the same exist for the other branches). As one can see, not every note in *M* is matched to an event in *S*, but every head note is.

As mentioned above, both theories, though opposite, deal with the salience of notes or
events. Salience can be implemented in our treeish framework by the mean of heads. These
heads may be obtained by formal rules, exactly as the hierarchical structure is. However, we
think that this binary notion of heads is too coarse. We may wish, for instance, to distinguish
between three degrees of salience or more. We may then add, for instance, an *ad hoc* notion
of secondary heads, and require them to be interpreted, but not by first-order heads, and

<footnote n="13"><sup>13</sup>The names *weak* and *strong* theories reflect the fact that Theory A requires more constraints.</footnote>

<footnote n="14"><sup>14</sup>This last idea is related to what Schlenker has suggested in Section 8.5 of Schlenker (2019). There are stronger or weaker ways to express this condition; we will be more precise below. Note that it is also possible to completely drop it.</footnote>
so on. Since the number seems to be unbounded, we rather replace the notion of heads by
that of weight<fnref n="15" />. We then suppose that *M* and *S* are rooted directed tree coming together
with weight functions. A weight function on *M* (resp. *S*) is just a function *p*: *V* → R*+* (resp.
*p' : V' → R*+*). How these are obtained may rest on formal rules akin to those developed
in Lerdahl and Jackendoff (1983), but we did not investigate it yet. We can require that *f*
preserves weights in a sense we now make precise.

Let us begin with Theory A, which is simpler. Since every note is interpreted, we can
associate every vertex $x \in V$ with a vertex $g(x) \in V'$ such that $f(g(x)) = x$ ($g$ is the
reverse function of $f$). We then require, for $M$ to be true of $S$, that $\forall x, y \in V$, $p(x)p(y) \Rightarrow
p'(g(x))p'(g(y))$. This ensures that more important musical events will be interpreted as more
important events in the situation.

Things are bit more complex with Theory B, where all notes need not to be interpreted
(whence the function *g* doesn't necessarily exist). Since heads are now implemented as
weighted events, we will require that weighted musical events are interpreted. This can be
done by requiring that if a vertex *y* ∈ *V* has greater weight than another vertex *x* ∈ *V* and if
*x* is interpreted, then *y* is interpreted too (and by a more weighted vertex in *V*'). Nevertheless,
we think that this global condition is too strong and should be local: it may happen that some
part of the musical snippet describes the situation in a very fine-grained fashion (*i.e.* almost
every note is interpreted) while another does it in a more coarse-grained fashion, and yet the
former's notes do not have bigger weights than the latter's. Local conditions in trees have
been formalised in linguistics through the notion of c-command<fnref n="16" />. Formally, our localised
condition becomes: ∀*y* ∈ *V*, (∃*x* ∈ Im*f* s.t. *y* c-commands *x* and *p(y)p(x)*) ⇒ *y* ∈ Im*f*. We
also add the weight-preservation condition as in the other theory.

An alternative notion to weight would be that of reduction<fnref n="17" />. Intuitively, a musical
passage can be reduced to another if it is heard as an elaboration of it — e.g. an ornamented
version. The point is that if the tree-structure of a musical snippet encodes its reduction steps

<footnote n="15"><sup>15</sup>though the same intuition is behind</footnote>

<footnote n="16">¹⁶Abbreviation for constituent-command. Here we say that a node *x* c-commands a node *y* if it dominates it or is a sister of one of *y*'s ancestors.</footnote>

<footnote n="17">¹⁷At least two notions of reduction are discussed in Lerdahl and Jackendoff (1983): time-span reduction and prolongational reduction. For more details, see Chapter 5 and following.</footnote>
— as it is the case of the structures that Lerdahl and Jackendoff (1983) deal with<fnref n="18" /> — weight functions may be redundant because it seems that low-weighted musical events are those which disappear after a few reduction steps. One advantage of replacing weight by reduction steps is that it now comes along with the structure and does not need to be computed separately. One drawback is that we lose the local character of weight functions, because every local branch will now reduce at the same speed<fnref n="19" />. We leave a closer investigation of these theoretical possibilities of implementing heads for future research.

## 4.3 The meaning of uninterpreted notes

Assuming now that some notes remain uninterpreted, as in Theory B, what would be their semantic role? Let us give a few clues.

We saw in the case of Figure 4 that the added notes change the *character* of the walking-situation, but do not seem to add extra events. There could be several variants of such a phenomenon, regarding how these extra notes affect the semantics of the whole.

According to one variant<fnref n="20" />, each extra note modifies the meaning of its local branch. For instance it could be that the relevant reduction level in the case of Figure 4 is the beat level, so that each beat is viewed as a semantic atom, packaging all the information of its musical sub-events. The second and fourth beats of each bar thus have a semantics computed from the two quavers it is made of, and this may indicate that it is, for example, a particularly supple bounce. That is, each bounce is further characterised by this extra note. According to an alternative variant, extra notes are first ignored, leading to the same denotation set as with the simple snippet from Figure 1. Only in a second stage will the extra notes add the inference that the walking-situation is more bounced or more energetic; or it will give the listener clues about the mental state of the walking character (according to our informants, he is cheerful and happy, or even wanting to dance). Using a (possibly dubious) analogy from language, the former variant predicts that low-weighted notes behave more like adjectives modifying a noun phrase (here: a musical event), while the latter predicts they behave more

<footnote n="18">¹⁸The trees here are not directed rooted trees as generally understood in mathematics, but rather something equivalent to directed rooted trees with heads, though a bit more complex.</footnote>

<footnote n="19">¹⁹Typically, with time-span reduction, each group of two quavers will be at some point replaced by a single crotchet. But it could well be that in some branch each quaver is interpreted, while in some other only one of them is.</footnote>

<footnote n="20">²⁰Thanks to Philippe Schlenker for suggesting this variant.</footnote>
like adverbs, modifying the whole sentence. As we have no clue for favoring one over the other — leaving aside the fact that both could partly hold together — we will not say more than just the discussion of this example, and leave these issues for future research.

## 4.4 Is every event musically represented?

As a reverse question, we may ask if every event in the situation should be represented by a note in $M$. This seems to be hardly the case, since a situation can never be exhaustively described even with language.<fnref n="21" /> This seems to invalidate our theory, since we required that the situation tree is ‘contained’ in the musical tree, which is much smaller. However, we can get out of this problem by considering *reductions* of the situation tree<fnref n="22" /> or by saying that we will assimilate the situation with what is relevant in its perception by a given agent. We will then ask that every head event in the situation is represented by a head event in the musical snippet.

All this being said, let us state as a summary a final formal formulation of one variant of Theory B. Theory A could be straightforwardly formulated in a very similar way.

**Theorem 1** Let $M = (V, E, p)$ be a musical snippet and $S$ a situation. $M \models S$ if, and only if, $M \models S$ in Schlenker's sense, and moreover there exists a reduction $S' = (V', E', p')$ of $S$, and an injective root-preserving function $f: V' \rightarrow V$ such that:

$$1. \quad \forall x, y \in V', (x, y) \in E' \Rightarrow f(x) \sim f(y)$$

$$2. \quad \forall x, y \in V', p'(x)p'(y) \Rightarrow p(f(x))p(f(y))$$

$$3. \quad \forall y \in V, (\exists x \in \mathrm{Imf} \text{ s.t. } y \text{ c-commands } x \text{ and } p(y)p(x)) \Rightarrow y \in \mathrm{Imf}$$

# 5 Conclusion

The investigation of walking-situations and walk-denoting music enlightened the necessity to come up with a theory of musical events. This theory needs to provide rules involving

<footnote n="21">²¹Think, for example, of the whole subtlety in the gestures of a character, or of all the particle interactions that take place everywhere.</footnote>

<footnote n="22">²²Just like there are notions of reduction for music, we can posit that the same things exist with situations, as suggested in Jackendoff (2009).</footnote>
the structural role of each musical event determined through the rigorous analysis of musical
structures, and explain how this impacts the very possibility for each event to be interpreted,
i.e. to have a counterpart in the denotation. Besides, our goal was to account for how this
interpretive semantic mechanism works (i.e. what happens to the musical structure when
interpreted, and how the formal tree-like structure of the music is related to the formal
tree-like structure of the events it denotes). Further research will investigate the experimental
extensions of this theoretical work, in order to check whether it has some cognitive reality in
listeners.
Bibliography

Harasim, D., Rohrmeier, M. and O'Donnell, T. J. (2018), A generalized parsing framework for generative models of harmonic syntax, in 'Proceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, Paris, France, September 23-27, 2018', pp. 152-159.

Jackendoff, R. (2009), 'Parallels and nonparallels between language and music', **26**(3), 195–204.

Lerdahl, F. and Jackendoff, R. (1983), *A generative theory of tonal music*, The MIT Press, Cambridge. MA.

Schlenker, P. (2017), ‘Outline of music semantics’, **35**(1), 3–37.

Schlenker, P. (2019), ‘Prolegomena to music semantics’, **10**(1), 35–111.

Zacks, J., Tversky, B. and Iyer, G. (2001), ‘Perceiving, remembering, and communicating structure in events’, *Journal of experimental psychology. General* **130**, 29–58.
HOW MUSIC EVOKES WALKING
A CASE STUDY IN MUSIC SEMANTICS *

Léo Migotti

Institut Jean Nicod
ENS-EHESS-CNRS

Léo Zaradzki

Laboratoire de Linguistique Formelle
Université Paris Diderot

ABSTRACT

Music has recently been claimed to have a semantics, i.e. it is able to refer to a non-musical
reality in systematic ways. In this framework, different musical properties convey different
kinds of information about abstract objects or situations through semantic rules that map
a given musical form onto a set of possible situations. Because music semantics is very
abstract (many mental representations are often licensed by a same musical excerpt), we
here chose to work with a concrete situation that music can denote: someone walking. By
manipulating fine-grained musical properties of a reference stimulus, we tried to identify
what it takes for a music to evoke a walk in listeners. We built a preference task in which
participants had to rate which of two excerpts was better at representing a walk. Our
results show that at least 5 properties are involved in making a certain music evoke a walk:
while some structural constraints map onto the structure of a walking event (steadiness,
chord alternation, binarity), some more specifically musical properties (consonance and tonal
chord proximity) seem to be associated with physical stability. Our results provide further
enrichments and updates on a possible model for music semantics, in particular on (i) the
type of rules involved and (ii) the structural level at which music is interpreted.

**Keywords** Music semantics | Music and movement | Walking | Syntax/semantics interface

<footnote>*Contributions*: LM and LZ designed the experiment. LZ composed most stimuli which LM generated and implemented in the online experiment. LM managed online data collection and statistical analyses. LM wrote the paper, taking some inspiration from LZ's PhD dissertation on the same topic.</footnote>
# Contents

<table><tr><td>1</td><td>Introduction</td><td>138</td></tr><tr><td>2</td><td>General paradigm</td><td>139</td></tr><tr><td>2.1</td><td>Target stimulus</td><td>139</td></tr><tr><td>2.2</td><td>Task</td><td>140</td></tr><tr><td>3</td><td>Some musical stimuli evoke walking better than others</td><td>142</td></tr><tr><td>3.1</td><td>Stimuli</td><td>142</td></tr><tr><td>3.2</td><td>Results</td><td>142</td></tr><tr><td>4</td><td>Identifying musical properties responsible for the music/walk association</td><td>144</td></tr><tr><td>4.1</td><td>Candidate properties</td><td>144</td></tr><tr><td>4.2</td><td>Degraded stimuli</td><td>145</td></tr><tr><td>4.3</td><td>Results</td><td>147</td></tr><tr><td>5</td><td>Enriching the model of music semantics</td><td>148</td></tr><tr><td>5.1</td><td>Ordering preservation-based model</td><td>148</td></tr><tr><td>5.2</td><td>Wrong predictions from RPOs</td><td>150</td></tr><tr><td>5.3</td><td>Stimuli</td><td>151</td></tr><tr><td>5.4</td><td>Results</td><td>152</td></tr><tr><td>6</td><td>Are all musical events semantically interpreted?</td><td>154</td></tr><tr><td>6.1</td><td>Motivation</td><td>154</td></tr><tr><td>6.2</td><td>Stimulus</td><td>155</td></tr><tr><td>6.3</td><td>Results</td><td>157</td></tr><tr><td>7</td><td>General discussion and conclusions</td><td>158</td></tr><tr><td>A</td><td>Predictions from stimuli</td><td>162</td></tr><tr><td>B</td><td>Statistics</td><td>169</td></tr></table>
# 1 Introduction

Although it is common to assume that music triggers many cognitive effects such as emotions (Meyer, 1956; Gabrielsson and Lindström, 2010; Koelsch, 2012; Sievers et al., 2019) or expectations (Huron, 2006), the idea that it can also evoke non-musical representations in systematic ways has received less attention, despite both formal and informal accounts on how listeners spontaneously make associations between music and imagined scenes or situations (Eitan and Granot, 2006; Eitan and Timmers, 2010; Eitan, 2013). It has in particular been argued that these associations can be accounted for by semantic rules that establish connections between music and what it evokes in listeners: this amounts to arguing that music has a semantics (Schlenker, 2017). Under this view, music does not only convey information about itself or its own form, but also about an extra-musical reality: when listening to a tune, a song or a symphony, one does not only draw inferences about the actual source of the sound (the violin, the singer, the orchestra), but one is able to represent music in more abstract terms, and associate musical properties with properties of imagined scenes and objects in a productive fashion.

Here, we propose to test the idea that music has a semantics experimentally through a case-study on how music can evoke a simple movement: walking. The purpose of this paper is to understand why some musical pieces represent walks better than others, and which musical properties and semantic rules are responsible for these effects. This paper is structured as follows. Section 2 presents the main stimulus whose ability to evoke a walk was tested throughout the experiment. The next sections present our experiment in three steps.<fnref n="2" /> First, Section 3 shows that the target stimulus was systematically judged as evoking a walk better than 3 other pieces. Section 4 presents 5 conditions we hypothesized were responsible for the semantic association between our musical excerpt and a walk. By destroying one condition at a time and asking for preference between degraded versions of the original stimulus and the original stimulus, we show that the latter was systematically judged as better evoking a walk. Section 5 makes a contribution to a possible model for music semantics by showing

<footnote n="2">²The data was collected in a single online experiment. Our conclusions remain unaffected by this experimental constraint; we however present the different blocks separately for the sake of clarity.</footnote>
that absolute levels of consonance were interpreted in terms of absolute physical stability of
the walking character. Finally, Section 6 explores the hypothesis that some musical events
might not be interpreted and presents results which reject this hypothesis: without further
evidence, it seems that all musical events are semantically interpreted.

2 General paradigm

2.1 Target stimulus

The main goal of the experiment was to understand why the stimulus in Figure 1 (referred
to as the ‘target stimulus’) was associated with a walking event in listeners. By ‘walking
event’, we mean an event involving a character or person walking. This stimulus was selected
for its intuitive ability to evoke a walk; yet, we provide further justification for why this
association might occur in the first place in Appendix A. The target stimulus, composed after
a common pattern of a ‘walking bass’,$^3$ consists of 4 repetitions of a same 4-beats motive: a
first major C chord in its fundamental state, split into a bass C on the first beat (evoking
the first step), and a higher-pitched C major chord (evoking a kind of ‘rebound’). A second
major C chord in its 2nd inversion state follows, split into its bass G on the 3rd beat (evoking
the second step) and the same chord on beat 4 as on beat 2. The stimulus was generated
through MuseScore © [Version 3.6.2.548020600] at a given tempo of 80 bpm.$^4$

<footnote n="3"><sup>3</sup>As noted in Zaradkzi (2021), this terminology does of course not determine the ability of such a pattern to denote walk per se. A walking bass only corresponds to a succession of bass notes typically played by a bass or by a piano, in which case the bass notes can be played together with higher-pitched chords off-beat. In our case, to simplify the analysis, the higher-pitched chords are placed on the 2nd and 4th beats respectively rather than off-beat, so that they can be analyzed as full musical events.</footnote>

<footnote n="4">⁴80 bpm approximately correspond to a slow gait pattern, the standard reference speed level for walking being usually around 90-100bpm. Note that in this experiment, tempo was not varied at any point. Although previous research had established that people are rather good at synchronizing their walk to the beat, suggesting that tempo is a very important factor contributing to the music-to-walk mapping, we were here more interested in revealing aspects that had not been addressed in the literature on the interaction between music and gait, and in particular in the investigation of musical properties unrelated to speed.</footnote>
<figure><img src="image_33.png" /><figcaption>Figure 1: Tentative target stimulus <fnref n="5" /></figcaption></figure>

## 2.2 Task

50 participants took part in the online experiment, in French. They were recruited through the online platform Crowdpanel© and gave online consent before starting the experiment. 5 participants declared having already taken a similar study; these participants were thus removed from the data to avoid potential biases from prior exposure to a pilot study. For each trial, participants were presented two audio files that could be played as many times as needed, one of which was always the target stimulus.<fnref n="6" /> In each of the following sections, we present the confronting stimuli paired up with the target. Participants were instructed to imagine a character walking before playing the two stimuli, and then asked to estimate which of the two excerpts evoked a situation in which somebody was walking better, using a continuous scale ranging from the excerpt on the left to the excerpt on the right. The choice was not binary: participants could place the cursor anywhere in between the two extremities. The position of each stimulus (right or left) was systematically randomized for all pairs of stimuli. Each participant was presented 18 couples of stimuli. Details are provided in the relevant sections.

For each trial, a picture of a cartoon character walking (as shown in Figure 2) was displayed.<fnref n="7" /> We reasoned that this visual anchor was needed for two main reasons. First, informal pilot studies on informants suggested that having a visual substantially increased engagement with the task: it appeared that having something to look at makes the listening

<footnote n="5">⁵The stimulus is accessible by clicking on the caption of the figure directly.</footnote>

<footnote n="6">⁶The stimuli are presented in the relevant sections, along with hyperlinks for the reader to listen.</footnote>

<footnote n="7">⁷Our paradigm is reminiscent of previous paradigms testing for associations between music and representations of body motion, in particular Eitan and Granot (2006) where participants had to imagine a character moving through space.</footnote>
experience easier, especially if the visual is relevant to the audio. Second, it was important that each participant imagined the same character walking when giving their judgment on how well the music could depict such a scene. If each participant could imagine their own walking scene freely, we would not have controlled for two major biases: (i) participants could have changed the mental scene they imagined throughout the experiment, resulting in the impossibility to compare how ‘well’ each stimulus did at triggering a representation of a character walking because different stimuli could have represented different characters, walking in different ways, with different gait patterns, etc. and (ii) even if participants did stick with such a mental scene throughout the experiment, it was possible that each participant had their own, which could have been different from that of other participants.<fnref n="8" />

<figure><img src="image_34.png" /><figcaption>Figure 2: A screenshot of the task (with English translations in italic)</figcaption></figure>

<footnote n="8">⁸Note that in any event, we have no way of knowing whether participants did indeed represent the scene with the given character; yet we take the use of the visual to be our best attempt to fix a reference level relative to which they had to judge how convincing the music they heard was at describing the scene.</footnote>
# 3 Some musical stimuli evoke walking better than others

The first experimental step in our procedure was to confirm introspective judgments and to make sure that the interpretation of our target stimulus as a walk was salient. We thus had participants compare the target stimulus to 3 other pieces, referred to as ‘control stimuli’ hereafter. Note that it was enough, for our purposes, that this target stimulus be significantly *more* associated with a character walking than these other stimuli. Details are provided in the next section.

## 3.1 Stimuli

The 3 control stimuli were selected from real music for their intuitive inability to evoke a walk mainly due to (i) rhythmic irregularities and/or slow tempo (i.e. pieces whose metrics was hardly identifiable by non-musicians, with no recurrent pattern), and (ii) their use of solo piano that matched the timber and instrumental setting of the target stimulus to keep spectral differences as minimal as possible and to avoid creating parasitic inferences from timber. As detailed in Figure 3, Control 1 was taken from the first seconds of Mendelssohn’s 1st piano concerto (2nd movement); Control 2 was taken from the first seconds of Scriabine’s Impromptu op. 12 n°1; and Control 3 was from the 1st movement of Schnittke’s concerto for piano and strings. The duration of each control roughly matched that of other stimuli, although we decided to make cuts that sounded as musically natural as possible to avoid any effect due to surprise or weirdness. The target stimulus formed 3 pairs of stimuli with each control, from which only 2 were randomly presented to each participant to reduce experimental duration and avoid redundancy.

## 3.2 Results

As expected, the target stimulus was significantly preferred over each control, with no significant differences in endorsement between controls: when asked to choose which excerpt
<table><thead><tr><th>Pair</th><th>Target</th><th>Control</th></tr></thead><tbody><tr><td>1</td><td></td><td>Mendelssohn</td></tr><tr><td>2</td><td></td><td>Scriabine</td></tr><tr><td>3</td><td></td><td>Schnittke</td></tr></tbody></table>

Scores retrieved from imslp.org

<figure><img src="image_35.png" /><figcaption>Figure 4: Preference for target over control stimuli</figcaption></figure>
represented a walking event better, participants strongly tended to choose the target stimulus over any of the 3 controls.<fnref n="9" />

The interpretation of these results is twofold. First, they show that music has indeed the ability to refer to non-musical events.<fnref n="10" /> Second, they show that it is not a mere property of any music to be associated with any event. It is however possible that participants would not have spontaneously interpreted the target stimulus as a walk in the absence of any context. Yet, we claim that a statistically significant tendency to pair a given imagined situation (a walk) with a musical excerpt (the target) more than others is enough empirical evidence for considering the target as a good candidate for evoking a walking event.

# 4 Identifying musical properties responsible for the music/walk association

## 4.1 Candidate properties

After having confirmed that our target stimulus could be interpreted as representing a walk, we were interested in identifying the properties of the stimulus that were responsible for this effect. After careful manipulation of several musical parameters in our target stimulus, we came up with a list of 5 conditions that we hypothesized had to be met to allow any listener to associate a piece with a walking event.<fnref n="11" /> These conditions are given in (1).

<footnote n="9">⁹Detailed statistical analyses are provided in B. The data and analyses script, along with pre-registered files are available at this link.</footnote>

<footnote n="10">¹⁰Whether the fact that music can refer to an extra-musical reality amounts to proving that music has a semantics is discussed in section 7.</footnote>

<footnote n="11">¹¹Here again, the five identified conditions can also be derived theoretically: not only do we find intuitive reasons for testing these conditions through direct rewriting and testing, but these conditions also made sense given (i) a model of a walking event and (ii) a set putative semantic rules explaining the mapping from an event to music. We provide further theoretical justification in Appendix A.</footnote>
(1) **Conditions on music to evoke a walking event.**

For a musical piece to be understood as referring to a walking event, it needs to be
reducible to:

1. A regular

2. Alternation

3. Of 2 chords

4. That are consonant<fnref n="12" />

5. And tonally close to one another

We then reasoned that, if these properties were responsible for the association of the
target stimulus with a walk, it was then possible to weaken or even block this association by
destroying these properties. The following section details how each condition was violated
through minimal rewritings of the target stimulus.

4.2 Degraded stimuli

The first manipulation was about breaking rhythmic evenness by varying the originally
uniform delay between notes (DS1). The second rewriting was about breaking the alternation
condition through repetition (DS2), while the third one was about breaking binarity by
alternating three chords instead of two (DS3).<sup>13</sup> The fourth rewriting breaks consonance
by replacing major triads with dissonant clusters (DS4). The last manipulation breaks
tonal proximity between the two alternating chords by increasing their distance in the tonal
space (DS5). The target stimulus formed 5 pairs of stimuli with each degraded stimulus,
from which only 3 were randomly presented to each participant so as to avoid redundancy
throughout the experiment. Details about the rewriting procedure and the expected effect of
each manipulation are provided in Figure 5.

<footnote n="12">¹²A discussion of conditions 4) and 5) and the underlying semantic model is given in Appendix B.</footnote>

<footnote n="13">¹³A clarification is needed at this point: by ‘binarity’, we here refer to the metrical structure of the stimulus, i.e. that each bar here contains an even number of beats (4). Despite the presence of four separate musical events in our stimulus, the harmonic structure can be reduced to only two chords; hence breaking binarity does not entail breaking the pairs of the form <bass, chord> but pairs of the form <chord₁, chord₂>, i.e. <<bass₁, chord₁>, <bass₂, chord₂>>.</footnote>
<table><thead><tr><th>Name</th><th>Condition violated</th><th>Rewriting procedure</th><th>Score</th><th>Change in the semantics*</th></tr></thead><tbody><tr><td>DS 1</td><td>steadiness</td><td>Steadiness was broken by inserting rests so the time interval between musical events was not constant anymore and the metrical structure not identifiable.</td><td></td><td>Destroying regularity creates a sense of imbalance or hesitation.</td></tr><tr><td>DS 2</td><td>alternation</td><td>The back and forth from C to G was removed from the bass line and only C was kept, leading to the repetition of a same bass note and a chord 8 times in a row.</td><td></td><td>Under a 'walk' interpretation: the character seems stuck. Under a more abstract interpretation: an event is repeated and seems prevented from evolving.</td></tr><tr><td>DS 3</td><td>binarity</td><td>Steadiness and alternation don't constrain the number of musical events involved. Binarity is naturally what can represent a human walk the best as two feet and hence two steps only are involved. We thus replaced the binary alternation from C to G with a ternary alternation from C to G and then to E. E was selected because it is the only remaining available note from the C major perfect chord; hence the introduction of E as a third bass note would not create any harmonic novelty in the stimulus.</td><td></td><td>The introduction of a third component in the binary structure of the stimulus creates a ternary cycle incompatible with a walk.</td></tr><tr><td>DS 4</td><td>consonance</td><td>In the target stimulus, the bass line only contained singles notes and the chords were perfect chords: everything thus sounded extremely consonant. Here, bass notes were replaced by dissonant clusters of notes matching the pitch change in the target on beats 1 and 3, while the C chord was replaced by a dissonant augmented fourth interval (present in the cluster, as the chord on beats 2 and 4 in the target was derived from C).</td><td></td><td>The high level of dissonance creates an impression of great unstability.</td></tr><tr><td>DS 5</td><td>chord proximity</td><td>We increased tonal distance between the two chords by replacing the second inversed C chord by a major F# chord. Just as for DS4, we almost preserved the structure of the target stimulus except that the chord on beats 4 is now different from that on beat 2, but we favored harmonic consistency between the bass and the corresponding chord over repetition (the only reason for which the chord was repeated in the target stimulus was that the two bass notes pertained to a same C major chord).</td><td></td><td>The great tonal distance between the two chords creates a sense of imbalance between the events, or, under a 'walk' interpretation, between the two steps (as if the character was limping).</td></tr></tbody></table>

* The purpose of this column is just to give an idea of how 'different' from the target the degraded stimuli sounded. These judgments come from the authors' own intuitions, along with some feedback collected during the piloting phases of the project. They do not aim at establishing any rule for characterizing the stimuli and only provide plausible descriptions of the semantic effects resulting from the destruction of the target's main musical properties.

<caption>Figure 5: Degraded stimuli. Stimuli can be accessed at the following links: DS1, DS2, DS3, DS4, DS5</caption>
## 4.3 Results

<figure><img src="image_36.png" /><figcaption>Figure 6: Preference for target over degraded stimuli</figcaption></figure>

<table><thead><tr><th colspan="2">Reminder: Degraded stimuli</th></tr></thead><tbody><tr><td>DS5</td><td>Breaks tonal proximity</td></tr><tr><td>DS4</td><td>Breaks consonance</td></tr><tr><td>DS3</td><td>Breaks binarity</td></tr><tr><td>DS2</td><td>Breaks alternation</td></tr><tr><td>DS1</td><td>Breaks steadiness</td></tr></tbody></table>

We found that the target stimulus did significantly better at evoking a walk in participants than any degraded stimulus.<fnref n="14" /> These results suggest that each of the five identified conditions did contribute to the association of the target stimulus with a walking event. A pairwise comparison of each possible pair of degraded stimuli revealed a few significant differences in the preference for the target depending on which degraded stimulus it was paired with. In particular, we found marginally significant differences between DS1 and DS3; DS4 and DS2; DS4 and DS3 respectively: even though all degraded stimuli were judged as being less evocative than the target stimulus, some of them were particularly less evocative than others. These differences might indicate some kind of hierarchy between the tested conditions. For instance, breaking steadiness appears to alter the inference significantly more than breaking binarity; and breaking consonance seemed to alter the inference significantly more than breaking the alternation condition or the condition on binarity. It is however too soon to state anything more about the exact contribution of each condition to the generation of the associated representations. Further work is required to make sure that some musical properties represent more fundamental properties of the denoted event. It seems however reasonable to posit that the musical properties that are interpreted in terms of the most

<footnote n="14">¹⁴The statistical details are provided in Appendix B.</footnote>
essential properties of the evoked scene are the ones whose violation blocks the evocation of
a walk the most.

# 5 Enriching the model of music semantics

After having (i) confirmed that the target stimulus was a good musical candidate for evoking a walk (Section 3) and (ii) identified and tested some properties responsible for this association (Section 4), we then wanted to check whether these results fitted within a model of music semantics such as that of Schlenker (2017). This model, which relies on Rules of Preservation of Orderings (RPOs), is presented in the next section. We will argue that our results reveal a limitation of this model, and we suggest ways to enrich it.

## 5.1 Ordering preservation-based model

We will first explain what rules of preservation of orderings are, and we will illustrate these rules with a single musical property (for clarity): harmony. In Schlenker (2017), harmonic stability is taken to be interpreted in terms of physical stability of objects through a rule of preservation of ordering (RPO):<fnref n="15" />

> Let $M =< M_1, ..., M_n >$ be a musical event, and $O$ an object involved in event $E =< e_1, ..., e_n >$. M is true of E if when $M_i$ is less harmonically stable than $M_k$, O is in a less stable position in $e_i$ than in $e_k$.

This rule is one of several in a model of musical truth, which involves other requirements on time structure and loudness. Here, we simplify the model and reduce it to its condition on harmonic stability for simplicity. A little detour through picture semantics will now be helpful to clarify this condition. Let us take an example from Schlenker (2017) in Figure 7.

<footnote n="15">¹⁵In the original piece, this rule is used to define musical truth, hence the rule provides conditions as to how a musical event can be ‘true of’ an event. While we take inspiration from this framework, we won’t be using the notion of truth here, and we will approximate the meaning of ‘is true of’ by ‘can evoke’.</footnote>
<figure><img src="image_37.png" /><figcaption>Figure 7: Preservation of ordering in pictures semantics</figcaption></figure>

The question here is: which picture among pictures a, b and c is the graphical representation on the left true of? In other words, which picture is accurately represented by the graphical representation? Intuitively, picture a. is the best candidate: the graphical representation could denote many other situations, but the situation depicted in picture a. is one of them. Why is this so? Schlenker argues that this comes from the fact that the height of the bars in the graph is ordered in the very same way as the height of the characters in picture a. Roughly: $height(B) < height(A) < height(C)$. The semantic rule involved in making picture a. a possible denotation of the three bars on the left is thus a RPO with regards to the relevant property: in this case, height.

Musical properties are however of course not visually but auditorily perceived. What does this then mean that harmonic stability obeys the same kind of rule (namely, a RPO) as height does in the visual example above? It means that, if it is true that harmonic stability is interpreted as physical stability (just as the height of bars is interpreted as the height of people in the visual example), then the ordering in physical stability in a situation *S* must match the ordering in harmonic stability in music *M* for *S* to be a possible denotation of *M*. For instance, the succession of a first harmonically very stable chord and a second harmonically very unstable chord<fnref n="16" /> could denote any succession of two events whose physical stability evolves in the same direction, i.e. decreases.<fnref n="17" /> A series of two events such that the first one is someone standing and the second one is someone falling is a possible denotation, as long as the latter event is represented as being less stable than the former.

<footnote n="16">¹⁶We will come back to this notion of harmonic stability later.</footnote>

<footnote n="17">¹⁷Note that in Schlenker’s model, rules on harmony are enriched with rules on time and loudness, which we will put aside here for the sake of simplicity.</footnote>
Because of reasons mentioned in Appendix A, we will not be using harmonic stability in our analysis and we will prefer two related musical properties: (i) consonance and (ii) chord proximity, for which we provide a semantic analysis largely inspired by Schlenker’s in Appendix A. We assume the rules derived from these analyses in the following sections.

## 5.2 Wrong predictions from RPOs

When applied to our target stimulus, a model based on RPOs appears to make wrong predictions. To show this, let us first consider the piece in Figure 8:

<figure><img src="image_38.png" /><figcaption>Figure 8: A piece which RPOs predict should evoke a walk</figcaption></figure>

Given the evolution of consonance throughout this stimulus, and assuming RPOs on consonance as detailed in Appendix B, the stimulus should be able to evoke a walking event. Now, let’s consider our target stimulus once again:

<figure><img src="image_39.png" /><figcaption>Figure 9: Target stimulus</figcaption></figure>

A similar formal analysis of the stimulus relying on RPOs, available in Appendix A, yields the same prediction: the target stimulus should be able to denote a walk.

As represented in Figure 10, both musical excerpts meet the criteria on consonance for denoting a walk.<fnref n="18" /> The prediction from RPOs is thus that both musical excerpts should be

<footnote n="18">¹⁸Formal details are available in Appendix A.</footnote>
<figure><img src="image_40.png" /><figcaption>Figure 10: Consonance to physical stability mapping</figcaption></figure>

evocative of a walk, and that no preference should be found for either stimulus. Yet, both excerpts are not as good at evoking a walk: the target stimulus is intuitively much better than the stimulus in Figure 8, which suggests that RPOs are insufficient to account for why the target stimulus evokes a walking event. We confirmed these intuitions experimentally and present the results in the following sections.

## 5.3 Stimuli

The stimuli consisted of the target stimulus, the excerpt presented in Figure 8 (referred to as ‘Ordering only 1’), and a second similar stimulus, shown in Figure 11 (referred to
<figure><img src="image_41.png" /><figcaption>Figure 11: Another piece which RPOs predict should evoke a walk</figcaption></figure>

as ‘Ordering only 2’), which also seemed to fail to evoke a walk despite complying with RPOs on consonance. We introduced this second stimulus to make sure that our results were not excerpt-specific and could be generalized to any stimulus with the right consonance structure. This stimulus follows the exact same change in consonance as that in Figure 8, while suppressing the changes in range between successive chords (i.e. the alternation of lower-pitched and higher-pitched chords): the chords on beat 1 and 3 were simply shifted up by octave. The analysis we provide in Appendix A thus holds for this stimulus just as much: RPOs predict that the stimulus in 11 can evoke a walk. The target stimulus formed 2 pairs with each of these two stimuli complying with RPOs. Each participant was presented with the 2 pairs.

## 5.4 Results

<figure><img src="image_42.png" /><figcaption>Figure 12: Preference for target over stimuli satisfying ordering conditions</figcaption></figure>
Since all three stimuli comply with RPOs, our prediction was that if music semantics only relied on RPOs, then all three excerpts should be associated with a walk with no distinction, and no preference for either stimulus would be found. However, the results show that the target stimulus was systematically and significantly preferred over any of the two other stimuli complying with RPOs.<fnref n="19" /> These results therefore suggest that we must reject the idea that RPOs alone account for the semantics of our target stimulus. We now provide two possible reasons for which we found that contrast, and suggest a way to enrich the model to capture the semantic differences we evidenced.

A first difference between the target stimulus and the stimuli in Figures 8 and 11 is that the former complies with the condition on relative chord proximity (all musical events in the target are either single tones from a major C chord, or a major C chord), while the two other stimuli do not: the Eb and F# chords on beats 1 and 3 are indeed very distant from one another in the tonal pitch space. This suggests that the similarity between the two steps<fnref n="20" /> had to be represented by two musical events that are close to one another in the tonal pitch space. In other words, chord proximity seems to be a necessary condition because it is interpreted as the similarity between the two denoted events (the two steps), i.e. in terms of gait symmetry. This is consistent with our results from Section 4 in which decreasing chord proximity resulted in an impression of ‘limping.’

Second, another main difference between the target stimulus and the two other stimuli is that all musical events in the target stimulus are *absolutely* consonant, while in the two other stimuli, the introduction of the very dissonant clusters on beats 2 and 4 seem to block any evocation of a walking event. It seems that the slight differences in stability occurring during a walking event (for instance, the moment where the foot hits the grounds does not have the same stability as the moment where the foot is lifted) cannot be represented by *any* difference in consonance. First, a *small* difference in physical stability (such as the slight asymmetry between the two feet) should probably be represented by a *small* difference in consonance (such as that between a chord in its fundamental state, and an inverse chord 6/4). But this

<footnote n="19">¹⁹Details are provided in Appendix B</footnote>

<footnote n="20">²⁰In Appendix B, we show that although both steps could be represented as having the same level of physical stability, footedness creates a slight asymmetry in stability based on which foot one is standing on.</footnote>
is not enough: it seems that *all* events need to be denoted by a consonant musical event for
the musical sequence to evoke a walk. Crucially, RPOs do not make this prediction: as long
as a difference in consonance in a musical sequence *M* maps onto a difference in physical
stability in an event *E*, RPOs predict that *M* can evoke *E*, regardless of the amplitude
of this difference, and regardless of the correspondence from single stable events to single
consonant musical events. Together with the results from Section 4, which established that
suppressing overall consonance for all events blocked the association with a walk, this new
piece of evidence gives extra-weight to an analysis under which absolute levels of consonance
are interpreted as absolute physical stability of events.

# 6 Are all musical events semantically interpreted?

The previous sections established that (i) our target stimulus was more associated with a
walking event than control stimuli, (ii) that the 5 conditions on this stimulus we identified
reinforced a preference for the target stimulus when violated and (iii) that rules of preservation
of orderings are insufficient to account for the association of our target stimulus and a character
walking. In this last section, we present a final contribution of the present study through
an investigation of the level of granularity at which this musical stimulus is semantically
interpreted.

## 6.1 Motivation

Formal accounts of musical syntax have shown that musical structure can be represented at
different hierarchical levels. One of the main contributions of the *Generative theory of tonal
music* (GTTM) by Lerdahl and Jackendoff (1983) was to introduce the notion of *reduction*,
which refers to the idea that listeners represent a complex musical piece at different levels
of complexity, and accounts for musical intuitions about which musical events are more
‘important’ in a musical piece. GTTM introduces two such systems: time-span reductions,
and prolongational reductions. Time-span reductions generate tree-like hierarchical structures
mainly based on rhythmic stability, with hierarchical heads at each structural level that
correspond to the most structurally important musical event in that level. Prolongational reductions also generate tree-like structures but based on tension and relaxation patterns (i.e. on the relative stability of events), which patterns are themselves determined by stability conditions on branching, pitch, melody and harmony.

Schlenker (2017) provides a model of the semantic interpretation of headed events from time-span structures. This model introduces a notion of difference in importance between denoted events: roughly, headed (i.e. structurally important) events must denote events that are more important than the events denoted by non-headed events.<fnref n="21" /> This presupposes that both types of events are semantically interpreted, a non-trivial presupposition discussed in great detail in Zaradkzi (2021). Here, we tested an alternative hypothesis that not all musical events are interpreted as events, and, more specifically, that it is possible that less structurally important musical events (non-heads) are not interpreted as events but merely as ‘modifiers’ operating at a larger scale on events or groups of events. We tested this hypothesis experimentally using the stimulus presented in the next section.

## 6.2 Stimulus

<figure><img src="image_43.png" /><figcaption>Figure 13: The target stimulus enriched with an extra-8th chord on beats 2 and 4 of each bar, accessible here.</figcaption></figure>

The stimulus in Figure 13 was built from the target stimulus in Figure 1, in which the already present C major chord was repeated on beats 2 and 4, resulting in two 8th C chords on these two beats in each bar. From a structural perspective, it is reducible to the target stimulus through a time-span reduction: since each new chord is subordinate to the previous one, it is not represented at any higher structural level and in particular not at the immediately higher level which is that of the target stimulus, as represented in Figure 14.

<footnote n="21">²¹Note that Schlenker writes that this notion of importance should be clarified; and we agree.</footnote>
<figure><img src="image_44.png" /><figcaption>Figure 14: time-span reduction</figcaption></figure>

Introducing this extra musical event in a recurrent way intuitively resulted in the stimulus sounding a bit happier or light-hearted, but without the clear impression that any event had been added to the denoted situation. In other words, it seemed to us that this new stimulus could denote the same kind of situations as the target stimulus did, suggesting that the addition of new musical events does not entail the addition of new denoted events.

We thus presented participants with a pair of stimuli made of the target stimulus and the enriched version with the additional chord in Figure 14 to check which of the two they would judge as denoting a walking event better. Our prediction was that if the stimulus was interpreted at a hierarchically higher level with less rhythmic complexity (i.e. the additional chord does not substantially modify the set of situations the piece can evoke), then no preference would be found for either stimulus. By contrast, if all musical events are interpreted as events (at the lower level of musical structure), then introducing a new chord should block the association with a walking event because this extra-chord would not correspond to any sub-event involved in the walking event; hence a significant preference for the target would be found. We also had participants compare this stimulus with previous control stimuli and degraded stimuli for more exploratory purposes; but a prediction was that if it were interpreted just as the target stimulus, then we would find a significant preference for it compared to any other type of stimulus.
<figure><img src="image_45.png" /><figcaption>Figure 15: Preference for extra-8th chord version over the target stimulus, control stimuli and degraded stimuli</figcaption></figure>

## 6.3 Results

The results in Figure 15 did not confirm our predictions: the enriched version of the target stimulus was judged as being evocative of a walk significantly less than the target stimulus.<fnref n="22" /> One must note that, qualitatively at least, the contrast found in endorsement remained small.

The enriched stimulus was very highly preferred compared to the three pieces used as control stimuli. This is not surprising, as that stimulus sounded much more similar to the target stimulus than to any of the control stimuli. The results from the comparison of the enriched stimulus and the degraded stimuli show that the former was only significantly more associated with a walking event than DS1 (breaking steadiness) and DS4 (breaking consonance). This suggests that introducing a new musical event without changing the fundamental structural properties of the target stimulus was perceived as altering the representation less than having two dissonant chords, and less than having an irregular sequence. This might suggest that these two conditions correspond to more fundamental properties of a walking event.

Together, these results indicate that introducing a higher hierarchical level (by means of doubling a musical event reducible to a more structurally important one) alters the association with a walking event less than violating some of the 5 conditions we identified. It was however

<footnote n="22">²²The statistical details are provided in Appendix B.</footnote>
not the case that listeners can simply disregard a new musical event, even if that musical
event was structurally less important.

7 General discussion and conclusions

We have presented an experimental paradigm investigating a case study in music semantics: the case of music evoking walks. We here summarize the main findings of this research and highlight some possible extensions.

First, our results give empirical support to the claim that music can be interpreted as
evoking non-musical events. Second, it shows that some specific musical properties are
responsible for evoking a walk, and we proposed that this was due to the form of our stimulus
being reducible to a steady binary alternation of consonant and tonally close chords. Third,
and drawing on a model developed in Schlenker (2017), we showed that rules of preservation
of orderings, which state that what it takes for a music to evoke an event is that musical
properties satisfy the same ordering as the interpreted properties, do not account for the
evocation of a walk in listeners, and we proposed that listeners might be able to interpret
some musical properties absolutely, and in particular interpret a consonant musical event as a
stable physical event. Fourth, we tested the hypothesis that some structurally less important
musical events might not be semantically interpreted, but our results suggest that this was
not the case.

Another contribution of this work is to show that the semantic mappings from music
to imagined scenes were independent of musical expertise. Each participant indeed had to
self-report their level of musical expertise through a question displayed at the end of the
experiment. We then checked whether there were differences in the responses based on musical
expertise. A statistical analysis (detailed in Appendix B) revealed that this was not the case:
musical expertise never had a significant effect on the preference for any stimulus. Although
further work is required to test the effect of musical knowledge on a wider range of stimuli,
these preliminary findings suggest that listeners are able to generate mental representations
from music regardless of whether they can consciously engage in an analysis of the structural
properties of the piece.

Our methodology and experimental paradigm can finally be extended to the investigation
of the semantics of any musical piece. In particular, this paradigm can be extended to
investigate specific semantic effects from program music, such as music representing natural
elements or animals, in which the semantic effects are strong. However, the difficulty in this
task remains that any inquiry in music semantics requires a precise enough model of music
semantics and a precise enough model of the *perception* or representation of the denoted event
under investigation. While the former is developing, the latter might not be available for
objects, situations or events traditionally represented by music. The case of walking, which
has received a lot of attention in physiology and human motion analysis, was particularly well
suited in our case. One could therefore privilege case-studies of musical sequences evoking
events which have been modeled in the literature to make formal analyses of these events
easier, and thus to make the formal analyses of the musical properties denoting these events
easier as well.
## References

Cutting, J. E. and Proffitt, D. R. (1981), Gait Perception as an Example of How We May Perceive Events, in R. D. Walk and H. L. Pick, eds, ‘Intersensory Perception and Sensory Integration’, Springer US, Boston, MA, pp. 249–273.

de Neeve, M. (2021), Larsons’ musical forces in Schlenker’s music semantics, Riga.

Eitan, Z. (2013), How pitch and loudness shape musical space and motion, in ‘The psychology of music in multimedia’, Oxford University Press, New York, NY, US, pp. 165–191.

Eitan, Z. and Granot, R. Y. (2006), ‘How Music Moves’, *Music Perception* **23**(3), 221–248.

Eitan, Z. and Timmers, R. (2010), ‘Beethoven’s last piano sonata and those who follow crocodiles: Cross-domain mappings of auditory pitch in a musical context’, *Cognition* **114**, 405–422. Place: Netherlands Publisher: Elsevier Science.

Gabbard, C. and Iteya, M. (1996), ‘Foot laterality in children, adolescents, and adults’, *Laterality* **1**(3), 199–205.

Gabrielsson, A. and Lindström, E. (2010), The role of structure in the musical expression of emotions, in ‘Handbook of music and emotion: Theory, research, applications’, Series in affective science, Oxford University Press, New York, NY, US, pp. 367–400.

Hart, S. and Gabbard, C. (1997), ‘Examining the stabilising characteristics of footedness’, *Laterality* **2**(1), 17–26.

Hausdorff, J. M. (2007), ‘Gait dynamics, fractals and falls: Finding meaning in the stride-to-stride fluctuations of human walking’, *Human Movement Science* **26**(4), 555–589.

Hayafune, N., Hayafune, Y. and Jacob, H. A. C. (1999), ‘Pressure and force distribution characteristics under the normal foot during the push-off phase in gait’, *The Foot* **9**(2), 88–92.

Helmoltz, H. v. (1877), ‘On the sensations of tone as a physiological basis for the theory of music.’, *Dover Publications*.

Huron, D. (2006), *Sweet Anticipation: Music and the Psychology of Expectation*, The MIT Press.

Jackendoff, R. (2009), ‘Parallels and Nonparallels between Language and Music’, *Music Perception: An Interdisciplinary Journal* **26**(3), 195–204. Publisher: University of California Press.

Koelsch, S. (2012), *Brain and music*, Wiley-Blackwell, Chichester, West Sussex; Hoboken, NJ. OCLC: ocn767563922.

Larson, S. (2012), *Musical forces: motion, metaphor, and meaning in music*, Musical meaning & interpretation, Indiana University Press, Bloomington. OCLC: ocn707212791.

Lerdahl, F. (1988), ‘Tonal Pitch Space’, *Music Perception: An Interdisciplinary Journal* **5**(3), 315–349. Publisher: University of California Press.

Lerdahl, F. and Jackendoff, R. (1983), *A generative theory of tonal music*, the MIT press, Cambridge, Mass. London.

Meyer, L. B. (1956), *Emotion and meaning in music*, paperback ed., [nachdr.] edn, Univ. of Chicago Press, Chicago. Ill.

Plomp, R. and Levelt, W. J. M. (1965), ‘Tonal Consonance and Critical Bandwidth’, *The Journal of the Acoustical Society of America* **38**(4), 548–560.
Sadeghi, H., Allard, P., Prince, F. and Labelle, H. (2000), ‘Symmetry and limb dominance in able-bodied gait: a review’, *Gait & Posture* **12**(1), 34–45.

Schlenker, P. (2017), ‘Outline of Music Semantics’, *Music Perception* **35**(1), 3–37.

Sievers, B., Lee, C., Haslett, W. and Wheatley, T. (2019), ‘A multi-sensory code for emotional arousal’, *Proceedings of the Royal Society B: Biological Sciences* **286**(1906), 20190513.

Todd, J. T. (1983), ‘Perception of gait.’, *Journal of Experimental Psychology: Human Perception and Performance* **9**(1), 31–42.

Zaradkzi, L. (2021), Les évenements en sémantique linguistique et musicale, PhD dissertation, Université Paris Cité.
# A Predictions from stimuli

The target stimulus used throughout the experiment was used because of its intuitive capacity to evoke a walk.<fnref n="23" /> Still, there are theoretical reasons for which one could justify this stimulus based on (i) the mereological structure of a walking event, i.e. how subjects spontaneously represent or approximate a walking event hierarchically, (ii) the physical, physiological and mechanical analysis of walking and gait patterns from the literature, and (iii) a set of semantic rules that makes the correspondence between musical features and features of a walking event more explicit, i.e. how subjects convert musical information into a mental representation of a walking event.

Although other structures could be assigned to our perception of a walking event (i.e. other models could be considered for (i)), we show that the model we propose does not only make intuitive sense, but it is also consistent with the literature on gait physics. By combining this model with a semantic model from Schlenker (2017), we provide both experimental and theoretical reasons for which the stimulus evokes a walking event in the first place.

## Modeling the hierarchical structure of a walking event

Although there exists an extensive literature on gait physiology and mechanics, there are only few studies on the perception of gait movement. Yet, we will rely on findings from Cutting and Proffitt (1981) and Todd (1983) to approximate the perceived structure of a walking event, along with general theories on how events are hierarchically and structurally perceived (e.g. Jackendoff (2009)). We are indeed interested in how music can evoke a walking event as perceived in listeners (not a walking event per se, regardless of how it is mentally represented in listeners). Let's thus first assume that a walking event can be represented as four successive subevents $e_1$, $e_2$, $e_3$ and $e_4$ as shown in Figure 16. We will then show how the stimulus used in the experiment matches this hypothetical mental representation of a walking event and how this mapping is grounded in the physics of walking.

Following intuitions by Jackendoff (2009), we assume that a walking event is, as any event, hierarchically structured: we represent events as successions of discrete sub-events, some of which are more salient than others and constitute 'heads', which less important sub-events depend on. Intuitively, any walking event involves the alternation of two steps: the first step can be roughly decomposed in a first event $e_1$ during which both feet are on the ground (which roughly corresponds to the stance phase in gait analysis), and a second event $e_2$ in which body weight is transferred onto one single foot (which roughly corresponds to the swing phase in gait analysis). The same pattern is then repeated with a third event $e_3$ with both feet on the ground, and $e_4$ with the second foot on the ground.

## Modeling a walking event in terms of physical properties

But event structure is not all there is to a walking event, and we must now come up with a model of how physical properties evolve through a walking event, for we do not only want

<footnote n="23">²³Here, we borrow our methodology for selecting a specific stimulus from linguistics rather than from more standard approaches in cognitive psychology. In linguistics, it is indeed standard to rely on introspective judgments on the grammatically and/or the semantics of sentences to motivate research paradigms. Here, the stimulus plays the same role as any such sentence: the stimulus evoked a walk in the authors and informants prior to building the experimental design, and the remainder of the experiment tried to understand (i) how robust or systematic this effect was, and (ii) why it was or was not.</footnote>
<figure><img src="image_46.png" /><figcaption>Figure 16: Tentative model of a walking event with tree-like structure</figcaption></figure>

musical structure to map onto event structure, but we also want the interpretation of music to map onto our representation of the event. Following the literature on gait analysis, we use two descriptors of gait patterns: support and footedness.<fnref n="24" /> Support refers to whether one foot or two feet bear the body weight, i.e. whether one foot or two feet are on the ground. Footedness refers to lower-limb laterality: just as handedness refers to the asymmetry in the use of hands, footedness is linked to foot preference in performing certain tasks such as climbing a step or kicking a ball. Both support and footedness result in differences in how body weight will be spread onto the floor surface, and eventually in how stable the event will be perceived. Intuitively, stability is higher when both feet are on the ground and lower when one foot is swinging (it is harder to stand on one foot than on two feet). Also, if the front foot is the dominant foot, stability will be slightly higher than if the non-dominant foot is leading. These intuitions are summarized in Table 17.

<table><caption>Figure 17: Support, footedness and stability during a walking event</caption><thead><tr><th>Event</th><th>e<sub>1</sub></th><th>e<sub>2</sub></th><th>e<sub>3</sub></th><th>e<sub>4</sub></th></tr></thead><tbody><tr><th>Support</th><td>Double</td><td>Single</td><td>Double</td><td>Single</td></tr><tr><th>Front foot</th><td>Dominant</td><td>/</td><td>Non-dominant</td><td>/</td></tr><tr><th>Stability</th><td>++</td><td>-</td><td>+</td><td>-</td></tr></tbody></table>

<footnote n="24">²⁴For several analysis of footedness, see Gabbard and Iteya (1996) and Hart and Gabbard (1997), and Sadeghi et al. (2000) for a review on limb dominance; See also Hausdorff (2007) and Hayafune et al. (1999) for detailed analyses of pressure and force distribution in gait.</footnote>
Let us now formalize physical stability in a walking event. Let $S_n$ be the physical stability of event $e_n$. We can now formally describe the evolution of stability of a walking event $E = \langle e_1, e_2, e_3, e_4 \rangle$ defined as the succession of four sub-events based on Figure 17:

(1) If $E$ is a walking event, then $S_1 > S_3 > S_2 = S_4$.

We will now need to turn to the musical properties whose interpretation is likely to map onto physical stability. We will present two such properties in the following section: consonance, and chord proximity.

## Deriving musical properties to evoke a walking event

The distinction between consonance and chord proximity is of particular importance with regards to how both properties are semantically interpreted. In the model proposed by Schlenker (2017), the corresponding musical property is harmonic stability, which comes with the following semantic rule: the more stable the musical event, the more stable the denoted event. However, the notion of harmonic stability has been disputed in both the musicological and psychological literature and seems to refer to a concept from music theory building on the notion of harmonic function, rather than a perceptual fact. Generally, harmonic stability refers to the harmonic function of a chord within the context of a given reference key (what Lerdahl (1988) refers to as a 'tonal region'): in the key of C (the context, the tonal region), a G chord has the function of a dominant, which is commonly used as a way to both get away from the tonic and get back to it in standard harmonic progressions in Western tonal music. However, the concept of harmonic stability does have a psychological counterpart. Arguably, the harmonic stability of a chord in a given tonal context amounts to assessing the proximity that one chord entertains with the tonic chord establishing the context: a given chord $C_1$ is perceived as stable in the tonal context provided by a chord $C_2$ if $C_1$ is perceived to be closely related to $C_2$. Crucially, chord proximity is not directly related to chord consonance, and most studies on chord proximity have only established the perceived psychological distance between consonant chords, for a good reason: the more dissonant the chords, the harder it gets to identify them in the tonal pitch space, hence the harder it gets to compute chord proximity. We will thus now tease apart these two notions, in particular because it seems that both properties can be semantically interpreted as physical stability, just as harmonic stability. We will first provide some information about consonance, before turning to chord proximity. We finally provide a semantic analyses of these two properties and derive the predictions for our stimuli.

## Consonance

Consonance refers to the subjective experience of pleasantness when two (or more) simple or complex tones are heard at once. One of the most consensual observations about interval consonance is that consonant intervals tend to be such that the frequency ratio of the two tones can be expressed with small integers: the smaller the integers, the more consonant the interval. This relies on the core fact that tones have harmonics (also named overtones), i.e. musical sounds (complex tones) contain partials at frequencies that are different from the root's. Helmoltz (1877) famously claimed that consonance depends on both (i) the difference in frequencies (aka interval width), and (ii) the pitch range of the constituent tones: a same interval is judged to be rougher in lower frequency ranges. As soon as two tones do not share a same fundamental frequency (i.e. the lowest partial), acoustic beats are created, i.e.
variations in perceived volume due to the periodic combination of the two sound waves with different frequencies, which results in an impression of roughness in the listener. The more complex the frequency ratio (i.e. the higher the integers), the more beats are created, hence the more dissonant the chord. Although dominant then and still now, this view was later updated, in particular by Plomp and Levelt (1965), who showed that the best predictor of whether an interval will be perceived as consonant or dissonant is actually not the interval width combined with the pitch range of each constituent tone, but the relationship with their critical bandwidth.

One must note that although this literature primarily focuses on the consonance and dissonance of intervals (made of two sounds), our experiment and the stimuli we tested involved both intervals and chords (made of more than two sounds; the word ‘triads’ being standard for three-tones chords). However, it was already acknowledged by Helmoltz that whatever theory accounts for interval consonance should generalize to chords consonance: the consonance of a chord is determined by the consonance of its constituent intervals. Typically, this generalization explains why only major and minor triads, along with their inversions, are considered consonant: those are the only ones that are made of the intervals which also sound consonant in isolation.

## Chord proximity

Consonance can be said to be an inherent property of musical events such as intervals or chords: for instance, a listener can rate the consonance level of a single chord. However, it has been both taken for granted in music theory and shown in music psychology that listeners do have intuitions about how ‘alike’, ‘similar’ or ‘close to one another’ two chords sound, regardless of whether they are heard as consonant or dissonant. In other words, listeners are able to compute a certain ‘distance’ between two chords: this is the notion of chord proximity. Note that because this phenomenon only arises when two musical events are heard, it has to be ordinal in nature: one must always rate the proximity of at least two musical events. Among several theories available, Lerdahl’s *Tonal Pitch Space* is among the ones that has been able to capture most empirical findings in psychology, along with most assumptions assumed in classical Western tonal music theory (Lerdahl, 1988). Roughly, Lerdahl’s idea was that it is possible to predict the perceived distance or proximity between any two chords (either in a same tonal ‘region’, i.e. key, or from different regions) through a rule summing (i) the number of fifths separating the two chords on the standard circle of fifths (if the two chords are from a same region), (ii) the number of tones in common between the two chords and (iii) the number of fifths separating the two regions on the circle of fifths (if the two chords are from different tonal regions). Although very simple, this additive rule has proved surprisingly useful to account for proximity judgments of chords in listeners. In this paper, we therefore use Lerdahl’s theory from TPS as a baseline for characterizing chord proximity in our own stimuli.

## Semantic interpretation of consonance and chord proximity

In Schlenker (2017), harmonic stability can be semantically interpreted in two ways, either in terms of (i) physical stability of the denoted object, event or situation and (ii) emotional stability of the listener. Examples from real music show that changing the harmonic structure of a piece results in this piece evoking situations with different stability patterns. For instance, going from a tonic chord (the most ‘stable’ one in a reference key) to a dominant chord (the
second most stable one in a same key) can denote an event that transitions from a stable state to a less stable state. Now that we have refined the underlying assumptions through the chord consonance/chord proximity distinction, we will generalize these assumptions on the interpretation of harmonic stability to chord consonance and chord proximity. We will assume that both consonance and proximity can be interpreted in terms of physical or emotional stability according to the following rules:

<table><caption>Figure 18: Possible rules for the interpretation of consonance and chord proximity</caption><thead><tr><td>Property</td><td>Semantic rule</td></tr></thead><tbody><tr><td>Harmonic function (Schlenker)</td><td>The more harmonically stable the musical event, the more stable the physical or emotional state of/in the denoted event</td></tr><tr><td>Consonance</td><td>The more consonant the musical event, the more stable the denoted event</td></tr><tr><td>Chord proximity</td><td>The closer two chords are in the tonal pitch space, the smaller the difference in stability level of the two denoted events</td></tr></tbody></table>

The reason for which we have introduced chord proximity might now become clearer: chord proximity captures cases where the notion of harmonic stability cannot be used. In particular, it states that it is possible to interpret chord proximity as a difference in stability in two denoted events when no tonal context is available, a prediction that a model based on harmonic function does not make. However, there are cases where despite the existence of a tonal context, the distance to the tonic remains constant through a stimulus: this could be said to be true of our target stimulus, which only involves musical events derived from a C chord. In this case, consonance helps predict which of two C chords will be perceived as more stable, based on their inversion state.

It is now possible to reconstruct the rule on harmonic stability from the rule on chord proximity by having one of the two musical events denote the tonic. It follows from the rule on chord proximity that

(2) The closer a chord to the tonic in the tonal pitch space, the more similar its level of stability to that of the tonic.

Given that the tonic is associated with maximal stability, it follows from (2) that:

(3) The further away a chord is from the tonic chord in the tonal pitch space (through a rule computing distance such as that of Lerdahl's in TPS), the less stable it is.

When a tonal context is available, the semantic rule can therefore be rewritten:

(4) Let $m_1$ and $m_2$ be two musical events in a tonal context $C$, $d_1$ and $d_2$ the distance to the tonic of $C$ in the tonal pitch space, $e_1$ and $e_2$ two events with levels of physical or emotional stability $S_1$ and $S_2$, respectively. If $d_1 < d_2$, then $S_1 > S_2$

If no tonal context is available, then the rule has to be derived from consonance instead:

(5) Let $m_1$ and $m_2$ be two musical events in a tonal context $C$, $c_1$ and $c_2$ their respective consonance, $e_1$ and $e_2$ two events with levels of physical or emotional stability $S_1$ and $S_2$, respectively. If $c_1 < c_2$, then $S_1 < S_2$.<fnref n="25" />

<footnote n="25">²⁵Note that the previous rule can also operate in combination with the first one when a tonal context is available.</footnote>
Now, let us apply these rules to derive the predicted denotations of the stimulus discussed in Section 5:

<figure><img src="image_47.png" /><figcaption>Figure 19: A stimulus which should refer to a walk based on RPOs</figcaption></figure>

The music provides no tonal context. We will thus reason by limiting any interpretation in terms of stability to the interpretation of consonance.<fnref n="26" />

Let our musical event be:

$M = < m_1, m_2, m_3, m_4 >$

with $m_1 = Cm(6/4)$, $m_2 = Cl$, $m_3 = F\#m(5)$ and $m_4 = Cl$

Let $C_n$ be the consonance of event $n$.

Following our definition of consonance based on Helmoltz (1877), chords in their root position are perceived as more stable than inversed chords. So: $C_3 > C_1$.

A dissonant cluster is necessarily more dissonant that any perfect chord. It follows that: $C_3 > C_1 > C_2 = C_4$

A final twist is required at this point. Because the stimulus in Figure 19 was repeated, it is possible to have the stimulus start on $m_3$; hence $m_3$ is mapped onto $e_1$, and $m_1$ is mapped onto $m_3$. If we reassign indexes to take this into account, it follows that:

$$C_1 > C_3 > C_2 = C_4$$

We earlier defined a walking event as being such that:

(1) If $E$ is a walking event, then $S_1 > S_3 > S_2 = S_4$.

We here see that the ordering in consonance of the stimulus in Figure 19 matches the ordering in stability of a walking event. So, RPOs predict that this stimulus can denote a walking event. Now, let us consider our target stimulus once again:

<footnote n="26">²⁶Note that this is an approximation used for simplicity and clarity in the model; it is for instance in this case possible to calculate the distance between chords on beats 1 and 3 based on Lerdahl's rules from TPS, and then simply consider that the cluster on beats 2 and 4 necessarily denotes a less stable event.</footnote>
<figure><img src="image_48.png" /><figcaption>Figure 20: Target stimulus</figcaption></figure>

Similarly, it is possible to compute the requirements on the physical stability of the denoted event of our target stimulus thanks to the very same rules. Let $M$ be our musical event.

$$M = < m_1, m_2, m_3, m_4 > \text{ with } m_1 = C, m_2 = C(6), m_3 = G, m_4 = C(6)$$

Let $d_n$ be the distance of event $n$ to the tonic $C$. Let $S_n$ be the level of physical stability of a denoted event. Then $d_3 > d_1$ (G is further away from C than C from itself) and it follows from rule (4) that $S_3 < S_1$

Now, given that a single tone is necessarily more consonant than a 6 chord, it follows from rule (4) that:

$$C_2 = C_4 < C_3, \text{ therefore } S_2 = S_4 < S_3$$

$$C_2 = C_4 < C_1, \text{ therefore } S_2 = S_4 < S_1$$

$$\begin{cases} S_2 = S_4 < S_3 \\ S_2 = S_4 < S_1 \\ S_3 < S_1 \end{cases}$$

Therefore: $S_2 = S_4 < S_3 < S_1$

Hence, the ordering in consonance of the target stimulus matches the ordering in stability of a walking event. Rules of preservation of orderings on chord proximity and consonance thus predict that a walking event is a possible denotation for our target stimulus, just as it was for the stimulus in 19.<fnref n="27" />

<footnote n="27">²⁷Another analysis of the target stimulus has been given by Zaradkzi (2021) reusing Schlenker’s notion of harmonic stability in combination with Larson’s theory of musical forces (Larson, 2012): The target stimulus is the repetition of a single bar made of four musical events: a single C, a C major chord in its first inversion state (6th), a bass G, and a C major chord in its 1st inversion state again. Harmonically, only one chord is present in the entire stimulus: the C major chord, made of the three notes C, E and G. It is usually acknowledged in music theory (and a sound perceptual hypothesis as well) that the most ‘stable’ form of a chord is its fundamental state. Here, the most stable event of the four has to be the bass C which is the tonic of the C scale. Comparatively, the bass G, which is the dominant of the same C scale, is usually considered to be a bit less stable (and often felt to be ‘attracted’ to the tonic for this very reason). Hence, in the key of C, in which the target stimulus is written, the bass C is a bit more stable than the bass G. Now, assessing the relative stability of the higher-pitch C chord requires an extra-step. We mentioned that this chord is in its 2nd inversion state, which sounds less stable because of the presence of the 6th (compared to the presence of a 5th in the fundamental state, that corresponds to a frequency ratio of smaller integers hence perceived as more consonant). Although the perceived stability of that chord depends on the musical event that comes right before it, the fact that (i) it is a chord and not a single tone and (ii) is played in a higher register strongly supports that it should be heard as less stable (Schlenker, 2017; Larson, 2012). Consequently, the evolution of harmonic stability in</footnote>
# B Statistics

In this section, we present the detailed results of the statistical tests performed on the data to assess the significance of the contrasts in endorsement of the target stimulus and other stimulus types. For each pair, we performed Student's T-tests to assess whether the endorsement of the target stimulus was significantly above chance level, and report the corresponding t-scores and p-values. To check for statistical difference in endorsement in the confronting stimuli, we also performed pairwise comparisons of each pair of confronting stimuli, and report (i) the difference in endorsement, (ii) the lower and upper bounds of the 95% confidence interval, and (iii) the associated p-value.

## Target vs control stimuli

<table><thead><tr><td>Control</td><td>t-score</td><td>p-value</td></tr></thead><tbody><tr><td>Control1</td><td>t(27*) = -14.984</td><td>&lt; 0.001***</td></tr><tr><td>Control2</td><td>t(31) = -10.52</td><td>&lt; 0.001***</td></tr><tr><td>Control3</td><td>t(29) = -9.0157</td><td>&lt; 0.001***</td></tr></tbody></table>

*Note that the df's differ because participants did not necessarily rate the same control stimuli.

<table><caption>Figure 21: Significance tests on the preference for the target stimulus over control stimuli</caption><thead><tr><td>Pair</td><td>diff</td><td>lwr</td><td>upr</td><td>p adj</td></tr></thead><tbody><tr><td>ctrl1-ctrl2</td><td>5.303571</td><td>-6.915831</td><td>17.52297</td><td>0.5569748</td></tr><tr><td>ctrl1-ctrl3</td><td>6.878571</td><td>-5.529455</td><td>19.28660</td><td>0.3869565</td></tr><tr><td>ctrl2-ctrl3</td><td>1.575000</td><td>-10.425203</td><td>13.57520</td><td>0.9474760</td></tr></tbody></table>

<caption>Figure 22: Pairwise comparisons of control stimuli</caption>

## Target vs degraded stimuli

<table><caption>Figure 23: Significance tests on the preference for the target stimulus over degraded stimuli</caption><thead><tr><td>Degraded stimulus</td><td>t-score</td><td>p-value</td></tr></thead><tbody><tr><td>DS1</td><td>t(26) = -8.9064</td><td>&lt; 0.001***</td></tr><tr><td>DS2</td><td>t(25) = -5.5438</td><td>&lt; 0.001***</td></tr><tr><td>DS3</td><td>t(25) = -3.2275</td><td>&lt; 0.005**</td></tr><tr><td>DS4</td><td>t(28) = -15.146</td><td>&lt; 0.001***</td></tr><tr><td>DS5</td><td>t(26) = -6.3371</td><td>&lt; 0.001***</td></tr></tbody></table>

<footnote>the target stimulus follows the same pattern as in the excerpt shown in Fig. 19. For a further detailed formal analysis of how Larson's theory of musical forces can be integrated with Schlenker's theory, see de Neeve (2021).</footnote>
<table><caption>Figure 24: Pairwise comparisons of degraded stimuli</caption><thead><tr><th>Pair</th><th>diff</th><th>lwr</th><th>upr</th><th>p adj</th></tr></thead><tbody><tr><td>PM2-PM1</td><td>10.403134</td><td>-6.0100208</td><td>26.8162886</td><td>0.4052957</td></tr><tr><td>PM3-PM1</td><td>15.941595</td><td>-0.4715593</td><td>32.3547502</td><td><u>0.0613849</u></td></tr><tr><td>PM4-PM1</td><td>-6.033206</td><td>-22.0080199</td><td>9.9416087</td><td>0.8339660</td></tr><tr><td>PM5-PM1</td><td>4.666667</td><td>-11.5909096</td><td>20.9242430</td><td>0.9319189</td></tr><tr><td>PM3-PM2</td><td>5.538462</td><td>-11.0288107</td><td>22.1057338</td><td>0.8868346</td></tr><tr><td>PM4-PM2</td><td>-16.436340</td><td>-32.5694593</td><td>-0.3032198</td><td><u>0.0435702*</u></td></tr><tr><td>PM5-PM2</td><td>-5.736467</td><td>-22.1496220</td><td>10.6766875</td><td>0.8695104</td></tr><tr><td>PM4-PM3</td><td>-21.974801</td><td>-38.1079208</td><td>-5.8416813</td><td><u>0.0022760**</u></td></tr><tr><td>PM5-PM3</td><td>-11.274929</td><td>-27.6880835</td><td>5.1382260</td><td>0.3223365</td></tr><tr><td>PM5-PM4</td><td>10.699872</td><td>-5.2749420</td><td>26.6746866</td><td>0.3480923</td></tr></tbody></table>

## Target vs RPOs-only stimuli

<table><caption>Figure 25: Significance tests on the preference for the target stimulus over stimuli satisfying RPOs only</caption><thead><tr><th>Stimulus</th><th>t-score</th><th>p-value</th></tr></thead><tbody><tr><td>Ordering only_1</td><td>t(44) = -6.9547</td><td>&lt; 0.001***</td></tr><tr><td>Ordering only_2</td><td>t(44) = -8.9731</td><td>&lt; 0.001***</td></tr></tbody></table>

<table><caption>Figure 26: Pairwise comparisons of stimuli complying with RPOs</caption><thead><tr><th>Pair</th><th>diff</th><th>lwr</th><th>upr</th><th>p adj</th></tr></thead><tbody><tr><td>Ordering1/ordering2</td><td>-4.2</td><td>-14.97398</td><td>6.573976</td><td>0.4405926</td></tr></tbody></table>

## Target vs Enriched version

<table><caption>Figure 27: Significance tests on the preference for the target stimulus over the enriched version with additional 8th note.</caption><thead><tr><th>Stimulus</th><th>t-score</th><th>p-value</th></tr></thead><tbody><tr><td>Target</td><td>t(44) = -2.1008</td><td>0.04143*</td></tr><tr><td>Control 1</td><td>t(31) = 14.992</td><td>&lt; 0.001***</td></tr><tr><td>Control 2</td><td>t(28) = 13.508</td><td>&lt; 0.001***</td></tr><tr><td>Control 3</td><td>t(28) = 10.591</td><td>&lt; 0.001***</td></tr><tr><td>DS1</td><td>t(12) = 3.4453</td><td>0.004847**</td></tr><tr><td>DS2</td><td>t(17) = 1.7053</td><td>0.1063 (ns)</td></tr><tr><td>DS3</td><td>t(19) = 1.5013</td><td>0.1497 (ns)</td></tr><tr><td>DS4</td><td>t(15) = 3.6575</td><td>0.002334**</td></tr><tr><td>DS5</td><td>t(19) = 0.090107</td><td>0.9291 (ns)</td></tr></tbody></table>
Effect of musical training

In this section, we present the details of the statistical tests performed on the data testing for the effect of musical training. Musical training was assessed through direct self-assessment by participant and coded *via* a binary variable (0 for no training, 1 for some to a lot of training). For each category of stimuli, we ran two linear models on the data either including training as a predicting variable or not. The results we report in the table below were computed through an ANOVA performed between the two models. If training had an effect on the responses, then we expected the model containing training as a variable to fit the data significantly better than the other one. However, as seen in the table, no test was statistically significant, suggesting that musical training never significantly affected the responses participants gave for any category of stimulus.

<table><caption>Figure 28: Significance of the difference in predictions in two models (with or without musical training)</caption><thead><tr><td>Stimulus category</td><td>Sum of squares</td><td>p-value</td></tr></thead><tbody><tr><td>Controls</td><td>-23.314</td><td>0.8957</td></tr><tr><td>Degraded stimuli</td><td>-72.012</td><td>0.8323</td></tr><tr><td>Ordering only</td><td>-2105.1</td><td>0.3012</td></tr><tr><td>Extra-8<sup>th</sup> version</td><td>-9.3577</td><td>0.9382</td></tr></tbody></table>
# Chapter 5

## Music-induced effects on body motion

### Purpose

The previous chapter presented experimental data showing that listeners can represent a walking event from a musical sequence as long as that sequence meets certain conditions. In this chapter, we present an experimental paradigm exploring the extent to which musical properties affect real walking events (when listeners are walking). More specifically, we tested how pitch affects gait patterns, and found that pitch height correlates with properties of gait, which suggests that musical meaning might interact with the motor system.
CRACKING THE PITCH CODE OF MUSIC-MOTOR SYNCHRONIZATION USING DATA-DRIVEN METHODS *

Léo Migotti<fnref n="1" />, Quentin Decultot<fnref n="2" />, Pierre Grailhe<fnref n="2" />, and Jean-Julien Aucouturier<fnref n="2" />

ABSTRACT

The study of auditory-motor synchronization with music has been so far mostly concerned with timing. For instance, research has established that people are able to spontaneously coordinate with musical beat when walking or running (Styns et al., 2007). Yet, music is more than a metronome, and the relation between the spectral dimension of music, i.e. parameters such as its pitch, timber, or harmonic structure, and simultaneous motion remains nearly unknown. Here, we introduce a novel data-driven paradigm in which participants were asked to walk on a treadmill while listening to a large variety of musical tones systematically varied in pitch. Using analysis techniques inspired by psychophysical reverse correlation, we show that participants' gait patterns while walking to music spontaneously encode pitch height: despite being instructed to simply synchronize in time, participants steps were both longer and heavier on tones with lower pitches. These findings reveal that, similarly to time perception, pitch cognition is not purely 'disembodied' and suggest that listeners' spontaneous motor reactions to pitch might ground their ability to mentally represent music.

**Keywords** music | auditory-motor synchronization | pitch | walking | gait

<footnote n="1">¹Institut Jean Nicod (CNRS - EHESS - ENS) - Département d'Etudes Cognitives, Ecole Normale Supérieure, Paris, France - PSL Research University</footnote>

<footnote n="2">²Department of Robotics and Automation - FEMTO-ST Institute (CNRS/Université de Bourgogne Franche Comté), Besançon, France</footnote>

<footnote>* *Contributions*: LM, PG, JJA designed the experiment; QD and JJA developed the experimental apparatus and collected data; LM, QD, JJA analysed the data; LM and JJA wrote the manuscript, with contributions from PG.</footnote>
# Contents

<table><tr><td>1</td><td>Introduction</td><td>170</td></tr><tr><td>2</td><td>Results</td><td>172</td></tr><tr><td>3</td><td>Discussion</td><td>175</td></tr><tr><td>4</td><td>Materials and Methods</td><td>179</td></tr><tr><td>5</td><td>Acknowledgments</td><td>184</td></tr><tr><td>6</td><td>Bibliography</td><td>185</td></tr></table>
# 1 Introduction

Music in all human cultures is often experienced in motion, be it in dance, trance, military marches or lullabies (Blacking, 1974) and, even when we stand still, how we hear it remains strongly influenced by our motor system (Repp and Su, 2013). Because both music and movement unfold in time, the idea that there should exist a temporal mapping between the two appears intuitive, and indeed a vast body of research has shown that walking (Styns et al., 2007) and running speed (Van Dyck et al., 2015) spontaneously align with beat frequency (tempo). More generally, an extraordinarily rich literature has investigated people's ability to move in synchronization with external auditory stimulation (Repp and Su, 2013; Thaut and Abiru, 2010), be it with classical finger tapping studies (Van Noorden and De Bruyn, 2009), limb movement (Torre and Delignières, 2008) or dancing (Toiviainen et al., 2010). Expressive timing in music performance, such as the final *ritardando* in the Baroque and Romantic period (Desain et al., 1996), also appear to share much of the temporal dynamics of motion (Sievers et al., 2013).

Yet, music is more than a sophisticated metronome and, while we know a lot about how music maps to motion ‘horizontally’ (in the time domain), little is known about whether the other ‘vertical’ (i.e. spectral) dimensions of music such as pitches, harmony and timbre relate to movement.

Indeed, while theories of embodied music cognition (Leman, 2007; Godøy and Leman, 2010) have been quick to extend beyond temporal/rhythmic parameters to propose that spectral aspects of music such as musical tension (Korsakova-Kreyn, 2018), pitch contours (Eitan and Granot, 2006) and consonance (Alves, 2012) also share cognitive representations with the physical features of body movement, empirical demonstrations of such links remain few and almost entirely limited to explicit judgements. For instance, in forced association tasks between music and dynamic visual stimuli, listeners tend to associate rising pitch contours with ascending motion (Eitan and Granot, 2006); random pitch intervals with jerky physical motion (Sievers et al., 2013); dissonant and unstable chords with motion having low physical stability (Schlenker, 2017, 2019); and alternations of consonant and harmonically-close chords
with the imagined situation of someone walking (the “Sorcerer’s Apprentice effect” (Migotti and Zaradzki, 2019)).

Such experiments demonstrate that music is able to evoke motion percepts or interpreta-
tions in listeners, which in itself is relatively uncontroversial (Tan and Kelly, 2004; Schlenker,
2017, 2019; Athanasopoulos and Moran, 2013) and is also reminiscent of well-studied pitch-
space associations (Rusconi et al., 2006). It is however an entirely different question whether
these associations are based on sensorimotor representations that are able to directly interact
with the human motor system. For instance, if listeners in (Migotti and Zaradzki, 2019)
reported less association to an imaginary walking situation for musical sequences including
dissonant than consonant chords, does this mean that dissonant events would also disrupt
their actual gait if they were to synchronize their steps to them?

In Komeilipoor et al. (2020), musical dissonance was found to alter the finger-tapping
performance of participants asked to synchronize to either two dissonant or two consonant
chords: in tasks where stimuli were dissonant, participants were less accurate in tapping,
and remained less steady and regular after the sequence had stopped. However, in such
a paradigm with a limited number of stimuli presented in separate blocks, it is difficult
to conclude that it was the spectral/tonal properties of each individual sound event that
matched and interfered with the corresponding motor event, or whether dissonance induced
a more general cognitive or attentional load which interfered with the task on a broader and
non-specifically-motor way (Bodner et al., 2007).

Here, we introduced a data-driven paradigm in which participants were asked to walk on a force-sensing treadmill while listening to a large variety of musical tones systematically varied in pitch. Such a paradigm has tremendous advantages for our research question. First, because each tone can be time-stamped to the X-Y-Z force time-series of the corresponding step, we were able to investigate how each individual sound event interferes with the corresponding motor event, similarly to the event-related methodology in neurophysiology. Second, because we can have participants step along with thousands of successive tones with randomly different pitch, the resulting data is amenable to analysis with psychophysical methods
<figure><img src="image_49.png" /><figcaption>Figure 1: **Steps that occurred on lower-pitch notes exhibited more positive vertical force in their later, propulsive phase.** Green: Average vertical (top), medio-lateral (middle) and anteroposterior (bottom) force data for all participant steps (M=668 steps per participant). Participants' step data displayed the expected sad-dle shape in the vertical and medio-lateral force dimensions. Blue/orange: Classification images indicating the direction of the statistical association between force at each time point and the z-scored, log-transformed pitch of the corresponding note, before (blue) and after (orange) note onset (marked by dashed line and note icon). Classification images showed a cluster of statistically-significant negative associations between pitch and vertical and medio-lateral force, post-onset, in the toe-off phase of the step.</figcaption></figure>

inspired by reverse-correlation and we are able to infer what exact part of a participant's
steps is statistically influenced by pitch, in a purely data-driven manner.

## 2 Results

We let N=20 (female=9, male=11) young, Western, educated participants walk on a force-sensing treadmill while they listened to a 10min. sequence of musical tones (piano notes), whose pitch was randomly sampled within the two medium octaves of the western 12-tone chromatic scale (among 240 possible tones between C4/255.8Hz and B5/1045.8Hz, sampled
at a frequency resolution of 10% of a semitone). Each successive tone in a sequence was
repeated three consecutive times, at a rate of 75 events per minute (IOI=.8s). We then
epoched the force time-series into individual steps and, in the manner of event-related analysis
in electrophysiology, paired each step with its corresponding sound event. This resulted in an
average M=668 trials per participant (see *Materials and Methods* for details). Six participants
were removed because more than 20% of their steps were not synchronized with a musical
event, leaving N=14 (female=6, male=8) for subsequent analysis.

Participants’ step data displayed a saddle shape in the vertical and medio-lateral force dimensions and a progressive transition from anterior to posterior force, which is the expected shape for a stance including an initial, braking heel strike followed by a final propulsive ‘toe off’ (see e.g. Vaverka et al. (2015)). Step duration was stable at M=105ms (SD=71ms). Consistent with similar synchronization tasks in the literature (Repp and Su, 2013), we found that participants’ steps steadily anticipated the onset of the notes, with a mean onset time occurring M=608ms (SD=83ms) after the start of support of the corresponding step, roughly at a local minimum of antero-posterior and latero-medial forces (Figure 1).

To analyze the relation between step force data and musical pitch, we used a data-driven
technique inspired by the psychophysical method of “classification images” (Murray, 2011).
Classification images indicate the strength of the statistical association between force at each
time point and the z-scored, log-transformed pitch of the corresponding note: at each time
point, we tested whether the images differed from zero with one-sample t-tests (see *Materials
and Methods* for details).

Participants’ classification images showed a cluster of statistically-significant negative
associations between musical pitch and vertical force from t=800 to 1000ms (t(13) ∈ [-2.20,-
3.58], all ps < .046; Figure 1), showing that steps that occurred on lower notes exhibited
more positive vertical force in their later, propulsive phase. This cluster co-occurred with a
similarly negative association in the latero-medial dimension (from t=780 to 880ms; t(13)
∈ [-2.24,-3.71], all ps < .043), indicating a medial direction consistent with the final toe off.
There was no association between musical pitch and participants’ steps in the antero-posterior
dimension.
We then grouped each participant's trials and computed separated classification images for steps that occurred on the first, second and third repetition of each note. This revealed that the above effect was driven by negative associations that occurred specifically the second and third consecutive time any given note was heard and stepped onto. In the vertical force dimension, step 2 had a significant negative cluster from t=710 to 810ms (t(13) ∈ [-2.23,-2.99], all ps < .044) and step 3 from 810 to 1060ms (t(13) ∈ [-2.18, -4.01], all ps < .049), but none in step 1 (Figure 2-top).

This pattern of data is consistent with the interpretation that participants' steps performed on tones with lower pitch were heavier (showing more vertical force), but also that these steps were longer: because classification images were computed by aggregating steps of varied duration, it is possible that steps that had (positive) vertical force for longer amounts of time occurred more often for lower pitch, and that shorter steps, whose vertical force reached zero earlier, were conversely associated with higher pitch. For a confirmatory analysis, we therefore computed separate repeated-measure correlations (Bakdash and Marusich, 2017) between the (log-transformed) pitch in each participant's trials and the weight (computed as the mean vertical force between 800 and 1000ms) and duration of the corresponding steps: both correlations were significant at step 3 (weight: r(2291)=-0.043 [-0.08, -0.0], p=.03; duration: r(2291)=-0.066 [-0.11, -0.03], p=.001; Figure 3). In addition, to verify that musical pitch had an influence on step weight beyond the effect on step duration, we repeated the classification image analysis by normalizing all steps to 100% duration (figure 2-bottom). Classification images in the vertical dimension continued to exhibit a significant negative cluster at step 2 (from t=670 to 740ms, t(13)∈[-2.23,-3.72], all ps < .044), and repeated-measure correlations between log pitch and weight in the same range (700-750ms) remained consistent, albeit non significant: r(2286)=-0.034 [-0.08, 0.01], p=.09.

All in one, we therefore found converging evidence that participants' steps were *both* longer and heavier when they stepped on musical notes with lower pitch. The effect on vertical force when lowering pitch from B5 to C4 was M=+3.6% [3.05%, 4.19%], and the effect on duration was M=+1.4ms [1.19ms, 1.61ms].
<figure><img src="image_50.png" /><figcaption>Figure 2: Negative associations between vertical force and pitch that occurred specifically the second and third consecutive time any given note was heard and stepped onto, and remained even when the effect of step duration was normalized. Top: Classification images between vertical force at each time point (actual duration, in ms) and the z-scored, log-transformed pitch of the corresponding note, before (blue) and after (orange) note onset; computed separately for the first, second and third consecutive repeat of each note. Bottom: Classification images in the same conditions, computed by normalizing all steps to 100% duration.</figcaption></figure>

Finally, to test whether the influence of pitch was an effect of musical expertise, we tested for possible differences between the classification images of musically- and non-musically trained participants, using paired t-tests at each time point. There was no difference between both types of participants at any time point, for any of the effects we tested, regardless of whether we used participants’ reported years of training or a standard test of musicianship (Gold-MSI) as a cutoff measure (e.g. vertical force, all ps > .55 from t=80 to 100ms).

# 3 Discussion

To investigate whether musical pitch influences the gait parameters of simultaneous motion, we used a data-driven paradigm in which participants were asked to walk on a treadmill while listening to a large variety of musical tones systematically varied in pitch. Using analysis techniques inspired by psychophysical reverse correlation, we show that participants’ gait patterns while walking to music spontaneously encode pitch height: despite being instructed
<figure><img src="image_51.png" /><figcaption>Figure 3: Confirmatory analysis that participants' steps performed on tones with lower pitch were heavier but also longer. Left: Negative repeated-measure correlation between the (log-transformed) pitch in each participant's trials and the toe-off weight (computed as the mean vertical force between 800 and 1000ms) of the corresponding step. Right: Negative repeated-measure correlation between the log-transformed pitch in each participant's trials and the duration of the corresponding step.</figcaption></figure>

to simply synchronize in time, participants steps were both longer and heavier on tones with lower pitches. This effect was demonstrated vertically and latero-medially in the toe-off stage of the steps (Figure 1); it occurred specifically the second and third consecutive times any given note was heard (Figure 2); and it was present similarly in musically- and non musically-trained participants.

Despite our analysis being entirely data-driven, the temporal extent of the effects found here (from 700 to 1000ms within step) was consistent with the constraints holding on the sensory-motor system. First, none of the effects occurred before the onset of the corresponding sound event, which was on average around 600ms within step. Second, the fact that the effect of a new note did not occur in the first time this pitch was heard is consistent with the sensory latency of processing musical pitch (Godey et al., 2001) and the motor latency of initiating walking motor commands (Sinkjær et al., 2000), which would collectively leave little time for the central nervous system to initiate any modulation of walking gait within the 100-400ms after the sound is first heard. In a comparable psychoacoustic study (Kadosh et al., 2008), motor reaction times (button press) measured in response to pitch deviations of
the range used here (2 to 13 semitones) were in the range of 750-1100ms post-onset. At a rate of 75bpm (IOI=800ms), such latencies would correspond to the range [-50ms, + 300ms] around the onset of the *next* consecutive note event, which is very similar to what we see here.

Our finding that our participants associate low pitch with longer and heavier motion
is interesting to discuss in the context of the larger theoretical debate whether music can
be said to have meaning (Leonard, 1956) and a semantics (Schlenker, 2017, 2019). Indeed,
our results are reminiscent of a number of other studies reporting cognitive associations
between musical pitch and other spatial concepts, such as elevation - the SMARC effect
(Rusconi et al., 2006) - or speed (Broze and Huron, 2013); between falling pitch contours
and descending motion (Eitan and Granot, 2006); between the repetition of low-pitched
sound and quantity of motion in dance music (Van Dyck et al., 2013); between consonant
and stable chords and motion stability (Schlenker, 2017, 2019); and between alternations
of consonant and harmonically-close chords and the imagined situation of someone walking
(Migotti and Zaradzki, 2019). That musical pitch can be interpreted as *representing* weight
or stability therefore reinforces an emerging view that music is not only iconically referential,
e.g. in the mapping between melodies and animal characters in Prokofiev's Peter and the
Wolf (Trainor and Trehub, 1992), but can support a rich typology of semantic inferences
about the properties and actions of non-auditory objects (Schlenker, 2019), in the same sense
as language, gestures or visual animations (Tieu et al., 2019).

Even though the emergence of such representations is often studied in a musical context,
one may question whether they are specifically musical, or of a more generic cognitive
nature. Many existing cognitive mappings between, e.g., pitch and speed (Broze and Huron,
2013) are thought to result from the associative learning of events co-occurring in the
natural environment (e.g. doppler effects (Mcbeath and Neuhoff, 2002)). For the behavior
demonstrated here, it appears plausible that participants have developed sensory associations
between heavy objects that hit the ground forcefully, and the resulting low or deep impact
sounds, and that there is nothing intrinsically musical to this mapping. This suggests, as
others have also proposed, that some of the semantics of music is “continuous” with generic
auditory cognition (Schlenker, 2017), in the same way that musical emotions are thought to
be partly evoked by mimicking the acoustic properties of non-musical emotionally significant
events, such as thunder (Ma and Thompson, 2015), or emotional vocalizations (Bedoya et al.,
2021). In the present study, the fact that musically and non-musically trained participants
showed similar effects shows that musical expertise is in any case not necessary for the
association between pitch and stepping weight/duration to exist.

In comparison with previous work, we find remarkable that the present results did not
emerge from self-report tasks asking participants to explicitly compare stimuli, but from
physiological data in an implicit walking task where participants were not instructed to
pay attention to the pitch quality of the musical events. This suggests that part of these
associations do not only derive from ‘disembodied’ mental representations, but are able to
recruit sensorimotor representations that directly interact with the human motor system,
in our case modulate the force characteristics of participants’ gait. From the point of view
of music semantics, the fact that musical pitch sequences should modulate overt walking
behavior suggests that music might not only able to refer to or denote external objects or
events (Putman, 1987), but also internal events (i.e. happening in the listener’s body): if
music has meaning, then maybe part of this meaning is about the listeners themselves. This
is reminiscent of the question whether the smile-like spectral features of certain musical
instruments can induce positive emotions and the imitation of a smile in listeners (Bedoya
et al., 2021).

A related question concerns the cognitive ‘chronometry’ of the emergence of music-semantic
representations relative to physiological responses: it is possible that pitch is first processed
semantically as denoting heavier and longer steps, and that this representation then interacts
with the (incidental) execution of the walking motor program; alternatively, pitch may first
trigger an actual (or simulated) physical reaction, from which a mental representation is
then lifted. Further experimental work, for instance with paradigms which examine stimulus
evaluation while inhibiting or interfering with simultaneous motor behavior (Strack et al., 1988;
Wagenmakers et al., 2016), will be needed to clarify the interaction of cognitive-evaluative
and sensorimotor processes in the behavior exhibited here. Conversely, it is also possible
that motor actions such as stepping lightly or heavily can interfere with specific auditory
perception tasks, in the same way that stepping on every second or third beat of a musical
phrase can influence its perception as a march or a waltz (Phillips-Silver and Trainor, 2005).
Finally, we believe our results may also motivate clinical applications where low-pitched
sounds are used to stabilize gait in e.g. Parkinson's patients (Hausdorff, 2007) or stroke
survivors (Roerdink et al., 2007), possibly in conjunction with other well-described beneficial
effects of tempo synchronization (Repp and Su, 2013).

Finally, if musical pitch has an effect on gait, it appears possible that more complex spectral
properties (rich harmonic structure, timbre, consonance) do too. The present data-driven
paradigm (for which we provide all experimental software as open-source<fnref n="2" />) therefore opens up
a vast domain of research to investigate the interaction of each of these dimensions with human
motion, and elucidate the intricate sensorimotor mechanisms that subserve our perception of
such elusive multimodal percepts as motor and musical stability (Schlenker, 2021), predictive
uncertainty (Hansen et al., 2021), tension (Costa and Nese, 2020) or smoothness/jerkiness
(Sievers et al., 2013).

# 4 Materials and Methods

## Participants

N=20 young (M=25.5, SD=7), Western, educated participants (male: 11, female: 9) partici-
pated in the experiment. All participants gave their informed consent prior to the experiment
and were debriefed about the purpose of the research immediately after. Each of them
received a 15 euros lunch voucher as a compensation.

N=13 participants (65%) self-declared as musicians (M=8.6y of musical training). In
addition, all participants completed a short self-report survey consisting of 15 items extracted
from the Goldsmith Music Sophistication Index (MSI) measuring their active engagement with
music, perceptual abilities, musical training and singing abilities (Müllensiefen et al., 2014).

<footnote n="2">²https://github.com/neuro-team-femto/treadmill</footnote>
Questionnaire scores significantly differed between musicians (M=51.4) and non-musicians
(M=38.8; t(11.7)=2.66, p=.0209, [2.26, 22.79], Cohen's d=1.3).

## Procedure

We tasked our participants to walk on a force-sensing treadmill while listening to generative music constituted by a large variety of musical tones systematically varied in pitch. Prior to each session, participants were asked to wear comfortable shoes, and stood still once on each side of the treadmill for calibration. Treadmill was then started, its speed set at a slow walking speed (M=2.5km/h) and participants practiced walking to the sound of a metronome (75 bpm, period=0.8s) until they were comfortable with the task. Participants were then instructed to walk on the treadmill and to synchronize their steps with a sequence of piano notes they heard through headphones, without the support of handrails and as naturally as possible. Each session used an uninterrupted sequence of 750 sound events, played at a constant rate of 75bpm (period=0.8s), amounting to a total of 10min. Participants were given the cover story that we were interested in how well people could adjust their timing to the period of the sounds, and that the pitch content of the notes was irrelevant and just introduced for variety. They were instructed to focus their attention on walking synchronously with the sequence. After the session, participants were explained the true purpose of the experiment, namely that it was to examine the effect of pitch height on their walking gait.

## Apparatus

Step data was acquired using a legacy instrumented treadmill (Tecmachine ADAL3D), equipped with two force platforms measuring the time series of forces of each limb during walking, along the X (medio-lateral), Y (antero-posterior) and Z (vertical) dimensions. The treadmill was interfaced to our software using a generic USB I/O acquisition card (NI-6008, National Instrument). Data was acquired at a sample rate of 1000Hz, and recorded continuously for the duration of one experiment (10min). While they walked on the treadmill, participants were presented with sequences of musical notes using commercial wireless headphones (Beats Solo3). Using custom software written in Python (https://github.com/
neuro-team-femto/treadmill), we synchronized step data acquisition with the triggering of each musical note, in order to be able to associate the time series of each step to the corresponding sound event.

## Sound stimuli

Sound stimuli were extracted from the University of Iowa Musical Instrument Samples (MIS) dataset https://theremin.music.uiowa.edu/MIS.html and consisted in high-quality recordings of a Steinway & Sons model B piano, made in Nov. 2001 at the Voxman Auditorium, University of Iowa (Iowa City, USA) using a Neumann KM84 microphone. We selected one recording for each of the 24 notes spanning octaves 4 and 5, i.e. from C4 (263.3 Hertz, Hz) to B5 (1016.0 Hz), by steps of one semitone), played at medium dynamics (*mf*). To ensure that stimuli could be accurately synchronized to step data later in the experiment, we then normalized the loudness of the files, trimmed any initial silence (using a threshold at -30dB) and faded out each recording after 1.5sec so that all had the same duration.

We then produced a second derivative dataset by applying an algorithmic pitch transformation (pitch shifting) to each of the original files, in such a way that its fundamental frequency was reduced by 10, 20, 30, 40 or 50% of a semitone (or cents). Pitch shifting was done using a Python implementation of the phase vocoder algorithm (Burred et al., 2019). Finally, we ran an automated pitch analysis algorithm (SWIPE, (Camacho and Harris, 2008)) to document each musical note with its actual fundamental frequency, after pitch shifting. This resulted in a dataset of 240 musical notes, spanning the entire range of frequencies from a low-tuned C4 (255.8Hz) to a high-tuned B5 (1045.8Hz), by steps of 10% of a semitone (10 cents, corresponding to logarithmically increasing steps in Hz, from 1.5Hz at C4 to 5.9Hz at G5).

We then presented each participant with a random sequence of 750 sound events extracted from this dataset of 240 notes. Notes were played at the constant rate of 75bpm (period=0.8s), amounting to a total of 10min, and each note was repeated three consecutive times. In pilot studies, we experimented with several configurations, using only original recordings or using all pitch-shifted recordings, using notes from 2 octaves or 5 octaves, and using isolated or
repeated notes. Because we found that, consistently with the literature (Repp and Su, 2013), participants tended to anticipate the onset of the note to synchronize their steps (Figure 1), we decided to present each note three consecutive times so that steps 2 and 3 could be initiated and performed after one complete hearing of the corresponding note.

## Step data signal processing

For each participant, we normalized the two time series of their left and right foot force data as percentage of the force corresponding to their standing weight, measured during calibration. We then resampled all time series at 100Hz and removed any linear trend across each session (due to treadmill electronics). Next, we segmented each participant's time series into individual steps using a simple threshold procedure: candidate start- and endpoints for each step were positioned where the vertical force time series crossed (resp. upwards or downwards) a threshold set at 5% of the amplitude range of the series, and we eliminated false positives that had a step duration shorter than 100ms. Finally, we associated each step with its corresponding sound event by selecting, for each step, the first sound whose time onset was between the step's start and end point. Steps for which no sound events were found (e.g. after the sequence ended) were deleted from the dataset, and steps for which more than one sound event were found were associated with the earliest of these events. The above procedure resulted in an average M=668 trials per participant (min=403, max=750, SD=114), each associating the time-series of a single step and the characteristics of the corresponding musical note. All analysis code (Python) is available on https://github.com/neuro-team-femto/treadmill.

## Outlier selection

We selected as outliers participants who had more than 20% of their steps not associated with a musical event, on the rationale that this indicated a misunderstanding of the task, a low capacity to synchronize their walking gait to external sounds, or long attentional lapses. N=6 participants were removed from the dataset, leaving 14 participants for analysis (male: 8, female: 6).