## Data-driven analysis

While previous work has focused on quantifying step data based on predefined characteristics (such as step and stride length, or heel and toe contact force (Vaverka et al., 2015)) and contrasting them along predefined stimulus properties (such as consonance (Komeilipoor et al., 2020)), we use here a data-driven strategy in which we let regions showing significant associations with sound emerge from an *a posteriori* analysis of participant responses to many, systematically-varied sounds.

In more details, we use a method inspired by the psychophysical technique of “classification images” (Murray, 2011). For each participant *k*, we compute a weighted time-series $\tilde{p}_k(t)$ by multiplying the time series of each the participant’s *n* steps $p_i(t)$ (with $t \in [0, 100]$ for a 1-sec step sampled at 100Hz) by a weighting factor $z_i$ corresponding to the z-score of the log-transformed pitch of the corresponding note. We then average these weighted time-series over all *n* steps: $\tilde{p}_k(t) = \sum_{i=1}^{n} p_i^k(t)z_i$. Doing so, we negatively weight the patterns of step data that are associated with sound events with lower-than-average pitch, and positively weight those associated with high pitch. The resulting “classification image” $\tilde{p}_k(t)$ is itself a time-series analog to a single step, which represents the strength of statistical association between step force and musical pitch, as a function of time.

Note that for this analysis, it is important that the distribution of stimuli is non-skewed, so that classification images that do not differ from zero indicate no statistical association. Because musical notes are spaced logarithmically in Hz (lower notes are closer to one another that higher notes) the theoretical distribution of random stimuli is skewed towards low Hz, while it is uniform in log-Hz (or cents). For this reason, time-series data is weighted with the logarithm of the pitch (log-Hz), rather than in linear Hz, in the above calculation.

## Ethics

The experiment was approved by the ethics evaluation committee of Inserm, the Institutional Review Board (IRB00003888, IORG0003254, FWA00005831) of the French Institute of medical research and Health, under the Opinion number 22-925.
# 5 Acknowledgments

This work was supported by ERC Grant StG 335536 CREAM (to JJA) and ERC Grant No 788077 Orisem (to P. Schlenker). Research was partly conducted at DEC, Ecole Normale Supérieure - PSL University. DEC is supported by grant FrontCog ANR-17-EURE-0017. Photographs of walking man featured in Figure 1 and 2 are adapted from public-domain photogravure by Eadweard Muybridge, 1887, Wellcome Collection gallery https://wellcomecollection.org/works/v5t3b2ar.
# 6 Bibliography

Alves, B. (2012), ‘Consonance and dissonance in visual music’, *Organised Sound* **17**(2), 114–119.

Athanasopoulos, G. and Moran, N. (2013), ‘Cross-cultural representations of musical shape’, **8**(3), 185.

Bakdash, J. Z. and Marusich, L. R. (2017), ‘Repeated measures correlation’, *Frontiers in psychology* **8**, 456.

Bedoya, D., Arias, P., Rachman, L., Liuni, M., Canonne, C., Goupil, L. and Aucouturier, J.-J. (2021), ‘Even violins can cry: specifically vocal emotional behaviours also drive the perception of emotions in non-vocal music’, *Philosophical Transactions of the Royal Society B* **376**(1840), 20200396.

Blacking, J. (1974), *How musical is man?*, University of Washington Press.

Bodner, E., Gilboa, A. and Amir, D. (2007), ‘The unexpected side-effects of dissonance’, *Psychology of Music* **35**(2), 286–305.

Broze, Y. and Huron, D. (2013), ‘Is higher music faster? pitch-speed relationships in western compositions’, *Music Perception: An Interdisciplinary Journal* **31**(1), 19–31.

Burred, J. J., Ponsot, E., Goupil, L., Liuni, M. and Aucouturier, J.-J. (2019), ‘Cleese: An open-source audio-transformation toolbox for data-driven experiments in speech and music cognition’, *PloS one* **14**(4), e0205943.

Camacho, A. and Harris, J. G. (2008), ‘A sawtooth waveform inspired pitch estimator for speech and music’, *The Journal of the Acoustical Society of America* **124**(3), 1638–1652.

Costa, M. and Nese, M. (2020), ‘Perceived tension, movement, and pleasantness in harmonic musical intervals and noises’, *Music Perception* **37**(4), 298–322.

Desain, P., Honing, H. et al. (1996), Physical motion as a metaphor for timing in music: the final ritard, in ‘Proceedings of the International Computer Music Conference’, International Computer Music Association, pp. 458–460.

Eitan, Z. and Granot, R. Y. (2006), ‘How music moves’, **23**(3), 221–248.

Godey, B., Schwartz, D., De Graaf, J., Chauvel, P. and Liegeois-Chauvel, C. (2001), ‘Neuro-magnetic source localization of auditory evoked fields and intracerebral evoked potentials: a comparison of data in the same patients’, *Clinical neurophysiology* **112**(10), 1850–1859.

Godøy, R. I. and Leman, M. (2010), ‘Musical gestures: sound, movement, and meaning’. OCLC: ocn298781501.

Hansen, N. C., Kragness, H. E., Vuust, P., Trainor, L. and Pearce, M. T. (2021), ‘Predictive uncertainty underlies auditory boundary perception’, *Psychological science* **32**(9), 1416–1425.

Hausdorff, J. M. (2007), ‘Gait dynamics, fractals and falls: Finding meaning in the stride-to-stride fluctuations of human walking’, **26**(4), 555–589.

Kadosh, R. C., Brodsky, W., Levin, M. and Henik, A. (2008), ‘Mental representation: What can pitch tell us about the distance effect?’, *Cortex* **44**(4), 470–477.

Komeilipoor, N., Rodger, M. W. M., Craig, C. M. and Cesari, P. (2020), ‘(dis-)harmony in movement: effects of musical dissonance on movement timing and form’, **233**(5), 1585–1595.

Korsakova-Kreyn, M. (2018), ‘Two-level model of embodied cognition in music.’, *Psychomusicology: Music, Mind, and Brain* **28**(4), 240.
Leman, M. (2007), *Embodied music cognition and mediation technology*, MIT press.

Leonard, M. (1956), 'Emotion and meaning in music', *Chicago: University of Chicago*.

Ma, W. and Thompson, W. F. (2015), ‘Human emotions track changes in the acoustic environment’, *Proceedings of the National Academy of Sciences* **112**(47), 14563–14568.

Mcbeath, M. K. and Neuhoff, J. G. (2002), ‘The doppler effect is not what you think it is: Dramatic pitch change due to dynamic intensity change’, *Psychonomic bulletin & review* **9**(2), 306–313.

Migotti, L. and Zaradzki, L. (2019), ‘Walk-denoting music: refining music semantics’, *Proceedings of the Amsterdam Colloquium 2019*.

Müllensiefen, D., Gingras, B., Musil, J. and Stewart, L. (2014), ‘The musicality of non-musicians: An index for assessing musical sophistication in the general population’, *PloS one* **9**(2), e89642.

Murray, R. F. (2011), ‘Classification images: A review’, *Journal of vision* **11**(5), 2–2.

Phillips-Silver, J. and Trainor, L. J. (2005), ‘Feeling the beat: movement influences infant rhythm perception’, *Science* **308**(5727), 1430–1430.

Putman, D. A. (1987), ‘Why instrumental music has no shame’, *The British Journal of Aesthetics* **27**(1), 55–61.

Repp, B. H. and Su, Y.-H. (2013), ‘Sensorimotor synchronization: A review of recent research (2006–2012)’, *Psychonomic Bulletin & Review* **20**(3), 403–452.

Roerdink, M., Lamoth, C. J., Kwakkel, G., van Wieringen, P. C. and Beek, P. J. (2007), ‘Gait coordination after stroke: Benefits of acoustically paced treadmill walking’, **87**(8), 1009–1022.

Rusconi, E., Kwan, B., Giordano, B. L., Umilta, C. and Butterworth, B. (2006), ‘Spatial representation of pitch height: the smarc effect’, *Cognition* **99**(2), 113–129.

Schlenker, P. (2017), ‘Outline of music semantics’, *Music Perception* **35**(1), 3–37.

Schlenker, P. (2019), ‘Prolegomena to music semantics’, *Review of Philosophy and Psychology* **10**(1), 35–111.

Schlenker, P. (2021), ‘Musical meaning within super semantics’, *Linguistics and Philosophy*.

Sievers, B., Polansky, L., Casey, M. and Wheatley, T. (2013), ‘Music and movement share a dynamic structure that supports universal expressions of emotion’, **110**(1), 70–75. Publisher: National Academy of Sciences Section: Social Sciences.

Sinkjær, T., Andersen, J. B., Ladouceur, M., Christensen, L. O. and Nielsen, J. (2000), ‘Major role for sensory feedback in soleus emg activity in the stance phase of walking in man’, *The Journal of physiology* **523**(Pt 3), 817.

Strack, F., Martin, L. L. and Stepper, S. (1988), ‘Inhibiting and facilitating conditions of the human smile: a nonobtrusive test of the facial feedback hypothesis.’, *Journal of personality and social psychology* **54**(5), 768.

Styns, F., van Noorden, L., Moelants, D. and Leman, M. (2007), ‘Walking on music’, *Human Movement Science* **26**(5), 769–785.

Tan, S.-L. and Kelly, M. E. (2004), ‘Graphic representations of short musical compositions’, **32**(2), 191–212.

Thaut, M. H. and Abiru, M. (2010), ‘Rhythmic auditory stimulation in rehabilitation of movement disorders: A review of current research’, **27**(4), 263–269.
Tieu, L., Schlenker, P. and Chemla, E. (2019), ‘Linguistic inferences without words’, *Proceedings of the National Academy of Sciences* **116**(20), 9796–9801.

Toiviainen, P., Luck, G. and Thompson, M. R. (2010), ‘Embodied meter: Hierarchical eigenmodes in music-induced movement’, **28**(1), 59–70.

Torre, K. and Delignières, D. (2008), ‘Distinct ways of timing movements in bimanual coordination tasks: Contribution of serial correlation analysis and implications for modeling’, **129**(2), 284–296.

Trainor, L. J. and Trehub, S. E. (1992), ‘The development of referential meaning in music’, *Music Perception* **9**(4), 455–470.

Van Dyck, E., Moelants, D., Demey, M., Deweppe, A., Coussement, P. and Leman, M. (2013), ‘The impact of the bass drum on human dance movement’, **30**(4), 349–359.

Van Dyck, E., Moens, B., Buhmann, J., Demey, M., Coorevits, E., Dalla Bella, S. and Leman, M. (2015), ‘Spontaneous entrainment of running cadence to music tempo’, *Sports medicine-open* **1**(1), 1–14.

Van Noorden, L. and De Bruyn, L. (2009), The development of synchronization skills of children 3 to 11 years old, in ‘Proceedings of ESCOM—7th Triennial Conference of the European Society for the Cognitive Sciences of Music. Jyväskylä, Finland: University of Jyväskylä’.

Vaverka, F., Elfmark, M., Svoboda, Z., Janura, M. et al. (2015), ‘System of gait analysis based on ground reaction force assessment’, *Acta Gymnica* **45**(4), 187–193.

Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams Jr, R., Albohn, D., Allard, E., Benning, S. D., Blouin-Hudon, E.-M. et al. (2016), ‘Registered replication report: strack, martin, & stepper (1988)’, *Perspectives on Psychological Science* **11**(6), 917–928.
# Chapter 6

General discussion and conclusions
# Contents

<table><tr><td>1</td><td>Main findings</td><td>190</td></tr><tr><td>1.1</td><td>Music conveys information about the extra-musical world</td><td>190</td></tr><tr><td>1.1.1</td><td>Music licenses inferences about an extra-musical reality</td><td>190</td></tr><tr><td>1.1.2</td><td>Music semantics obeys systematic rules</td><td>191</td></tr><tr><td>1.2</td><td>Musical representations interact with other cognitive domains</td><td>192</td></tr><tr><td>1.2.1</td><td>Musical meanings can be integrated to linguistic environments</td><td>193</td></tr><tr><td>1.2.2</td><td>Pitch height shapes gait patterns</td><td>193</td></tr><tr><td>1.2.3</td><td>Music semantics in music cognition</td><td>194</td></tr><tr><td>2</td><td>Limitations</td><td>195</td></tr><tr><td>2.1</td><td>Investigations restricted to multi-modal interactions</td><td>195</td></tr><tr><td>2.2</td><td>Simplified musical stimuli</td><td>196</td></tr><tr><td>2.3</td><td>An imperfect notion of meaning?</td><td>197</td></tr><tr><td>3</td><td>Avenues for future research</td><td>198</td></tr><tr><td>3.1</td><td>Exploring new semantic effects</td><td>198</td></tr><tr><td>3.2</td><td>The syntax/semantics interface</td><td>199</td></tr><tr><td>3.3</td><td>Considerations on aesthetics</td><td>201</td></tr></table>
We are reaching the end of our investigation of musical meaning. Throughout this
dissertation, we presented several paradigms exploring a variety of music-induced effects.
Because of the abstract nature of music semantics, and building upon the claim from
Schlenker (2017) that “Semantic intuitions that would otherwise be very unclear can
be sharpened by reducing the set of possible denotations,” we conducted research by
having music interact with other systems such as language, visual perception, or the
motor system. In this final chapter, we summarize the main contributions of the papers
included in this dissertation in light of the theory of music semantics that we presented.
We also provide a few insights regarding future avenues for research.

The diversity of the effects we reported supports at least two claims pertaining to different dimensions of music cognition. First, we showed that there is more to music and to musical representations than just reasoning about its form: music can also refer to an extra-musical reality through systematic rules, and we have argued that the set of inferences licensed about this musical reality can define musical meaning. Second, we showed that more general cognitive systems such as language or the motor system can process musical information and operate on musical mental representations.

# 1 Main findings

## 1.1 Music conveys information about the extra-musical world

Our results first show that music is able to convey information about an extra-musical reality in a rule-governed fashion. We first summarize the semantic associations which were experimentally confirmed, and then discuss the corresponding semantic rules involved.

### 1.1.1 Music licenses inferences about an extra-musical reality

Mostly consistent with previous findings and hypotheses on associations between music and movement (Eitan and Granot, 2006; Eitan and Timmers, 2010; Eitan, 2013; Schlenker, 2017), we confirmed that it is possible for listeners to associate different musical properties such as pitch, dynamics and timbre with designated properties of objects, be they visual objects in audiovisual animations, or imagined events such as a character
walking. In particular, we found that pitch and loudness can both be interpreted as the
level of energy or the distance of a denoted object, while timbre is interpreted as the na-
ture of an object, i.e. it is used for object identification in audiovisual scenes. From our
study case on music evoking walks, we found that more fine-grained structural properties
of music (such as steadiness and binarity) need to match the structure of the denoted
event (in our case, a walk), and we also established that consonance can be interpreted
as physical stability, while chord proximity (of two musical events) can be interpreted
as a difference in stability between two denoted events. These findings therefore show
that beyond musical properties whose semantics is lifted from normal auditory cognition
(such as loudness and pitch), some music-specific properties (such as consonance and
chord proximity) play a role in music semantics too.

1.1.2 Music semantics obeys systematic rules

We then showed that the preceding semantic associations obey systematic rules, and pro-
posed three updates on these rules.

(i) While we confirmed that rules of preservation of orderings can account for the in-
terpretation of pitch and loudness (i.e. musical properties and the corresponding object
properties need to co-vary), we showed that they made wrong predictions as to which
musical sequence could denote a walk. In particular, we provided evidence that conso-
nance might be interpreted in terms of absolute levels of stability: a consonant musical
event must denote a physical event, regardless of how its consonance level relates to that
of other musical events.

(ii) We established that the following rule holds: When two objects are present in an audiovisual scene, the musical sound is paired with the object which satisfies the musical properties. This entails that it is not enough to state that *some* object satisfies the musical properties in the scene, but one and a same object has to satisfy these properties.
This was particularly salient with loudness and timbre, which binds objects such that the object/timbre association must remain consistent throughout a scene.

(iii) We provided preliminary evidence that all musical events have to correspond to an
event in the denoted scene.<fnref n="1" />

Together, our results show that it is possible for listeners to assess the compatibility of music with an-extra-musical reality by applying systematic rules of interpretation. We provide a summary of these results in Figure 1. As mentioned in the introduction, several theories had argued that musical meaning was only concerned with reasoning about musical form (i.e. musical meaning is only about processing musical structure, and draw inferences about this structure, for instance through anticipation (Huron, 2006)) or to musical emotions (Meyer, 1956). While our results do not entail that these theories are false, they suggest that musical meaning cannot be restricted to the kind of meaning these theories presuppose, and in particular not to musical form and representations of musical form. An exhaustive theory of musical meaning must account for the inferences music licenses about non-musical objects.

<table><caption>Figure 1: Summary of the semantic rules associating musical properties with non-musical events</caption><thead><tr><th>Property</th><th>Interpretation</th><th>Rule</th></tr></thead><tbody><tr><td rowspan="2">Pitch</td><td>Distance</td><td>The higher the pitch, the closer the denoted object (marginally)</td></tr><tr><td>Energy</td><td>The higher the pitch, the higher the energy of the denoted object</td></tr><tr><td rowspan="2">Dynamics (loudness)</td><td>Distance</td><td>The louder the music, the closer the denoted object</td></tr><tr><td>Energy</td><td>The louder the music, the higher the energy of the denoted object</td></tr><tr><td>Timbre</td><td>Object identification</td><td>The timbre/object association needs to be consistent across the denoted scene</td></tr><tr><td>Consonance</td><td>Physical stability</td><td>1. The more consonant the musical event, the more stable the denoted event/object (relative)<br>2. A consonant musical event must denote a physically stable event/object (absolute)</td></tr><tr><td>Chord proximity</td><td>Symmetry (in terms of physical stability)</td><td>The closer two musical events in the tonal pitch space, the smaller the difference in the physical stability of the denoted events/objects</td></tr></tbody></table>

## 1.2 Musical representations interact with other cognitive domains

Besides establishing correlations with properties of objects, some of our results suggest that music triggers representations and behaviors that perhaps fall beyond the scope of

<footnote n="1">¹This rule relates to considerations on the syntax/semantics interface discussed in Section 3.2.</footnote>
what can be captured by a music semantics (until further developments at least).

1.2.1 Musical meanings can be integrated to linguistic environments

First, and consistent with Tieu et al. (2019) on gestures and visual animations, we showed
that when embedded in language, the information conveyed by musical stimuli can be
divided into the different slots of the inferential typology. More specifically, we found
that pro-speech musical gestures (musical stimuli replacing words in complex sentences)
can give rise to (i) presuppositions (some of the informational content of music is taken
for granted and unaffected by logical operations such as negation or question formation,
while some of it is interpreted as being at-issue), (ii) scalar implicatures (it is possible to
create musical stimuli with contrasted levels of logical informativity), (iii) supplements
(musical stimuli can exhibit the same properties as non-restrictive relative clauses), and
(iv) homogeneity inferences (musical stimuli can refer to a definite plurality of objects).
We thus established that the mechanisms responsible for triggering different inferential
types can process information which is neither linguistic in nature, nor visual.

1.2.2 Pitch height shapes gait patterns

Second, we showed that pitch height affects gait patterns in listeners engaged in walking, i.e. spectral properties produce physical effects on the body. While previous research had established time-related correlations between metric and rhythmic properties of music and walking patterns,<fnref n="2" /> these can easily be accounted for by purely syntactic rules from musical structure to motion structure (time alignment for instance). The correlation we found between pitch and steps duration and force does not seem to be so easily accounted for by a purely syntactic account. In our case, pitch was physically interpreted in terms of properties of movement. Although not a semantic interpretation of pitch per se, this conversion of pitch into physical features obeys a systematic rule: a lower pitch is associated with a longer step with greater force. Interestingly, the physical reaction to changes in pitch (increase in stepping force, which can be associated to an increase in

<footnote n="2">²Among many examples: Styns et al. (2007); Moumdjian et al. (2019); Murata et al. (2017); Ready et al. (2019), see also Repp and Su (2013) for a review</footnote>
weight) seems indirectly consistent with some hypotheses on the semantic interpretation
of pitch as object size (lower-pitched sounds are associated with bigger objects (Eitan and
Granot, 2006)), which suggests that semantic musical representations affect body motion.

Although very different in nature, these two findings from music embedded in language
and music co-occurring with body motion supply additional evidence supporting the claim
that musical information can be integrated with other systems. In other words: music
can be taken as input in mental algorithms which are not music-specific, such as those
generating the reported linguistic inferences, or those responsible for modulating the
motor system.

1.2.3 Music semantics in music cognition

These findings however do not necessarily require an explanation in terms of music seman-
tics (although they are consistent with it, under a few assumptions). It is for instance
possible (although unlikely) that the meaning of musical stimuli is conventionally en-
coded, i.e. that listeners do not apply any particular rule to assign meaning to these
stimuli which just happen to have been associated with certain things through exposure
or learning. Similarly, the correlation we found between pitch height and gait properties
might just be a low-level effect which does not imply any semantic representation.

It is therefore essential that any observation of a fact related to music cognition be
followed by (i) a detailed analysis of the reasons why it is or is not relevant to try
to account for that fact within a theory of musical meaning (all music-induced effects
are potential candidates by default, but a theory needs to motivate the relation with
music semantics), and (ii) a theory of how to integrate the relevant cognitive facts under
investigation. For instance, Schlenker (2017) proposed an account of musical emotions in
terms of a referential semantics, by positing that harmonic stability is interpreted as the
emotional stability of an event (or object) or of the observer of that event. In the case of
the correlations we found between pitch and gait, one should first ask: is it necessary to
explain the role music semantics plays in this, and if so, how do we proceed? We can for
instance speculate that if music is able to refer to listeners themselves, then it becomes
possible that they end up exhibiting motion properties inferred from the semantics of
pitch. This is close to a point made in Schlenker (2017) about the possibility for music
to be interpreted as *experienced* events. However, further theoretical work will first be
needed to ground this hypothesis and derive the corresponding predictions.

## 2 Limitations

While we argued that most of our results fit nicely with a theory of music semantics, one
could argue that the paradigms we developed and the stimuli we used raise questions as
to (i) whether our findings generalize to music alone (typically when it does not interact
with language, visual perception or body motion), and (ii) whether they generalize to
real music.

### 2.1 Investigations restricted to multi-modal interactions

First, one might raise an objection related to the nature of the tasks involved in some
of our experiments, namely that we never directly tested for spontaneous musical infer-
ences in listeners. It is of course likely that participants would not have spontaneously
interpreted our stimuli as someone walking, a white cube gaining energy, etc. But this
is a much broader issue which applies to many cases of abstract representations, such as
pictures or paintings. But even abstract visuals often seem more compatible with some
scenes, and less with others. The problem with such abstract representations is that
we do not necessarily have conscious access to their meaning, a problem that language
typically does not have.

But precisely because music semantics is very abstract, one might argue that it is
easier to ‘force’ meaning in music. This is a legitimate view: it is probable that someone
who is forced to find meaning in something that does not have referential meaning tries
to make sense of the object’s properties and starts making associations with other objects
with some kind of systematicity. However, if listeners were just forcing interpretations,
we would predict much weaker contrasts between stimuli complying with semantic rules
and stimuli not complying with semantic rules. That the contrasts we found can be
explained by differences in productive rules generating meaning gives weight to the claim
that these contrasts are not arbitrary. Although it is always possible that some non-
semantic reasoning mechanism could approximate a semantic rule quite well, it remains
unlikely that any music/world association results from such approximations.

## 2.2 Simplified musical stimuli

Second, one might argue that our findings are hardly replicable with real music because
the stimuli we used were not musical enough. In other words, they might have lacked
the structural complexity and sophistication of real music, hence a question of whether
our findings generalize to cases of real music. But there seems to be no a priori reason to
posit that the basic properties we worked on should behave differently: a crescendo should
always be able to denote an increase in energy or a decrease in distance of an object to a
viewpoint, unless these effects are canceled out by the interaction with other properties
that we have not investigated in this dissertation. Any refinement of the theory, and
in particular any additional hypothesis on the interpretation of a new property should
therefore specify how that new feature is to be integrated with the properties for which
a theory (and/or empirical data) already exists.

The use of simplified music might however have deeper implications. It is possible
that the semantic mechanisms involved in the interpretation of minimal musical sounds
are not quite the same as the ones involved when listeners are actively engaged in real
music and listen to it as such. In other words, a potential risk of over-using these types of
stimuli is to draw conclusions about the interpretation of auditory items that might not
even be perceived as music. If that is the case, then it is not perfectly accurate to claim
that we are testing for musical meaning, and it is more accurate to claim that we are
testing for a more general class of meaning derived from non-musical sound properties.
This is however not necessarily a problem, for at least some of music semantics is lifted
from auditory cognition. Besides, some of our stimuli did have some reasonable musical
structure (typically the ones used in our experiment on music evoking walks).
## 2.3 An imperfect notion of meaning?

The previous limitations might be seen as suggesting a limitation on the nature of the theory we have been relying on: that our definitions of ‘meaning’ and ‘semantics’ have been too permissive, or that we have used an ill-adapted notion of meaning throughout our investigation. This issue arises in other domains as well, where standard definitions of meaning cannot directly apply, such as in studies on animal communication, where meaning can be defined as a set of features of the contexts in which a given signal occurs (Berthet et al., 2023). In our case, one could argue that our notion of meaning is in particular different from a definition of meaning which entails intention, or a notion of ‘utterer’s meaning’ (Grice, 1957). Although there might be effects in music that are closer to reconstructing an intention (typically the composer’s or a narrator’s), none of our findings clearly showed this. We rather investigated things at a ‘pure’ semantic level, without considering pragmatic reasoning.

This begs the following question. Is it possible for a given representation (be it musical, pictorial, gestural, etc.) to denote something if there is no prior intention of conveying meaning? It is possible that, in the case of music, the answer depends on the type of music considered. For composed music, there is arguably an intention to convey information about the world to be reconstructed.<fnref n="3" /> For non-composed music, there seems to be different cases. In improvized music, there might still be an intention of conveying extra-musical information to be reconstructed (the performer’s). Although in other instances, such as computer-generated music, one could argue that no intention is present, a listener might still attribute an intention to a piece as long as it meets certain criteria. In any case, the results we reported here showed that both stimuli which were created with an intention of representing something (a character walking for example) and stimuli which were created with a less explicit intention (in the case of musical sounds simply satisfying structural constraints on loudness increase for instance) could give rise to semantic effects. It might then be that intention is irrelevant to musical meaning, at

<footnote n="3">³Whether the one that is reconstructed is actually the composer’s or a narrator’s, or to what extent the success of this reconstruction process conditions the aesthetics or the understanding of the piece, are interesting questions that we will leave aside for now.</footnote>
least as we have chosen to explore it.

# 3 Avenues for future research

We have just seen that there is more to music than information about its form, and that musical properties can convey information about non-musical things. We will now argue that it is necessary to (i) evidence more semantic effects (and we suggest a few methodological strategies to do so), (ii) investigate how these relate to musical form through the exploration of the syntax/semantics interface, before providing (iii) some concluding remarks on aesthetics.

## 3.1 Exploring new semantic effects

Given the state of the field of music semantics, it is first necessary to document more semantic effects. From a methodological perspective, it seems that one can choose one of at least two strategies. First, through the exploration of the repertoire (and in particular of iconic programmatic music) and systematic controlled rewritings, it is possible to identify musical properties associated with semantic effects, and to isolate their semantics. Importantly, one should make sure to control for consistency when rewriting pieces: a rewritten musical excerpt should still comply with composition rules to avoid parasitic effects triggered by a violation of musical syntax. Just as is standard in linguistics, introspective judgments (to be confirmed by informants and possibly by experimental investigations, in particular in case the effects seem ambiguous) may prove very useful in exploring candidate semantic effects. Second, it might be that collecting experimental data from judgments on expressive music might help narrow down the investigation of new semantic effects to certain pieces and musical properties.

We will here provide two examples of musical properties to be explored in the future, which belong to two different categories of properties, each of which having some kind of homogeneity in the nature of the effects they trigger. We will not be discussing possible theories for which these properties convey the semantics we propose, and only provide these examples to show how the realm of properties to be investigated can be extended.
A first class of musical parameters that seem to convey strong semantic information are musical articulations. Those refer to how a musical event is being played by the instrument player through different techniques modifying its acoustic envelope. A clear example of how playing techniques affect musical meaning is the *pizzicato* technique, i.e. when bow-stringed instruments are plucked with fingers instead of being played *arco* (with a bow). Wherever it occurs, it seems like the pizzicato conveys some sense of light-heartedness, joy or playfulness.<fnref n="4" /> A second effect, of a more pragmatic nature, seems to arise especially when the pizzicato is used a lot: it then feels like the composer is being ironical. A second class of highly evocative parameters are those from which listeners seem to infer properties of the acoustic space. Reverb belongs to that class of parameters. In recent music and songs from Ethereal Wave and Dream Pop genres,<fnref n="5" /> the high level of reverb seem to be highly evocative of space and some kind of ‘boundlessness.’ A possible (speculative) semantic rule at this point could then be: the higher the level of reverb, the larger the inferred space. This would make sense under a theory in which listeners reuse some cognitive mechanisms from normal auditory cognition into music cognition.

As for any intuition about any semantic effect, both these suggestions should now be theoretically justified and then tested through introspective judgments first to make sure that the mappings we propose have some cognitive reality.

## 3.2 The syntax/semantics interface

Although we have already provided evidence in favor of some semantic effects found in music, referencing new ones will give weight to the claim that music can license inferences about extra-musical objects. But if music has both (i) a rule-governed syntax, i.e. rules

<footnote n="4">⁴While usually found only sparsely, some composers have had the audacity to go full pizzicato, such as Austrian composers Johann and Josef Strauss in their *Pizzicato Polka* from 1869., which Schubert (2004) describes the piece as ‘a short, lighthearted, almost comical piece’, which is very much in line with our own introspective judgments.</footnote>

<footnote n="5">⁵These genres first developed in the late 1980’s with artists and bands such as American singer Julee Cruise (Mysteries of Love (1989)) and Scottish rockband Cocteau Twins (I wear your ring (1990), Heaven or Las Vegas (1990)), and later developed by incorporating more electronic features with, among many others, American indie rock band Yo La Tengo (Our Way to Fall (2000), American solo group Grouper featuring Liz Harris (Heavy Water I’d rather be sleeping (2008), and more recently American musical duo Beach House (Myth (2012), Space Song (2015)) or Icelandic alternative rock band Sigur Ros (I sjaki (2013).)</footnote>
on its form and (ii) a rule-governed semantics, i.e. rules on its meaning, then one must
dig into how (i) relates to (ii).

For Schlenker (2017, 2019), the interface between syntax and semantics prompts two
questions. First: Whenever two musical pieces are contrasted in terms of what they
evoke in listeners, are those contrasts due to syntactic (structural) reasons, or to semantic
reasons (related to the denoted objects)? Second: How does the syntactic structure affect
the semantics (and if it does, is it the structure itself, or some more abstract mental
construct based on this structure), and how does the semantics affect syntax in return?

Let us revisit some of our findings in light of these questions. Some of our results
were indeed directly intended to investigate whether musical form was interpreted as
it is, or whether a more abstract structure was rather interpreted. We tested some
preliminary hypotheses on this matter in the case of music evoking walks. We found
indeed that adding a hierarchically less important musical event to an excerpt evoking a
walk significantly weakened the evocation of a walk. If our stimulus was interpreted at
a higher level of musical structure, where this musical event merged into a hierarchically
more important one and does not appear, then we should have found no contrast between
the version containing this additional event, and the version which did not (because
the two versions were then identical at this level). These preliminary results therefore
suggested that our musical stimulus was interpreted at the lowest level of representation
(where all musical events were represented). Further work is however needed to investigate
other cases in more details, in particular by exploring more levels of representation of
musical structure.

But if musical structure certainly determines musical meaning, it is also possible that
the reverse is true, and that musical structure inherits some of its properties from music
semantics, in particular ‘from the perceiver’s attempt to recover the structure of the
denoted events’ (Schlenker, 2019). In other words: it can be that it is because music
evokes an extra-musical reality with its own structural properties<fnref n="6" /> that listeners assign a
given mental structure which aims at preserving (some of) the structure of the denoted

<footnote n="6">⁶Events have indeed been shown to have a hierarchical structure, see for instance Cotnoir and Varzi (2021) or Jackendoff (2009).</footnote>
events. In our example: it is possible that the additional musical event was perceived
as structurally less important because it denoted a less important event in the walk (for
instance, a non-salient moment of the transition from one foot to the next).

These questions of the level of granularity at which a musical form is interpreted are
at the core of the syntax/semantics interface. But these issues are however not specific
to music. They arise just as much in pictorial representations. In this final section, we
present an analogy between pictures and music, and explain how it may provide insights
in terms of more general aesthetic considerations.

3.3 Considerations on aesthetics

The issues of granularity we mentioned call for a system of rules stating which level of
the musical structure is interpreted, which should enrich the semantics that we have been
developing, and it is possible that these rules are close to *marking rules*, which have
semantic and aesthetic implications. Willats (1997) argues that artistic representational
systems are essentially composed of pairs of (i) marks in the pictures, and (ii) features
of the denotation that are marked.<fnref n="7" /> For instance, an artist might choose to mark only
edges of a given denoted scene (say, a landscape), and to use black lines to represent
these edges; that would be close to a very simplistic way of drawing. But the artist might
well choose to do something entirely different, and mark only surfaces of a certain color,
and use dotted surfaces to represent these surfaces. The very same theoretical framework
might well apply to music: a full theory of music semantic must make these marking
rules explicit (if any), and in particular clarify at which level(s) of musical structure are
being marked.

Just as marking rules in pictorial representations seem to covary with pictorial styles
(Willats, 1997), it might be that the analysis of marking rules in music proves useful in
establishing a semantic-based typology of musical styles and systems of musical composi-
tion. Crucially, these marking rules may overlap with issues of granularity: some musical

<footnote n="7">⁷Importantly, these accounts of artistic representational systems are different from ‘pure’ pictorial representations, which Greenberg (2013, 2021) argued can be accounted for by a semantics in terms of geometrical projection onto a surface relative to a viewpoint.</footnote>
system might require that all events be interpreted as world events, while some others
might not. The question similarly applies to the level of granularity of the denotation:
which properties of the denoted event have to be represented/marked?<sup>8</sup>

A final fundamental question bridges music semantics and music aesthetics in an even
more direct fashion: how does musical content or meaning relate to aesthetic proper-
ties of music? In particular, a theory of music semantics needs to explain why listeners
draw inferences from any kind of music, regardless of its aesthetic properties, and re-
gardless of whether it is considered good or bad.<fnref n="9" /> It is possible that music semantics has
implications in terms of aesthetic value, and that musical meaning (the set of denoted
events) provides constraints as to both (i) what the best structure for a musical piece
intended to represent a given event is, and (ii) what the best interpretation of a piece is
(in the sense of a performance). This seems particularly relevant as there seems to exist a
great consistency in what musicians rate as a good or bad interpretations (Geringer and
Madsen, 1998); but further work on the interaction between music semantics and mu-
sic aesthetics will be needed to explore the extent to which musical meaning contributes
to aesthetic properties and aesthetic judgment of musical pieces and their interpretations.

In a nutshell, the expansion of music semantics will not only continue to give insights
as to how music has been used, intentionally or not, to refer to and represent the world,
but it will also prove very useful in addressing fundamental questions about how form
relates to meaning and how meaning relates to aesthetic properties and judgments, in
music and beyond.

<footnote n="8">⁸See Zaradkzi (2021) for a detail analysis of the notion of event in music semantics.</footnote>

<footnote n="9">⁹As shown in Hesmondhalgh (2007), listeners have strong intuitions as to what qualifies as good or bad music. Washburne and Derno (2004) showed that these aesthetic judgments are often independent from aesthetic pleasure, in the sense that many listeners enjoy listening to music which they characterize as bad.</footnote>
References

Berthet, M., Coye, C., Dezecache, G., and Kuhn, J. (2023). Animal linguistics: a primer. *Biological Reviews*, 98(1):81–98.

Cotnoir, A. J. and Varzi, A. C. (2021). *Mereology*. Oxford University Press, New York.

Eitan, Z. (2013). How pitch and loudness shape musical space and motion. In *The psychology of music in multimedia*, pages 165–191. Oxford University Press, New York, NY, US.

Eitan, Z. and Granot, R. Y. (2006). How Music Moves. *Music Perception*, 23(3):221–248.

Eitan, Z. and Timmers, R. (2010). Beethoven’s last piano sonata and those who follow crocodiles: Cross-domain mappings of auditory pitch in a musical context. *Cognition*, 114:405–422. Place: Netherlands Publisher: Elsevier Science.

Geringer, J. M. and Madsen, C. K. (1998). Musicians’ Ratings of Good versus Bad Vocal and String Performances. *Journal of Research in Music Education*, 46(4):522–534.

Greenberg, G. (2013). Beyond Resemblance. *The Philosophical Review*, 122(2):215–287.

Greenberg, G. (2021). Semantics of Pictorial Space. *Review of Philosophy and Psychology*, 12(4):847–887.

Grice, H. P. (1957). Meaning. *The Philosophical Review*, 66(3):377.

Hesmondhalgh, D. (2007). Audiences and Everyday Aesthetics: Talking About Good and Bad Music. *European Journal of Cultural Studies*, 10:507–527.

Huron, D. (2006). *Sweet Anticipation: Music and the Psychology of Expectation*. The MIT Press.

Jackendoff, R. (2009). Parallels and Nonparallels between Language and Music. *Music Perception*, 26(3):195–204.

Meyer, L. B. (1956). *Emotion and meaning in music*. Univ. of Chicago Press, Chicago. Ill., paperback ed., [nachdr.] edition.

Moumdjian, L., Moens, B., Maes, P.-J., Van Nieuwenhoven, J., Van Wijmeersch, B., Leman, M., and Feys, P. (2019). Walking to Music and Metronome at Various Tempi in Persons With Multiple Sclerosis: A Basis for Rehabilitation. *Neurorehabilitation and Neural Repair*, 33(6):464–475. Publisher: SAGE Publications Inc STM.

Murata, H., Bouzarte, Y., Kanebako, J., and Minamizawa, K. (2017). Walk-In Music: Walking Experience with Synchronized Music and Its Effect of Pseudo-gravity. In *Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology*, pages 177–179, Québec City QC Canada. ACM.

Ready, E. A., McGarry, L. M., Rinchon, C., Holmes, J. D., and Grahn, J. A. (2019). Beat perception ability and instructions to synchronize influence gait when walking to music-based auditory cues. *Gait & Posture*, 68:555–561.

Repp, B. H. and Su, Y.-H. (2013). Sensorimotor synchronization: A review of recent research (2006–2012). *Psychonomic Bulletin & Review*, 20(3):403–452.

Schlenker, P. (2017). Outline of Music Semantics. *Music Perception*, 35(1):3–37.
Schlenker, P. (2019). Prolegomena to Music Semantics. *Review of Philosophy and Psychology*, 10(1):35–111.

Schubert, E. (2004). Modeling Perceived Emotion With Continuous Musical Features. *Music Perception*, 21(4):561–585.

Styns, F., van Noorden, L., Moelants, D., and Leman, M. (2007). Walking on music. *Human Movement Science*, 26(5):769–785.

Tieu, L., Schlenker, P., and Chemla, E. (2019). Linguistic inferences without words. *Proceedings of the National Academy of Sciences*, 116(20):9796–9801.

Washburne, C. and Derno, M. (2004). *Bad music: the music we love to hate*. Routledge, New York. OCLC: 55981688.

Willats, J. (1997). *Art and representation: new principles in the analysis of pictures*. Princeton University Press, Princeton, N.J.

Zaradkzi, L. (2021). *Les évenements en sémantique linguistique et musicale*. PhD dissertation, Université Paris Cité.
# Publications

## Peer-reviewed articles

MIGOTTI, L. & GUERRINI, J. (2023). Linguistic inferences from pro-speech music: Musi-cal gestures generate presuppositions, scalar implicatures, supplements and homogeneity inferences. *Linguistics and Philosophy*.

https://doi.org/10.1007/s10988-022-09376-9

## Articles to be submitted

MIGOTTI, L., DECULTOT, Q., GRAILHE, P. and AUCOUTURIER, J-J. (2023). Cracking the pitch-code of motor-music synchronization using data-driven methods.

https://psyarxiv.com/zkbn3/

MIGOTTI, L., CHEMLA, E., SCHLENKER, P. (2023). How music can refer to objects: Coreference from dynamics, pitch and timbre in audiovisual animations.

MIGOTTI, L. & ZARADZKI, L. (2023). How music evokes walking: A case-study in music semantics.

## Conference Proceedings

MIGOTTI, L. & ZARADZKI, L. (2019). Walk-denoting music: Refining music semantics. *Proceedings of the 22nd Amsterdam Colloquium*.

https://archive.illc.uva.nl/AC/AC2019/uploaded_files/inlineitem/1AC2019_Proceedings.pdf
ABSTRACT

Does music have meaning? And if so, what kind of meaning? In this dissertation, we provide experimental data supporting the view that music has a semantics: (i) music is able to refer to an extra-musical reality, and (ii) it does so in a rule-governed fashion. We present four experimental paradigms designed to test these two assumptions by having music interact with other cognitive faculties such as language, visual perception, or body motion.

In the first experiment, we presented participants with hybrid sentences containing music in lieu of words and tried to determine what meaning they get from such sentences. The results show that the informational content of embedded musical stimuli could be divided among the different slots of the linguistic typology (i.e. the different types of inference that complex sentences trigger). This suggests that the cognitive mechanisms classifying meaning in language can accommodate musical information.

In the second experiment, we had participants judge audiovisual scenes in which musical properties such as pitch height, dynamics, and timbre were manipulated and paired with visual objects. The results show that (i) both pitch height and dynamics can be semantically interpreted as the energy level or the distance of a visual object, (ii) pitch height and dynamics can be used to retrieve which object is being referred to when multiple objects are present, and (iii) timbre is interpreted as the very nature of an object, hence the timbre/object association cannot be violated in audiovisual scenes. Together, these results show that musical properties are not only associated with general scenes, but that they are bound to objects in those scenes.

In the third experiment, we built a case study about music that evokes walking. We sought to determine what it takes for a musical piece to evoke a walk through a preference task in which participants had to pick stimuli based on how well they evoked a character walking. The results show that at least five properties are involved in the association of a musical stimulus with a walk: three of which (steadiness, alternation, binarity) relate to structural properties of a walk, and two of which (absolute consonance and relative chord proximity) relate to physical properties of a walk such as stability. Consonance was arguably interpreted as the stability of each step, while chord proximity was associated with gait symmetry. The results further show that this fine-grained model accounts for the semantics of music evoking walks better than previous models and provide preliminary evidence that each musical event might have to be interpreted as a real-world event.

In the fourth experiment, we finally checked whether it is possible to find footprints of the semantics of pitch on body motion. Participants were tasked to walk in synchrony with a series of sounds varied in pitch. We found that their steps were both longer and produced with greater force on lower pitches. The similarity between the physical effect produced by lower pitch (mimicking an increase in weight) and the semantics of lower pitch (associated with bigger objects, or less energy) suggests that there is an interaction between musical representations and the motor system.

Together, our results provide evidence for a music semantics: music can refer to an extra-musical reality according to a set of systematic rules. They also show that music interacts with other cognitive systems either directly bearing meaning (language) or not (body motion), which raises questions as to how music-induced effects relate to musical meaning.

KEYWORDS

music, semantics, meaning, music cognition