# ANIMATED NOTATION, SCORE DISTRIBUTION AND AR-VR ENVIRONMENTS FOR SPECTRAL MIMETIC TRANSFER IN MUSIC COMPOSITION

Jonathan Bell, Benedict Eris Carey

► To cite this version:

Jonathan Bell, Benedict Eris Carey. ANIMATED NOTATION, SCORE DISTRIBUTION AND AR-VR ENVIRONMENTS FOR SPECTRAL MIMETIC TRANSFER IN MUSIC COMPOSITION. TENOR, Jul 2019, Melbourne, Australia. hal-02280057

HAL Id: hal-02280057

https://hal.science/hal-02280057

Submitted on 5 Sep 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# ANIMATED NOTATION, SCORE DISTRIBUTION AND AR-VR ENVIRONMENTS FOR SPECTRAL MIMETIC TRANSFER IN MUSIC COMPOSITION

**Jonathan Bell**

Aix Marseille Univ, CNRS, PRISM
Perception, Representations, Image, Sound, Music
jonathan.BELL@univ-amu.fr

**Benedict Eris Carey**

Hamburg University of Music and Theatre
Hamburg, Germany
benedict.carey@hfmt-hamburg.de

## ABSTRACT

This paper seeks to make a case for a compositional ideal (the mimetic transfer of a recorded or synthesized sound to the instrumental/vocal domain) which today's technologies for animated/distributed musical notation have made more realistic than when it first appeared as a general aesthetic (with composers such as Tristan Murail and Gérard Grisey, or, in the realm of computer music, as with the practice of Jean-Claude Risset), simultaneously with the birth of the digital era in the 1970s. The concept of *mimesis* is here examined both as a (post-) spectral compositional technique and as a common feature of many forms of musical score/representation. These theoretical considerations are then exemplified by musical examples and software demonstrations extract from the “*In memoriam Jean-Claude Risset*” cycle of compositions, scored for ensembles of various sizes (small chamber music group with players wearing head-mounted displays, choir and electronics, large instrumental groups with choir, and for the performance of an opera), all performed with the help of the SmartVox Score distribution system.

## 1. INTRODUCTION: MIMESIS

Just as mimesis was a governing principle of artistic creation in Ancient Greece, it was most certainly also at the heart of the French spectral movement of the 1970s. In this context, composers rarely put into question why for instance the faithful imitation of the harmonic spectrum of a bell sound qualified as art. Assuming that harmony in music is paramount, one can now examine how, in the case of singers for instance, this mimetic transfer might be facilitated when one receives an audio-score [1] through his/her ear, reproducing microtonal harmonies by just imitating the perceived pitch (as is the case with SmartVox[2]). In this case the (auditory) notation closely resembles the desired effect. However, Nelson Goodman demonstrated that this is not necessarily the case with musical notation: in fact even beyond the musical realm, the philosopher is famous for showing that one thing need not resemble an-

other in order for that "other" to represent it. Working on real-time animated music notation and virtual reality [3], the author has remarked in the past of an essential aspect of musical notation: *'the closer a musical unit gets to representing a direct action (that is, the movement of an object in space) the more mimetic it becomes.'* [4] This observation highlights the great complexity of rhythmic notation in common practice musical notation, when compared to modern video game-like notations such as, for instance, *Digital Audio Workstations* where a cursor scrolls from left to right, or in the popular arcade and video game *Guitar Hero* and modern piano-roll notation, where notes or tableture fall from top to bottom of the screen (a practice known as animated notation). In traditional western music notation on the contrary, the historically inherited practice of the arithmetic division of tuplets, their distribution into bars and beats, and the constant adjustments instrumentalists must make to coordinate with the gestures of a conductor, show as a whole how *mimetic* representations are—for better or worse—much easier to understand for beginners than the culturally inherited *memetic* (i.e. symbolic) common practice.

## 2. MIMETICS VS MEMETICS

### 2.1 The Memetic Nature of Animated Notation

When we score music by hand, we do so to varying degrees of mimetic accuracy, albeit often without reference to essential *extra-musical* information required to interpret the music, we are heavily engaged in a *memetic* process. The authors propose that an animated score can be described, in some part, by where it exists in terms of its *mimeticness* and its *memeticness*. Guitar tablature, and *Guitar Hero* make use of very *mimetic* forms of notation. That is, notation which very closely related to the (prescribed) action to be performed, and less bound by the semiotic or *memetic* information one finds in common practice notation, which imply abstract representations engaged with descriptions of the “end-result” sound, allowing for a wider range of possible interpretations, more so with graphic notation, and even more so in more abstract forms of animated notation. The authors believe that *Mimetics* is therefore a useful term for the field scores which rely on imitation, to convey their desire for *audification* (a term attributable to Lindsay Vickery [7] and the author [4]).

Copyright: © 2019 Author Name. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 Unported License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
## 2.2 The Limits of Mimetic Replication

Mimetics serves as a broader term encompassing the study of all forms of replication. Richard Dawkins' seminal and influential concept of memetics takes this idea further into the domain of semiotic thought. Dawkins claims that since universal Darwinism suggests that replication and natural selection are the primary forces through which the universe comes into being, memes are but a tiny part of a far grander vision. Computer music is increasingly pollinated with ideas from all over the spectrum of the sciences – an art-science [5] in emergence as it is referred to by Gérard Grisey. So how do we dismantle and rebuild our understanding of the musical score into the ‘essential’ or ‘base’ form that analyzing animated notation in relation to static notation demands? One that isn’t bound by two dimensions X and Y (generally seen as time and pitch in the score space), which would be far too specific for a multimedia focused compositional system to give but one example. As described by John Blacking in his book *How Musical is Man*, music contains a qualitative dimension. Would a non-systematic approach to animated notation be better suited to tackling this challenge? Here he describes with exactitude how cultural information is essential to our understanding of music:

> ‘Consider the matter of ‘feeling in music,’ which is often invoked to distinguish two technically correct performances of the same piece. This doctrine of feeling is in fact based on the recognition of the existence and importance of deep structures in music.’ [6]

Why people seek to represent through abstraction is not a total mystery, yet untangling the twisted ropes of meaning wrapping up every idea anyone has ever conceived probably presents a completely unsurpassable challenge. The utopian desires of universal Darwinism will likely not be realized through ever increasingly complex quantizations of the natural elements, and while defining moieties in an artistic context may be seen as futile by some, the so-called ‘memes’ that comprise the universal replicators of meaning imagined by Richard Dawkins and significantly expanded upon by Susan Blackmore (even to the point of suggesting that the meme is being superseded by a new unit, the ‘teme’), present a very real and useful concept when applied to animated notation analysis.

## 2.3 Defining Mimetic and Memetic Scoring Systems

As mentioned, we propose that closer a musical unit comes to representing a “direct action”, the more *mimetic* it becomes. Within the scope of this definition, the most *mimetic* score achievable is a person performing any copyable action. Further, this paradigm can be abstracted into the domain of video, and through acousmatic methods, into sound exclusively (such as is the case in many aural musical traditions for example). Lyrebird’s and other parrots engage in this practice also, often with a far greater degree of accuracy than many human musicians. Further levels of abstraction can be seen in Ryan Ross Smith’s particular brand of animated notation involving radials, cursors and other devices designed to control complex rhythmic coordination in ensemble contexts. Cat Hope’s scrolling scores introduce

a similar kind of mimesis, as do David Kim-Boyle’s AR and 3D scores. Common practice notation introduces a higher degree of assumed knowledge and therefore is heavily dependent on *memetic* transfer, so in the same sense, direct copying of a gesture resulting in sound is a *mimetic* transfer, done without requiring a theory of mind, but once the sound begins to stimulate the physiological, auditory system of someone memetic and semiotic processes spring into action.

## 3. MIMETIC ORCHESTRATION

With the exception of the first opus (*In Memoriam JC Risset 1*), whose pitch material was extracted from the analysis of FM synthesis spectra, all the pieces of the cycle presented below constitute *mimetic* orchestrations of the PRISM laboratory synthesizer, developed by Richard Kronland-Martinet [8], who worked for thirty years with Jean-Claude Risset at LMA, Marseille, France.

This synthesizer is based on a perceptive model (as opposed to a physical model [9]) and carries the influence of Risset’s exploration of timbre synthesis, examined through the perspective of human perception [10]. The great majority of the electronics in other pieces of the cycle (*In Memoriam 2, 3 and 4*) makes extensive use of the *liquefaction* parameter, which turns a continuous sound into small grains evoking the resounding impacts of drops on multiple resonant metallic surfaces, or some form of imaginary rain on metallic chimes.<fnref n="1" /> The *Bach* [11] library was very helpful for musical transcription, subsequently allowing for transfer to the instrumental domain.

The authors forthcoming doctoral thesis ‘Between Mimetics and Memetics’ examines the credo assumed by various spectral composers, for whom the very act of composing was equivalent to translation of sounds from the acoustic to the instrumental domain. Classical contemporary music scenes such as those represented in Europe by the Gaudeamus or Darmstadt academies and festivals reveal, however, that such preoccupations are often today very far removed from what young generations of composers seemingly wish to express. Therefore, one can only talk of neo or post-spectralism when referring to such techniques. Whilst still taught in institutions like IRCAM, or briefly mentioned as part of the undergraduate composition curricula, one can explore the fact that this aesthetic canon has seen its day now that technologies (such as the tools presented in the present paper) finally allow more intuitive and user-friendly interfaces to the composer (Bach, Ableton, PRISM synthesizer), auditory and visual computer assistance for microtonal intonation and synchronization adjustments to the performer (SmartVox [2]), and thus probably more convincing results from an audience’s point of view.

### 3.1 Time Domain: Temporal Precision and Mimetic Transfer

At IRCAM in January 2018, Benjamin Matuszewski highly improved the synchronization possibilities of SmartVox

<footnote n="1">¹ A *liquefaction* demonstration is available here: https://youtu.be/2kdIaqAhUGs</footnote>
by implementing a client-side algorithm that puts back in sync devices whose drift exceeds a certain threshold ([12], Chapter 4.3). Until this new release, SmartVox had to deal with a more approximate temporality, but it can now explore a wide range of tightly synchronized musical situations, between musicians, electronics and video for instance.

In none-pulsed music, in spite of the great conducting tradition in chamber music and orchestral works, left-to-right scrolling cursors distributed on the performers' devices (as seen in the Decibel ScorePlayer or SmartVox) seem a far more straightforward strategy to obtain tight synchronization rather than the bars and beats 'encoding' (quantification by the composer with or without the help of algorithms) and decoding processes (a compromised interpretation by the instrumentalist, between the rhythmic values written on the page and the gestures of the conductor), inherited from a scoring tradition in which a regular meter was assumed.

The bach.roll object displays notation in proportional time, and outputs notifications of its playback status in real-time. These notifications can be interpreted in *Max For Live* in order to synchronize 1/ the notation for human players in *Bach* and 2/ the electronics in Ableton Live. Figure 1 shows how Ableton's playback controls can be accessed through the live-set path of the live object model (LOM), which makes constant back and forth playbacks possible between the score and the electronics, during the compositional process. This interface facilitated the composition of *Mit Allen Augen, In Memoriam J.-C. Risset 2*, a piece involving large forces performed in Paris in March 2019 (12 voices, 12 instruments and electronics).<fnref n="2" />

<figure><img src="image_2.png" /><figcaption>**Figure 1.** *Max For Live* device syncing *Bach* and *Ableton*.</figcaption></figure>

Pieces of the Risset cycle all benefit from this improvement: in the first piece of the cycle, a passage is based on rhythmical games between performers, taking advantage of the cursor-type display of time.<fnref n="3" /> In *Das Hoheslied, In Memoriam Jean-Claude Risset 3*, premiered in June 2019, students of HfMT Hamburg were able to sight-sing (for the very first time) passages which strikingly resembled the target model.<fnref n="4" />

## 3.2 Frequency Domain: Harmonic Analysis of Frequency Modulation

Rather than citing the oft-used Shepard-Risset glissando illusion, the author found it more appropriate to re-activate one of the most successful techniques of the beginning of computer music: frequency modulation, which Risset was amongst the first composers to use in his piece Mutations (1969), thanks to Chowning's generous permission. This is why nearly all the harmonic material from *In Memoriam J.C. Risset 1 and 3* was generated by FM synthesis.

For frequency analysis, although *Iana* [13] was formerly used (and also favored by Daniele Ghisi [11] in some of his transcription patches), the *zsa,freqpeak* descriptor [14] offered seemingly more accurate results. A *Max For Live* device was built for pitch visualization purposes.<fnref n="5" />

As a reference to Risset's findings regarding the temporal evolution of the harmonic content of digitally recorded trumpet sounds (which becomes richer in high frequency harmonic when loudness increases [15], the evolution of frequency's 'Modulation index'<fnref n="6" />, was orchestrated by successive entries of voices, from the lowest register (bass) to the highest (soprano).<fnref n="7" />

## 3.3 Conclusion on Mimetic Spectralism and Vocal Imitation

Some of the tools presented in this chapter show that the possibilities offered by today's technologies would have appeared to the supporters of the 'mimetic spectralism' imagined in the 1970s. If the compositional aesthetics followed by these composers is not to everyone's taste, they may at least provide a fertile ground for artistic/scientific research, because their aims and methods, concerned with the reproducibility of a model, are suitable to measurements and, for instance, quantifiable assessments.<fnref n="8" />

In a more scientific context, members of the PRISM<fnref n="9" /> laboratory undertook an experimental study of vocal imitation which recalls the sight-reading experiment in *Das Hoheslied, In Memoriam Jean-Claude Risset 3*. According to the author, Thomas Bordonné, this study "aimed at determining the main characteristics of sounds used by participants during vocal imitations". Bordonné concludes: "Vocal imitations seem to be a good tool to access perception and determine which aspects of the sounds are relevant" [16]. In this setup, therefore, the spontaneous vocal response of participants could be interpreted in similar ways to the SmartVox-led reading sessions in which the singers are asked to imitate what they hear.

<footnote n="2">² The following example shows the convenience of the Bach/Ableton inter-application communication https://youtu.be/VJvY5wY1LcM</footnote>

<footnote n="3">³ A rhythmical passage in *In Memoriam Jean-Claude Risset 1* https://youtu.be/hQtyu1dcCal?t=349</footnote>

<footnote n="4">⁴ *A Target sound* is followed by its imitation by the choir in a sight-reading session of *Das Hoheslied, In Memoriam Jean-Claude Risset 3* https://youtu.be/EHYq9nFF6sE</footnote>

<footnote n="5">⁵ Real-time spectral analysis with *Bach* and *Zsa.FreqPeak* https://youtu.be/D6mCgx4pSxs</footnote>

<footnote n="6">⁶ See https://youtu.be/OnT-Zgkh5MA for demo purposes.</footnote>

<footnote n="7">⁷ See https://youtu.be/sgSjIpSD8yQ.</footnote>

<footnote n="8">⁸ As, for instance, in the aforementioned Hambourg's choir sight-reading example: *Das Hoheslied, In Memoriam Jean-Claude Risset 3*.</footnote>

<footnote n="9">⁹ https://www.prism.cnrs.fr/</footnote>
# 4. COMPOSITIONAL/PERFORMATIVE EXPERIMENTS WITH ANIMATED AND DISTRIBUTED NOTATION

## 4.1 Global Context

Ryan Ross Smith's *animatatednotation.com* website demonstrates that today many composers find fixed common practice notation limiting. To solve this issue, attendees of the Tenor Conference<fnref n="10" /> have proposed elements of response. With Kagel's *Prima Vista* (1962/64) setting a major precent, animated notation often relies on a large screen projecting the parts, as most famously exemplified in Smith's compositional practice. The score can therefore be seen by the performers but also by the audience, thus making notation part of the theatrical performance.

Other performance-oriented systems (SmartVox [2] [12], Zscore [17], Decibel [18] [19], Maxscore [20], compressor [21]) endeavor to distribute and synchronize each part of the score on the performer's devices (whether Smartphones, tablets or laptops). With SmartVox for instance, rendering the score in the browser directly revealed itself to be very effective for this kind of setup. Thanks to cross-platform web technologies, the application works with any browser capable device, and no installation is required by the client. The node.js<fnref n="11" /> /websocket architecture of SmartVox will hopefully inspire more composers, researchers and developers to investigate the emerging musical practice of distributed notation.

## 4.2 Graphical Notation in *Bach*

Most pieces performed with SmartVox rely highly on *Bach* [11], a realtime computer-aided composition package for Max/MSP. The use of color and shape in pieces like *In Memoriam JC Risset 1* or *Mit Allen Augen, In Memoriam JC Risset 2* was inspired by Cat Hope [22] and Lyndsey Vickery [23], two active composers, researchers, developers of the Decibel ScorePlayer [19], whose approach to animated graphical notation opens new territories for musical scores today.

Unlike other notation packages in Max (MaxScore [20], or the Symbolist [24]) which were imagined as symbolic representations in the first place (Rama Gottfried describes the Symbolist as a version of adobe illustrator in Max), the *Bach* environment emerged from *Open Music* [25] and computer-aided composition, which explains some of its limitations from a graphical point of view: for instance, it is impossible to insert a picture in a bach.roll (in proportional notation), and the bach.roll object does not support articulations (unlike the bach.score object in bars and beats notation). However, one can get around this graphical limitation by adjusting the duration line settings (see Figure 2). The use of colors is also easily customizable with the help of 'slots'.

In spite of its few display limitations, one of the strengths of *Bach* consists of its ability to control synthesizers from the notation directly. Slots can therefore be understood as metadata or temporal automations that can subsequently be

<figure><img src="image_3.png" /><figcaption>Figure 2. Duration line adjustment in *Bach*.</figcaption></figure>

mapped onto any Max-programmed synthesizers, enabling staves of a score to be manipulated as tracks of a DAW <fnref n="12" />.

## 4.3 Distributed Musical Notation

Composers and researchers increasingly acknowledge the strong analogy which can be drawn between the traditional 'score and parts' musical practice led by a conductor, and the modern distributed systems or web applications (Zscore [17] - MASD [26] - SmartVox [2]), in which multiple clients coordinate their actions by passing messages to one another.

The Tenor 2018 conference in Montreal revealed an interesting similarity between the Decibel ScorePlayer and SmartVox, in the sense that the two software packages are rather elementary solutions both converging towards the *score player* performance-oriented paradigm, whilst other notation packages (Bach, Maxscore, Symbolist...) were also designed for more elaborate computer-aided composition and real-time processes. Although the Decibel ScorePlayer is today's most reliable iOS application solution for distributed animated notation, Cat Hope and Aaron Wyatt have evoked the possibility to migrate their system to the browser, for instant access over the internet presumably, but also optimization of networking issues, and cheaper production cost in local settings—e.g. for projects involving over fifty devices, the iPad becomes a very expensive solution). The most likely architecture to allow this transfer of technology would be a node.js server <fnref n="13" /> (server-side javascript), with the WebSocket communication protocol, a solution used by SmartVox as early as 2015, thanks to Norbert Schnell and Benjamin Matuszewski from IRCAM in Paris. SmartVox sends and synchronises mp4 audiovisual scores, which has demonstrated undeniable robustness in large scale concerts and rehearsals, but also reveals the potential weakness of a 'non-realtime' solution (as a sort of fixed multichannel tape) in which performances of the piece would be similar each time. Other composers/researchers (Georg Hajdu, Rama Gottfried, Slavko Zagorac) are currently investigating forward thinking solutions which will allow the control of SVG (Scalable Vector Graphic) directly on the client-side HTML page. The authors MaxScore.Netcanvas object acted as a precursor to a later project which made use of this notion, through performance of the 50 part real-time generated piece *Magnetic Visions 50* relying on WebSockets to transfer instantaneously generated score fragments. The architecture chosen for this by Georg Hajdu and Rama Gottfried is node.js, sending osc over WebSockets. This project resulted in a realtime distributed 144-part site-specific composition in Hamburg in May 2019, which was justifiably well received due in part to its technically ambitious and

<footnote n="12">¹² see https://youtu.be/s4qS2khwKT0 for demonstration.</footnote>

<footnote n="13">¹³ According to a private email conversation with the developer Aaron Wyatt.</footnote>

<footnote n="10">¹⁰ http://www.tenor-conference.org/</footnote>

<footnote n="11">¹¹ Server-side JavaScript, see: https://nodejs.org/en/</footnote>
groundbreaking nature.

## 4.4 Augmented Reality Distributed Notation

In *Memoriam JC Risset 1*<fnref n="14" />, premiered in September 2018 at the Gaudeamus Festival (Utrecht), and subsequently performerd in New York in ICMC-NYCEMF 2019, constitutes the author’s first experiment using head-mounted displays for notational purposes, principally inspired by Benedict Carey’s *SpectraScore VR* [3] in this regards. By simply displaying each part of the score over the heads of the performers (for flute and clarinet only), the piece opened promssing perspectives.

Just as traditional scores placed on a music stand, screen-scores displayed on a tablet (for instrumentalists) or on a phone (for singers) oblige musicians to look and orientate their body constantly in the direction of the score. This well-established convention of the classical concert setup considerably limits the possibilities of staging music, in a theatrical context for instance. In such cases as the Ictus Ensemble’s interpretation of *Vortex Temporum* by Gérard Grisey<fnref n="15" />, where the musicians had to learn the score by heart, and perform without a conductor, Wi-Fi-synchronized head-mounted displays might be an interesting way to help musicians coordinate in time and space, while moving on stage or around the audience, without insurmountable performance challenges. The idea was therefore explored further on a larger scale in *Mit Allen Augen, In Memoriam JC Risset 2*.<fnref n="16" />

According to the performers’ feedback, head-mounted setups provide a large and comfortable display, since the environment is still visible around the score (see Figure 3), or through the score in the case of holographic display (such as the Aryzon headset for instance). The performers also showed interest in their ability to move freely on stage or elsewhere (the piece was choreographed differently according to the venue: in a church, the flute and clarinet started the piece behind the audience and gradually approached the altar), which relates to the term *phygital* (physical+digital) coined by Fabrizio Lamberti [27], who claims that with AR and VR, the possibilities of physicalization of gaming will soon encompass other fields.

<figure><img src="image_4.png" /><figcaption>**Figure 3.** Score display from the performer’s point of view.</figcaption></figure>

While awaiting more discrete (such as Google glasses or Vufine displays, which would be invisible from an au-

dience’s point of view) but still affordable solution, AR opens vast domains for further research, which could go far beyond the mere display of animated notation. However, important questions then arise regarding what would be pertinent to achieve with such technology: in a VR context for instance, with a player on stage, how can the limitation due to the destabilization of the performer’s proprioception be overcome? Also, now that any form of immersive score can be prototyped,<fnref n="17" /> how would this be of musical interest, and how could this have a convincing musical impact on the compositional/performative outcome? SmartVox, for instance, delivers and synchronizes mp4 files, and should soon support 360-degree videos<fnref n="18" />, but how can this benefit the performer in a musical sense? Australia holds today one of the world’s largest contingents of composers/researchers working in animated and XR notation, and Australian composers such as David Kim Boyle [28] and the author [3], who specifically investigate 3D-scores in VR environments, present elements of response to this issue.

# 5. NETWORKED MUSICAL PERFORMANCES - NMPS

## 5.1 SmartVox.eu, Score Distribution over the Web and Remote NMPs

With the exponential growth of the web, hosting **Smartvox**<fnref n="19" /> on the internet (i.e. on a remote server rather than a local one) appeared as a necessity. However we demonstrated (in Chapter 4.2 ‘Measurements of timing accuracy’ [12]) that, although the synchronization of different parts was quite accurate in this way, local solutions remained safer (in the same room, over Wi-Fi). Also, notwithstanding the fact that SmartVox undoubtedly belongs to the realm of Networked Music Performances, it became clear that its practical application falls under the ‘local NMPs’ sub-category defined by Gabrielli and Squartini [29].

Since the advent of the internet, thorough research has been undertaken in the realm of remote Networked Musical Performances (NMP) [30]. Today however, local NMPs still seem today more viable, both technically and artistically speaking. Despite many ‘millennials’ being known for their enthusiastic engagement in online gaming (i.e. remotely), with friend avatars or anonymous players often distributed widely around the globe, many examples would show that musicians still feel more inclined to reading a score or improvising together in the same room. In *Embodiment and Disembodiment in Networked Music Performance* [31] Georg Hajdu explains that “appreciation [*of the musical experience*] relies on the plausibility between physical action and sonic result.” Since the emergence of NMPs,<fnref n="20" /> some disappointment arose from the fact that, from the audience’s

<footnote n="17">¹⁷ Such as the ones imagined by Mauricio Kagel’s film *Ludwig Van (1969)*, see for instance: https://youtu.be/7l8vPWFIgXI?t=1591</footnote>

<footnote n="18">¹⁸ This could be done by simply implementing a 360 mediaplayer in HTML5, e.g.: https://bitmovin.com/demos/vr-360</footnote>

<footnote n="19">¹⁹ Each instrumental/vocal part of the piece *And the Sea* is accessible through the following url www.SmartVox.eu, and can be accessed simultaneously from e.g., an iPad for the flute, Android tablets for piano and cello, and a phone for the singer. A trailer of the premiere of the piece is accessible here: https://youtu.be/prcXUbhd-ZY</footnote>

<footnote n="20">²⁰ The American computer music network band ‘the Hub’, formed in 1986, contributed to the popularization of this new genre.</footnote>

<footnote n="14">¹⁴ A recording of the piece is available at the following address: https://youtu.be/hQty1dcCal</footnote>

<footnote n="15">¹⁵ See: https://youtu.be/JFvrYy6EeWE</footnote>

<footnote n="16">¹⁶ A recording of the piece is available at the following address: https://youtu.be/ETOBgFWx04</footnote>
point of view (but also for the musicians themselves), very
little can be seen and therefore understood, which worsens
with distant performers: “Because of the remoteness of the
participants locations, these actions may not always be
perceived directly or immediately [...], classical cause-and-
effect relationships [...] are replaced by plausibility, that is
the amount to which performers and spectators are capa-
ble of ‘buying’ the outcome of a performance by building
mental maps of the interaction. In NMP, this can be facili-
tated by the use of avatars, projected visually, and carefully
orchestrated dramaturgies, involving participants in game-
like scenarios.” Many VR environments today focus on this
interaction between a physical action and its sonic result.
New Atlantis ( [32]), for instance, is a multi-user sound
exploration platform in which several players can interact
with each other.<fnref n="21" /> With their ability to represent avatars of
musical performers, these new interfaces may soon have
the ability to recreate the visual entity producing sound
(or music). By recreating virtual causal links between an
action and its sonic result, these online game interfaces
should provide users with the missing visual element that
prevent NPMs from becoming a genuine globalized musical
practice.

## 5.2 The ‘BabelBox’, a Raspberry Pi Hardware Embedded System Solution for Local NMPs

SmartVox and MaxScore.Netcanvas have proved to be suit-
able for large scale projects such as *le temps des nuages*<fnref n="22" />
and Magnetic Visions 50, in which eighty singers and mu-
sicians [12] and fifty musicians had their score synchro-
nized through the same network respectively. These pro-
ductions still enjoy increasing interest from choirs and en-
sembles,<fnref n="23" /> requiring a very modest technical setup (only
one Wi-Fi access point, e.g. Ubiquiti Unifi) for choirs up
to thirty singers, with a node.js server running on a Mac
computer. However, in the more intimate context of cham-
ber music (as in *In Memoriam JC Risset 1*), the focus is
slightly different: the network load is much lighter, which
encouraged finding a minimal hardware solution, in order
to make possible rehearsals without the physical presence
of the composer or the mobilization of a technician only to
setup a network. Installing the server on the performer’s
computer remotely has often been successful with musi-
cians unfamiliar with technology, thanks to the flexibility
of the Node Package Manager (NPM), which reduces the
installation of SmartVox to a few command lines.<fnref n="24" /> This
installation process nevertheless remained an obstacle for
the dissemination of SmartVox. In search of a light plug-
and-play dedicated system to be sent over the post, the
Raspberry Pi quickly appeared as the best option to host
SmartVox on an embedded system. Node.js runs on Rasp-

bian, and SmartVox proved to be very stable on a Raspberry
Pi 3, so, once installed, the only two steps for a *0-conf*
deliverable hardware were:

* Setting up a static address from the dedicated router (e.g. tp-link...).

* Starting SmartVox at boot.

Starting a script at boot can be done on Raspbian with a
file containing the following in the etc/systemd/system:

```
[Unit]
Description=My service

[Service]
ExecStart=/home/pi/Desktop/hello.sh

[Install]
WantedBy=multi-user.target
```

With the hello.sh script containing the following to launch
the server:

```
#!/bin/bash
cd /home/pi/Desktop/risset
rpm run start
exec bash
```

This low-cost system (less than 65 €, for a Raspberry Pi
and a router) now allows the sending of ready-to-use scores.
Once the system is power-supplied, all the performers need
to do is to join the dedicated Wi-Fi, and type the static IP
address of the server on their smartphone/tablet (i.e. for
the performers: 192.168.0.100:8000, and for the conduc-
tor: 192.168.0.100:8000/conductor). In January 2019, the
system was rented to the Caen French conservatoire via Ba-
belScores<fnref n="25" />, thus proposing a rental of performing scores
(separate parts) of a new kind.

<figure><img src="image_5.png" /><figcaption>**Figure 4.** Score display from the performer's of view.</figcaption></figure>

# 6. CONCLUSION

Animated notation, score distribution, AR and VR tech-
nologies were here presented as tools for composition and
performances in which *mimesis* is examined under multi-
ple notational perspectives. The *mimetic* value of notation
relates, on the ‘prescriptive’ side, to its ability to mimic
an instrumental gesture (such as the representation of the
position of fingers on tablature notation), but also, on the
‘descriptive’ side, to its ability to help performers imitate

<footnote n="21">²¹ Towards the end of the following extract, three players can be observed producing sounds together: https://vimeo.com/264626943</footnote>

<footnote n="22">²² A recording of the piece is available at the following address: https://youtu.be/SyFdR2HiF00</footnote>

<footnote n="23">²³ SmartVox was used for the rehearsals and performance of *To See The Invisible*, an opera by Emily Howard, Aldeburgh, Snape Maltings 2018 https://snapemaltings.co.uk/concerts-history/aldeburgh-festival-2018/to-see-the-invisible/</footnote>

<footnote n="24">²⁴ SmartVox is open source and ready to download via GitHub: https://github.com/belljonathan50/SmartVox0.1.</footnote>

<footnote n="25">²⁵ Babelscores (https://www.babelscores.com/) is an online score
database for classical contemporary music, currently actively support-
ing the SmartVox project: http://luh2.mj.am/nl2/luh2/lgi4u.html.</footnote>
a target sound, as exemplified in the workflow of spectral
composers <fnref n="26" />. Initially conceived almost exclusively as
a rehearsal tool for choral practices [2] [12], recent use
cases have shown that SmartVox is better described as a
distributed score player, allowing for vocalists, instruments
and electronics or multimedia to interact in a Networked
Music Performance.

Acknowledgments

We would like to thank Cat Hope, Lyndsey Vickery, Georg
Hajdu, Daniele Ghisi, Benjamin Matuszewski, Norbert
Schnell and Richard Kronland-Martinet.

7. REFERENCES

[1] J. Bell, “Audio-scores, a resource for composition and computer-aided performance,” Ph.D. dissertation, Guildhall School of Music and Drama, 2016. [Online]. Available: http://openaccess.city.ac.uk/17285/

[2] J. Bell and B. Matuszewski, “SmartVox. A web-based distributed media player as notation tool for choral practices,” in *Proceedings of the 3rd International Conference on Technologies for Music Notation and Representation (TENOR)*. Coruña, Spain: Universidade da Coruña, 2017.

[3] B. E. Carey, “Spectrascore vr: Networkable virtual reality software tools for real-time composition and performance,” in *International conference on New Interfaces for Musical Expression (NIME)*, Brisbane, Australia, 2016.

[4] ——, “From mimetics to memetics,” Ph.D. dissertation, Hochschule für Musik und Theater Hamburg, (unpublished).

[5] C. Arrell, *Pushing the Envelope: Art and Science in the Music of Gérard Grisey*. Cornell University, Aug., 2002. [Online]. Available: https://books.google.fr/books?id=lzJ3GwAACAAJ

[6] J. Blacking, *How Musical Is Man? (Jessie and John Danz Lectures)*. University of Washington Press, 1974.

[7] L. Vickery, “Hybrid real/mimetic sound works,” in *Proceedings of the International Conference on Technologies for Music Notation and Representation – TENOR’16*, R. Hoadley, C. Nash, and D. Fober, Eds. Cambridge, UK: Anglia Ruskin University, 2016, pp. 19–24.

[8] S. Conan, E. Thoret, M. Aramaki, O. Derrien, C. Gondre, S. Ystad, and R. Kronland-Martinet, “An intuitive synthesizer of continuous-interaction sounds: Rubbing, scratching, and rolling,” *Computer Music Journal*, vol. 38, no. 4, pp. 24–37, Dec 2014.

[9] N. Ellis, J. Bensoam, and R. Causse, “Modalys demonstration,” in *Proceedings of International Computer Music Conference*, Barcelona, Spain, Sep. 2005, pp. 101–102, cote interne IRCAM: Ellis05a. [Online]. Available: https://hal.archives-ouvertes.fr/hal-01161344

[10] J.-C. Risset and D. Wessel, “Exploration of timbre by analysis and synthesis,” in *The Psychology of Music*, D. Deutsch, Ed. Academic Press, 1999, pp. 113–169. [Online]. Available: https://hal.archives-ouvertes.fr/hal-00939432

[11] A. Agostini and D. Ghisi, “Bach: an environment for computer-aided composition in max,” in *Proceedings of the 38th International Computer Music Conference (ICMC)*, Ljubljana, Slovenia, 2012.

[12] J. Bell, “Audiovisual Scores and Parts Snchronized Over The Web,” in *TENOR 2018*, Montreal, France, 2018. [Online]. Available: https://hal.archives-ouvertes.fr/hal-01779806

[13] T. Todoroff, É. Daubresse, and J. Fineberg, “IANA: A real-time environment for analysis and extraction of frequency components of complex orchestral sounds and its application within a musical realization,” in *Proceedings of the 1995 International Computer Music Conference, ICMC 1995*, Banff, AB, Canada, September 3-7, 1995, 1995. [Online]. Available: http://hdl.handle.net/2027/spo.bbp2372.1995.088

[14] M. Malt and E. Jourdan, “Zsa.Descriptors: a library for real-time descriptors analysis,” in *5th Sound and Music Computing Conference, Berlin, Germany*, ser. 5th Sound and Music Computing Conference, Berlin, Germany, Berlin, Germany, Jul. 2008, pp. 134–137. [Online]. Available: https://hal.archives-ouvertes.fr/hal-01580326

[15] J. C. Risset, “Computer study of trumpet tones,” *The Journal of the Acoustical Society of America*, vol. 38, no. 5, pp. 912–912, 1965. [Online]. Available: https://doi.org/10.1121/1.1939648

[16] T. Bordonné, M. Dias-Alves, M. Aramaki, S. Ystad, and R. Kronland-Martinet, “Assessing sound perception through vocal imitations of sounds that evoke movements and materials,” in *Computer Music Multidisciplinary Research (CMMR2017)*, Matosinhos, Portugal, Sep. 2017. [Online]. Available: https://hal.archives-ouvertes.fr/hal-01810880

[17] S. Zagorac and P. Alessandrini, “Zscore: A distributed system for integrated mixed music composition and performance,” in *Proceedings of the International Conference on Technologies for Music Notation and Representation – TENOR’18*, S. Bhagwati and J. Bresson, Eds. Montreal, Canada: Concordia University, 2018, pp. 62–70.

<footnote n="26">²⁶ Composers for who the rhythmic and harmonic characteristics of a recorded or synthesised sound serve as a basis for composition and orchestration.</footnote>
[18] C. Hope, A. Wyatt, and D. Thorpe, "Scoring an animated notation opera - the decibel score player and the role of the digital copyist in 'speechless'," in Proceedings of the International Conference on Technologies for Music Notation and Representation – TENOR'18, S. Bhagwati and J. Bresson, Eds. Montreal, Canada: Concordia University, 2018, pp. 193–200.

[19] C. Hope, L. Vickery, A. Wyatt, and S. James, "The decibel scoreplayer - a digital tool for reading graphic notation," in Proceedings of the First International Conference on Technologies for Music Notation and Representation – TENOR'15, M. Battier, J. Bresson, P. Couprie, C. Davy-Rigaux, D. Fober, Y. Geslin, H. Genevois, F. Picard, and A. Tacaille, Eds., Paris, France, 2015, pp. 58–69.

[20] G. Hajdu and N. Didkovsky, "Maxscore: Recent developments," in Proceedings of the International Conference on Technologies for Music Notation and Representation – TENOR'18, S. Bhagwati and J. Bresson, Eds. Montreal, Canada: Concordia University, 2018, pp. 138–146.

[21] P. Louzeiro, "Improving sight-reading skills through dynamic notation – the case of comprovisador," in Proceedings of the International Conference on Technologies for Music Notation and Representation – TENOR'18, S. Bhagwati and J. Bresson, Eds. Montreal, Canada: Concordia University, 2018, pp. 55–61.

[22] C. Hope, "Electronic scores for music: The possibilities of animated notation," Computer Music Journal, vol. 41, no. 3, pp. 21–35, 2017.

[23] L. Vickery, "Some approaches to representing sound with colour and shape," in Proceedings of the International Conference on Technologies for Music Notation and Representation – TENOR'18, S. Bhagwati and J. Bresson, Eds. Montreal, Canada: Concordia University, 2018, pp. 165–173.

[24] R. Gottfried and J. Bresson, "Symbolist: An open authoring environment for user-defined symbolic notation," in Proceedings of the International Conference on Technologies for Music Notation and Representation – TENOR'18, S. Bhagwati and J. Bresson, Eds. Montreal, Canada: Concordia University, 2018, pp. 111–118.

[25] J. Bresson, C. Agon, and G. Assayag, "OpenMusic – Visual Programming Environment for Music Composition, Analysis and Research," in ACM MultiMedia (MM'11), Scottsdale, United States, 2011. [Online]. Available: https://hal.archives-ouvertes.fr/hal-01182394

[26] D. G. Nathan Magnus, "Musician assistance and score distribution (masd)," in Proceedings of The International Conference on New Interfaces for Musical Expression – NIME'2012. Ann Arbor.: University of Michigan, 2012.

[27] M. L. Lupetti, G. Piumatti, and F. Rossetto, "Phygital play hri in a new gaming scenario," in 2015 7th International Conference on Intelligent Technologies for Interactive Entertainment (INTETAIN), June 2015, pp. 17–21.

[28] D. Kim-Boyle, "The 3-d score," in Proceedings of the International Conference on Technologies for Music Notation and Representation – TENOR'17, H. L. Palma, M. Solomon, E. Tucci, and C. Lage, Eds. A Coruña, Spain: Universidade da Coruña, 2017, pp. 33–38.

[29] L. Gabrielli and S. Squartini, Wireless Networked Music Performance. Singapore: Springer Singapore, 2016, pp. 53–92. [Online]. Available: https://doi.org/10.1007/978-981-10-0335-6_5

[30] C. Rottondi, C. Chafe, C. Allocchio, and A. Sarti, "An overview on networked music performance technologies," IEEE Access, vol. 4, pp. 8823–8843, 2016.

[31] G. Hajdu, "Embodiment and disembodiment in networked music performance," in Body, Sound and Space in Music and Beyond: Multimodal Explorations. Taylor & Francis, 2017.

[32] P. f. Sinclair, R. Cahen, J. Tanant, and P. Gena, "New Atlantis: Audio Experimentation in a Shared Online World," in Bridging People and Sound. 12th International Symposium, CMMR 2016, São Paulo, Brazil, July 5–8, 2016, Revised Selected Papers, ser. Lecture Notes in Computer Science, S. Y. Mitsuko Aramaki, Richard Kronland-Martinet, Ed. Springer, 2017, vol. 10525, pp. 229–246. [Online]. Available: https://hal.archives-ouvertes.fr/hal-01791975

[33] K. Fox, "Accretion: Flexible, networked animated music notation for orchestra with the raspberry pi," in Proceedings of the First International Conference on Technologies for Music Notation and Representation – TENOR'15, M. Battier, J. Bresson, P. Couprie, C. Davy-Rigaux, D. Fober, Y. Geslin, H. Genevois, F. Picard, and A. Tacaille, Eds., Paris, France, 2015, pp. 103–108.