<figure><img src="image_1.png" /></figure>

HAL
open science

# Exploring Magical Musical Interactions In Web Based Virtual Environments

David Schnug, Liam Tsang

<figure><img src="image_2.png" /></figure>

## To cite this version:

David Schnug, Liam Tsang. Exploring Magical Musical Interactions In Web Based Virtual Environments. Web Audio Conference 2024, Purdue University Department of Music, Mar 2024, West Lafayette, IN, United States. hal-04619605

HAL Id: hal-04619605

https://hal.science/hal-04619605

Submitted on 21 Jun 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Exploring Magical Musical Interactions In Web Based Virtual Environments

David Schnug, Liam Tsang

<figure><img src="image_1.png" /></figure>

## To cite this version:

David Schnug, Liam Tsang. Exploring Magical Musical Interactions In Web Based Virtual Environments. Web Audio Conference 2024, Purdue University Department of Music, Mar 2024, West Lafayette, IN, United States. hal-04619605v2

HAL Id: hal-04619605

https://hal.science/hal-04619605v2

Submitted on 10 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Exploring Magical Musical Interactions In Web Based Virtual Environments

David Schnug ID

djs794@nyu.edu

Liam Tsang

liamtsang@gmail.com

## ABSTRACT

This article describes the creation of a web based virtual reality music composition called *Pisces* and the implementation of an emerging design trend called the Magical Musical Interaction. An overview of the relevant subfields of research related to the project are delineated first, followed by a brief overview of three recent publications that describe and define the Magical Musical Interaction. Following this, an expanded definition is offered by the authors that widens the concept's scope to encompass more elements of the virtual reality experience. A description of *Pisces* in artistic and technical terms follows along with an analysis of the Magical Musical Interactions present in the piece. Lastly, a future work section describes how the techniques employed in *Pisces* might be expanded to foster new modes of performance method, compositional technique, and audience interaction in networked virtual musical experiences.

## CCS Concepts

* Applied computing → Sound and music computing;

* Information systems → Web applications;

## 1. INTRODUCTION

As the web becomes ever more capable of supporting real time interactions in multiuser virtual environments [13], the possibility of crafting immersive virtual musical experiences emerges. While many composers and technicians have experimented with networked and virtual art in the past [1, 9] the current commercial viability of Head Mounted Displays (HMDs), widespread broadband access, and an emerging Internet of Musical Things [17], allow many more creators to take up the project, as well as provide access to a much wider audience.

A term currently in use for such works is *Audio-First* virtual content. Meant to distinguish itself from audiovisual material like interactive game environments where music is incidental to progress, Audio-First content makes musical developments the main locus of a user's attention [6]. This orientation establishes a conceptual space for composers to

create musical content in the emerging domain of the metaverse [10], and in a virtual context more generally.

Given these developments, artists are now faced with establishing coherent techniques and practices. Possible approaches are still under investigation, and guiding principals are currently being developed through both academic writing and composition. While reviewing the literature on this topic, consensus on a specific goal presented itself: the design of Magical Musical Interactions (MMIs). Serafin, et al. describe a MMI as "an interaction or an instrument ... not limited by real-world constraints, such as the ones imposed by the laws of physics, human anatomy, or the current state of technological development" [14, p. 28]. MMIs reveal the unique potential of a virtual environment to produce new artistic experiences. This article describes the creation of a piece composed for a virtual environment on the web called *Pisces* and its relationship to MMIs.

We will begin by briefly introducing the relevant subfields of research that have contributed to the formulation of the project and identify three recently published papers that reference MMIs as a design goal. After we offer an extended definition of the MMI, we will recount the creation of our work from artistic principals to technological design. An analysis of the piece through the lens of MMIs will follow, and a future work section will reveal how fundamental design features of *Pisces* can be used to realize MMIs on a greater scale.

## 2. RELEVANT SUBFIELDS, RECENT RESEARCH, AND MAGICAL MUSICAL INTERACTIONS

In order to properly discuss the MMIs found in *Pisces*, it is first necessary to lay out the artistic and technological landscape in which they can exist. The following is a list of the fields of study that informed the creation of *Pisces*:

* *Musical Metaverse* Within the metaverse at large, there exist virtual spaces dedicated to the creation and performance of musical experiences. This is the realm of the musical metaverse, and it is this domain in which Audio-First content can be realized on the web. An example of such an environment is a festival on the platform Decentraland from 2021 that featured virtual performances of contemporary popular musicians [16].

* *Musical Extended Reality* Extended Reality (XR) can be used as an umbrella term for all virtual, augmented,

<figure><img src="image_4.png" /></figure>

Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Attribution: owner/author(s).

Web Audio Conference WAC-2024, March 15–17, 2024, West Lafayette, IN, USA.

© 2024 Copyright held by the owner/author(s).
# Exploring Magical Musical Interactions In Web Based Virtual Environments

David Schnug ID

djs794@nyu.edu

Liam Tsang

liamtsang@gmail.com

## ABSTRACT

This article describes the creation of a web based virtual reality music composition called *Pisces* and the implementation of an emerging design trend called the Magical Musical Interaction. An overview of the relevant subfields of research related to the project are delineated first, followed by a brief overview of three recent publications that describe and define the Magical Musical Interaction. Following this, an expanded definition is offered by the authors that widens the concept's scope to encompass more elements of the virtual reality experience. A description of *Pisces* in artistic and technical terms follows along with an analysis of the Magical Musical Interactions present in the piece. Lastly, a future work section describes how the techniques employed in *Pisces* might be expanded to foster new modes of performance method, compositional technique, and audience interaction in networked virtual musical experiences.

## CCS Concepts

* Applied computing → Sound and music computing;

* Information systems → Web applications;

## 1. INTRODUCTION

As the web becomes ever more capable of supporting real time interactions in multiuser virtual environments [13], the possibility of crafting immersive virtual musical experiences emerges. While many composers and technicians have experimented with networked and virtual art in the past [1, 9] the current commercial viability of Head Mounted Displays (HMDs), widespread broadband access, and an emerging Internet of Musical Things [17], allow many more creators to take up the project, as well as provide access to a much wider audience.

A term currently in use for such works is *Audio-First* virtual content. Meant to distinguish itself from audiovisual material like interactive game environments where music is incidental to progress, Audio-First content makes musical developments the main locus of a user's attention [6]. This orientation establishes a conceptual space for composers to

create musical content in the emerging domain of the metaverse [10], and in a virtual context more generally.

Given these developments, artists are now faced with establishing coherent techniques and practices. Possible approaches are still under investigation, and guiding principals are currently being developed through both academic writing and composition. While reviewing the literature on this topic, consensus on a specific goal presented itself: the design of Magical Musical Interactions (MMIs). Serafin, et al. describe a MMI as "an interaction or an instrument ... not limited by real-world constraints, such as the ones imposed by the laws of physics, human anatomy, or the current state of technological development" [14, p. 28]. MMIs reveal the unique potential of a virtual environment to produce new artistic experiences. This article describes the creation of a piece composed for a virtual environment on the web called *Pisces* and its relationship to MMIs.

We will begin by briefly introducing the relevant subfields of research that have contributed to the formulation of the project and identify three recently published papers that reference MMIs as a design goal. After we offer an extended definition of the MMI, we will recount the creation of our work from artistic principals to technological design. An analysis of the piece through the lens of MMIs will follow, and a future work section will reveal how fundamental design features of *Pisces* can be used to realize MMIs on a greater scale.

## 2. RELEVANT SUBFIELDS, RECENT RESEARCH, AND MAGICAL MUSICAL INTERACTIONS

In order to properly discuss the MMIs found in *Pisces*, it is first necessary to lay out the artistic and technological landscape in which they can exist. The following is a list of the fields of study that informed the creation of *Pisces*:

* *Musical Metaverse* Within the metaverse at large, there exist virtual spaces dedicated to the creation and performance of musical experiences. This is the realm of the musical metaverse, and it is this domain in which Audio-First content can be realized on the web. An example of such an environment is a festival on the platform Decentraland from 2021 that featured virtual performances of contemporary popular musicians [16].

* *Musical Extended Reality* Extended Reality (XR) can be used as an umbrella term for all virtual, augmented,

<figure><img src="image_3.png" /></figure>

Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Attribution: owner/author(s).

Web Audio Conference WAC-2024, March 15–17, 2024, West Lafayette, IN, USA.

© 2024 Copyright held by the owner/author(s).
and mixed realities which to a greater or lesser extent
place a user in a digitally produced immersive environ-
ment [11]. Much like the musical metaverse, there ex-
ists a subfield of XR research and practice that focuses
on musical experiences [18]. Çakmak and Hamilton
discuss the creation of a piece that uses this modal-
ity to virtually reproduce a live performance featuring
spatialized sound and enveloping visual projections in
[5].

* *Virtual Reality Musical Instruments* Composers of electronic music have over the years sought to create interfaces for real time control of synthesis and sound production. These Digital Musical Instruments (DMIs) vary greatly- some rely on commercial hardware control devices, while others use bespoke controllers; some seek to recreate the sounds of acoustic instruments and others generate sounds unique to electronic synthesis [8]. When this work is undertaken in a virtual environment, the result is a Virtual Reality Musical Instrument (VRMI). These instruments contain a visual component in the virtual world, and can be replicas of acoustic instruments or completely novel constructions that can only exist in the virtual space [14].

* Spatial Composition Since the birth of electronic music as an art form, the use of physical space has been a primary concern. With sound generation transferred from an acoustic instrument to a loudspeaker that could be placed anywhere in a performance space, composers sought techniques to place their sound sources in a musically relevant manner. As immersive audio technology like binaural filtering and ambisonics evolved, it became possible to liberate sound sources from even the loudspeakers themselves, and to position a sound source anywhere in three dimensional space around the listener [12]. The confluence of these artistic ambitions and evolving spatial audio technology is what can be called Spatial Composition: the inclusion of sound object location and movement as composable elements in a work of music [3].

Figure 1 represents each of these fields in a Venn diagram. Their intersection reveals the space in which this musical work was conceived. When summarized in these terms, we can say that *Pisces* is a spatial composition, in which the movement of sound objects in the piece is an integral part of the work. The sound sources in the piece are given visual form and exist as VRMIs. The instruments are presented in a VR space on the web, creating an Audio-First work that showcases the potential of musical XR in the musical metaverse.

**2.1 Recent Research**

Several distinguished artists and researchers have recently published articles that define design principals and areas for future work in several of the fields mentioned above. When taken as a whole, they represent the first attempts at establishing guidelines for creating Audio-First VR content. Below is a list and short summary of three such papers <fnref n="1" />:

* *Music In Extended Realities*, Turchet, et al., 2021 [18] In this article, Turchet at al., describe the nature of Musical XR and offer a research agenda for its development.

* Doing vs. Being: A philosophy of design for artful VR, Atherton and Wang, 2019 [2] This article is a wide ranging exploration of how to properly design musical VR experiences. A comprehensive list of recommendations is presented, complete with philosophical rationales for their adoption, examples, and criticism.

* Virtual Reality Musical Instruments: State of the Art, Design Principals, and Future Directions, Serafin, et al., 2016 [14] This article describes VRMIs, gives design examples, and presents some core design principals.

The recommendations in these articles frequency coincide;
in fact, they reference one another quite regularly. Points
of contact include agreement on the importance of creating
social interactions, designing with the current limitations
of VR hardware in mind, and acknowledging the potential
for cybersickness. Another common thread relates to the
creation of MMIs.

**2.2 Magical Musical Interactions**

The term was coined by Serafin, et al. in reference to
designing VRMIs and the resulting interactions between
performer and instrument. To reiterate, these interactions
leverage the unique ability of a virtual environment to dis-
pense with real-world constraints such as “physics, human
anatomy, or the current state of technological development”
and in doing so create playing techniques and audible results
that cannot occur with acoustic instruments.

Turchet, et al. make the remark that, “XR can evoke
levels of abstraction, immersion, and imagination not possi-
ble with conventional musical interfaces, ushering in entirely
new ways of engaging with music for both the performer
and the listener” [p. 15805]. This summarizes a commentary
throughout the article that suggests progress in Audio-First
XR content lies in the discovery of interactions and designs
that cannot be achieved with the use of acoustic instruments
and physical spaces.

Finally, Atherton and Wang declare plainly in one of their
design principals: “Don’t port: Make things that would be
impossible in the real world” [p. 2]. They argue that if the
medium of VR is to be used to make art, its specific affor-
dances should be fully exploited. Since virtual worlds are
capable of removing many physical constraints, it behooves
artists to explore what can be accomplished if we are freed
of them. An example is found in a piece by Atherton enti-
tled 12 Sentiments for VR. This Audio-First VR work moves
through the life cycle of a plant, offering the user different
interactive environments in which to experience scenes like
the dispersion of a seed floating through the wind to the
gathering up of rays of sunlight by a mature plant, all while
activating musical sequences through varying degrees of user
control. In the work, the user *is* the plant, frequently from
the first person perspective.

When considering the conversations about Musical XR
and VR design by Truchet, et al. and Atherton and Wang,
the definition of a MMI can expand beyond a performer's

<footnote n="1">¹All further mentions of these authors will reference these
articles.</footnote>
and mixed realities which to a greater or lesser extent
place a user in a digitally produced immersive environ-
ment [11]. Much like the musical metaverse, there ex-
ists a subfield of XR research and practice that focuses
on musical experiences [18]. Çakmak and Hamilton
discuss the creation of a piece that uses this modal-
ity to virtually reproduce a live performance featuring
spatialized sound and enveloping visual projections in
[5].

* *Virtual Reality Musical Instruments* Composers of electronic music have over the years sought to create interfaces for real time control of synthesis and sound production. These Digital Musical Instruments (DMIs) vary greatly- some rely on commercial hardware control devices, while others use bespoke controllers; some seek to recreate the sounds of acoustic instruments and others generate sounds unique to electronic synthesis [8]. When this work is undertaken in a virtual environment, the result is a Virtual Reality Musical Instrument (VRMI). These instruments contain a visual component in the virtual world, and can be replicas of acoustic instruments or completely novel constructions that can only exist in the virtual space [14].

* Spatial Composition Since the birth of electronic music as an art form, the use of physical space has been a primary concern. With sound generation transferred from an acoustic instrument to a loudspeaker that could be placed anywhere in a performance space, composers sought techniques to place their sound sources in a musically relevant manner. As immersive audio technology like binaural filtering and ambisonics evolved, it became possible to liberate sound sources from even the loudspeakers themselves, and to position a sound source anywhere in three dimensional space around the listener [12]. The confluence of these artistic ambitions and evolving spatial audio technology is what can be called Spatial Composition: the inclusion of sound object location and movement as composable elements in a work of music [3].

Figure 1 represents each of these fields in a Venn diagram. Their intersection reveals the space in which this musical work was conceived. When summarized in these terms, we can say that *Pisces* is a spatial composition, in which the movement of sound objects in the piece is an integral part of the work. The sound sources in the piece are given visual form and exist as VRMIs. The instruments are presented in a VR space on the web, creating an Audio-First work that showcases the potential of musical XR in the musical metaverse.

**2.1 Recent Research**

Several distinguished artists and researchers have recently published articles that define design principals and areas for future work in several of the fields mentioned above. When taken as a whole, they represent the first attempts at establishing guidelines for creating Audio-First VR content. Below is a list and short summary of three such papers <fnref n="1" />:

* *Music In Extended Realities*, Turchet, et al., 2021 [18] In this article, Turchet at al., describe the nature of Musical XR and offer a research agenda for its development.

* Doing vs. Being: A philosophy of design for artful VR, Atherton and Wang, 2019 [2] This article is a wide ranging exploration of how to properly design musical VR experiences. A comprehensive list of recommendations is presented, complete with philosophical rationales for their adoption, examples, and criticism.

* Virtual Reality Musical Instruments: State of the Art, Design Principals, and Future Directions, Serafin, et al., 2016 [14] This article describes VRMIs, gives design examples, and presents some core design principals.

The recommendations in these articles frequency coincide;
in fact, they reference one another quite regularly. Points
of contact include agreement on the importance of creating
social interactions, designing with the current limitations
of VR hardware in mind, and acknowledging the potential
for cybersickness. Another common thread relates to the
creation of MMIs.

**2.2 Magical Musical Interactions**

The term was coined by Serafin, et al. in reference to
designing VRMIs and the resulting interactions between
performer and instrument. To reiterate, these interactions
leverage the unique ability of a virtual environment to dis-
pense with real-world constraints such as “physics, human
anatomy, or the current state of technological development”
and in doing so create playing techniques and audible results
that cannot occur with acoustic instruments.

Turchet, et al. make the remark that, “XR can evoke
levels of abstraction, immersion, and imagination not possi-
ble with conventional musical interfaces, ushering in entirely
new ways of engaging with music for both the performer
and the listener” [p. 15805]. This summarizes a commentary
throughout the article that suggests progress in Audio-First
XR content lies in the discovery of interactions and designs
that cannot be achieved with the use of acoustic instruments
and physical spaces.

Finally, Atherton and Wang declare plainly in one of their
design principals: “Don’t port: Make things that would be
impossible in the real world” [p. 2]. They argue that if the
medium of VR is to be used to make art, its specific affor-
dances should be fully exploited. Since virtual worlds are
capable of removing many physical constraints, it behooves
artists to explore what can be accomplished if we are freed
of them. An example is found in a piece by Atherton enti-
tled 12 Sentiments for VR. This Audio-First VR work moves
through the life cycle of a plant, offering the user different
interactive environments in which to experience scenes like
the dispersion of a seed floating through the wind to the
gathering up of rays of sunlight by a mature plant, all while
activating musical sequences through varying degrees of user
control. In the work, the user *is* the plant, frequently from
the first person perspective.

When considering the conversations about Musical XR
and VR design by Truchet, et al. and Atherton and Wang,
the definition of a MMI can expand beyond a performer's

<footnote n="1">¹All further mentions of these authors will reference these
articles.</footnote>
<figure><img src="image_5.png" /><figcaption>Figure 1: A Venn diagram that shows the intersection of several fields of study related to the composition of *Pisces*.</figcaption></figure>

interaction with a VRMI as defined by Serafin, et al. to encompass the set of interactions between instrument, performer, environment, and audience. This results in the following expanded definition:

*Magical Musical Interaction:* A musically relevant interaction in an XR environment between any combination of performer, instrument, audience, and environment occurring beyond the current limits of real-world constraints.

This definition contains several nuances which will expand the possible set of MMIs and helps to reveal them in a work. First, it is important to note the inclusion of the phrase *musically relevant interaction*. With the addition of the audience and environment, the number of total interactions increases dramatically. Some of these interactions may not have a direct bearing on the presentation of musical material, and as such, should not be included in a discussion of MMIs, even if they depart from real-world constraints. Musically relevant interactions would include any interactivity that contributes to the unfolding or experience of the organized sonic content in an Audio-First work.

In order to sort through the numerous potential interactions that could exist in any given Audio-First work, we suggest first investigating each element and asking the question: "does this agent, object, or environment *reproduce* something we see in the real world?" While VR environments as they are currently instantiated on the web are limited in their ability to fully recreate a natural environment, we can assess whether the approximations present in the work are fundamentally aimed at recreating real-world objects and actions. Evaluations must include not only the sounds generated, but all of the visual components present in any given environment, agent, or object as well as the abilities they possess.

Imagine for instance, a simple virtual drum kit played with a MIDI foot pedal and VR hand controllers. The cur-

rent state of haptics integration will limit the ability of a performer to feel the resistance present in a drum stroke, but the instrument's, sound, overall visual design, and mode of engagement would be pointed towards the real-world analog. We can say that the interaction between instrument and performer is not magical in nature because, while not a perfect recreation, uses shapes, sounds, and gestures all familiar to a performer of a real world kit.

As we will see in the upcoming sections, it is possible to create instruments that have no easily identifiable real-world analog. These objects may produce sounds that cannot be readily tied to an acoustic instrument or natural phenomenon and their visual design and behavior in the environment may be impossible to recreate in the real world. These objects by their very nature hold the potential for creating MMIs. In fact, their very experience by a passive audience member may qualify, as their encounter with such an object cannot occur within the bounds of real-world constraints.

We can now begin to see a landscape in which the MMI can emerge. If we first investigate the nature of the agents, objects, and environment in the virtual space we can see how closely they reproduce the real world. As they deviate from the known constraints of reality, MMIs become possible. We can then review relevant musical interactions to identify specific MMIs.

In the next section, we will look at how *Pisces* was composed in order to describe the nature of the performer, instrument, audience, and environment and offer an analysis based on these descriptions.

## 3. CREATING PISCES: A SPATIAL COMPOSITION FOR THE WEB

The following section is divided into two parts. First, we will explain the artistic genesis of *Pisces*, followed by a technical description of how it was composed and implemented
<figure><img src="image_4.png" /><figcaption>Figure 1: A Venn diagram that shows the intersection of several fields of study related to the composition of *Pisces*.</figcaption></figure>

interaction with a VRMI as defined by Serafin, et al. to encompass the set of interactions between instrument, performer, environment, and audience. This results in the following expanded definition:

*Magical Musical Interaction:* A musically relevant interaction in an XR environment between any combination of performer, instrument, audience, and environment occurring beyond the current limits of real-world constraints.

This definition contains several nuances which will expand the possible set of MMIs and helps to reveal them in a work. First, it is important to note the inclusion of the phrase musically relevant interaction. With the addition of the audience and environment, the number of total interactions increases dramatically. Some of these interactions may not have a direct bearing on the presentation of musical material, and as such, should not be included in a discussion of MMIs, even if they depart from real-world constraints. Musically relevant interactions would include any interactivity that contributes to the unfolding or experience of the organized sonic content in an Audio-First work.

In order to sort through the numerous potential interactions that could exist in any given Audio-First work, we suggest first investigating each element and asking the question: "does this agent, object, or environment *reproduce* something we see in the real world?" While VR environments as they are currently instantiated on the web are limited in their ability to fully recreate a natural environment, we can assess whether the approximations present in the work are fundamentally aimed at recreating real-world objects and actions. Evaluations must include not only the sounds generated, but all of the visual components present in any given environment, agent, or object as well as the abilities they possess.

Imagine for instance, a simple virtual drum kit played with a MIDI foot pedal and VR hand controllers. The cur-

rent state of haptics integration will limit the ability of a performer to feel the resistance present in a drum stroke, but the instrument's, sound, overall visual design, and mode of engagement would be pointed towards the real-world analog. We can say that the interaction between instrument and performer is not magical in nature because, while not a perfect recreation, uses shapes, sounds, and gestures all familiar to a performer of a real world kit.

As we will see in the upcoming sections, it is possible to create instruments that have no easily identifiable real-world analog. These objects may produce sounds that cannot be readily tied to an acoustic instrument or natural phenomenon and their visual design and behavior in the environment may be impossible to recreate in the real world. These objects by their very nature hold the potential for creating MMIs. In fact, their very experience by a passive audience member may qualify, as their encounter with such an object cannot occur within the bounds of real-world constraints.

We can now begin to see a landscape in which the MMI can emerge. If we first investigate the nature of the agents, objects, and environment in the virtual space we can see how closely they reproduce the real world. As they deviate from the known constraints of reality, MMIs become possible. We can then review relevant musical interactions to identify specific MMIs.

In the next section, we will look at how *Pisces* was composed in order to describe the nature of the performer, instrument, audience, and environment and offer an analysis based on these descriptions.

### 3. CREATING PISCES: A SPATIAL COMPOSITION FOR THE WEB

The following section is divided into two parts. First, we will explain the artistic genesis of *Pisces*, followed by a technical description of how it was composed and implemented
on the web. These descriptions will aid the reader in establishing the landscape of interactions formed between the performer, instrument, audience, and environment. After this landscape has been populated, we will search for MMIs in the manner described above, which will set the stage for speculation on how the techniques employed in this work might increase MMIs in new web based content.

## 3.1 Sound Entities: A New Class of VRMIs

*Pisces* is a recorded performance of two VRMIs in a virtual space <fnref n="2" />. No avatars of performers playing instruments are present however- the instruments themselves are seen as the performers. The two objects present in the space are *Sound Entities* (SEs), a term coined by Schnug to designate a particular type of sound source in a spatial composition. This class of digital instrument is designed to generate sound and trajectories in space simultaneously, and in doing so, creates the illusion of a *living being* moving through space, with its own peculiar motion and sound profile.

The complete artistic rationale for constructing a virtual instrument in this manner is beyond the scope of this article, but it is, in short, a response to the question of the relevance of space and motion in electronic music by creating a marriage of sonic identity and movement/placement. A SE's movements are directly tied to its timbral state as a function of a numerical descriptor (in this case, its spectral centroid), and gives its movement a direct and immediate relationship to its spectromorphology [15], thus justifying the use of motion as an expression of a sonic compositional element <fnref n="3" />.

The visualization of the SE is an attempt to create a body appropriate for the nature of the sound coming from it. While meant to be seen as a discrete being with agency, we felt that creating an anthropomorphized creature, while certainly a valid solution, was not absolutely necessary. In keeping with the spirit of MMIs, we decided to create purely abstract shapes that move and morph dynamically with the sound to give the impression of agency we seek while assuring that the SE remained a creature removed from the real world.

The imagery to be described in the visual/web implementation section below is meant to invoke the fluid, droning sounds coming from the SEs and suggests flowing water- an element found in the story of the namesake of the piece, Pisces. In the legend, Aphrodite and Eros flee from the monster Typhon in a river and are saved by or become fish in order to survive.

## 3.2 Technical Details of Composition

The piece was constructed in two separate stages: audio composition, and visual design/web implementation. The next two subsections describe each in turn.

### 3.2.1 Audio composition with the Sound Entity Creator System

SEs are currently constructed using Max/MSP. A SE design begins as many DMIs do in Max: synthesis parameters or prerecorded samples are assigned mappings to a physical controller. In this case, rather simple mappings of control parameters of a Buchla Twisted Waveform Generator program were instantiated on a Novation Launch Control XL.

The audio of the given sound source moves to a series of patches that generate trajectories in three dimensional space via calculations of the signal's spectral centroid. This Sound Entity Creator System (SEC) uses Max's jitter environment to limit possible motion to positions on the surface of one of several 3D wireframe shapes contained in Max's *jit.gl.gridshape* object. (This shape is not visualized. Instead, it is used only to produce a finite set of locations for the SE to occupy <fnref n="4" />.) The spectral centroid is calculated thrice- the first two instances are translated into coordinates of cells found in the shape's jitter matrix. Each cell contains cartesian coordinates of a vertex on the wireframe shape on planes 1, 2, and 3. When a cell is selected, it outputs that coordinate. As the centroid values change, different cells output their coordinates, and when iterated over time, a trajectory is built. The third centroid value is converted to a value ranging from [0.,1.]. This acts as a scalar on the coordinate values, drawing the SE closer to the origin as the value approaches 0. These trajectories are then converted into three audio signals, one per dimension in space, and joined with the audible signal and any other control signals into a single multichannel audio file. Figure 2 shows a flow chart outlining this process.

This arrangement allows a composer or performer to engage with the control scheme of a SE to create a recording of both the audible output and the trajectories generated by SEC in real time, affording an instant and accurate connection between sound and trajectory as well as easy editing of both sound and spatial material in a DAW capable of supporting multichannel files such as Reaper. The audio files that represent the trajectories can easily be sampled into floating point values, scaled, and used to control a spatialization program via a protocol such as Open Sound Control.

The shape selected in SEC for *Pisces* was a torus, and a steady distance scalar of 1. was used. This resulted in trajectories that moved only on the surface of a torus centered about the origin in cartesian space.

*Pisces* is a meditation on subtle variations in timbre and the juxtaposition of various overtones produced through such variations in sonic and 3D space. The SEs, dubbed *Aphrodite* and *Eros*, have fundamental frequencies a perfect fifth apart. Ten distinct timbral states were found to be of compositional interest, and the piece moves from one state to another, with each SE moving to a different location on the surface of the torus as the change occurs. One slider on the Launch Control XL generates these changes through direct control of several parameters on the Buchla, and one is assigned to amplitude. Figure 3 shows the user interface of *Aphrodite*. Each small circle represents a selected timbral state. The slider moves a circular selector smoothly between these choices. Simple traditional musical notation was used to represent these states: ten notes on the clef were arbitrarily selected to represent them in order to state the location of the selector, the time to stay on a particular state, and the time spent moving between the states.

<footnote n="2">²www.houseofthegoldenthread.com/pisces</footnote>

<footnote n="3">³The justifications for using space and movement in electronic music has a long history, and the decision to tie movement to some computable attribute of the sound is a contribution to this ongoing conversation [7].</footnote>

<footnote n="4">⁴This method of limiting potential motion was inspired in</footnote>
on the web. These descriptions will aid the reader in establishing the landscape of interactions formed between the performer, instrument, audience, and environment. After this landscape has been populated, we will search for MMIs in the manner described above, which will set the stage for speculation on how the techniques employed in this work might increase MMIs in new web based content.

## 3.1 Sound Entities: A New Class of VRMIs

*Pisces* is a recorded performance of two VRMIs in a virtual space <fnref n="2" />. No avatars of performers playing instruments are present however- the instruments themselves are seen as the performers. The two objects present in the space are *Sound Entities* (SEs), a term coined by Schnug to designate a particular type of sound source in a spatial composition. This class of digital instrument is designed to generate sound and trajectories in space simultaneously, and in doing so, creates the illusion of a *living being* moving through space, with its own peculiar motion and sound profile.

The complete artistic rationale for constructing a virtual instrument in this manner is beyond the scope of this article, but it is, in short, a response to the question of the relevance of space and motion in electronic music by creating a marriage of sonic identity and movement/placement. A SE's movements are directly tied to its timbral state as a function of a numerical descriptor (in this case, its spectral centroid), and gives its movement a direct and immediate relationship to its spectromorphology [15], thus justifying the use of motion as an expression of a sonic compositional element <fnref n="3" />.

The visualization of the SE is an attempt to create a body appropriate for the nature of the sound coming from it. While meant to be seen as a discrete being with agency, we felt that creating an anthropomorphized creature, while certainly a valid solution, was not absolutely necessary. In keeping with the spirit of MMIs, we decided to create purely abstract shapes that move and morph dynamically with the sound to give the impression of agency we seek while assuring that the SE remained a creature removed from the real world.

The imagery to be described in the visual/web implementation section below is meant to invoke the fluid, droning sounds coming from the SEs and suggests flowing water- an element found in the story of the namesake of the piece, Pisces. In the legend, Aphrodite and Eros flee from the monster Typhon in a river and are saved by or become fish in order to survive.

## 3.2 Technical Details of Composition

The piece was constructed in two separate stages: audio composition, and visual design/web implementation. The next two subsections describe each in turn.

### 3.2.1 Audio composition with the Sound Entity Creator System

SEs are currently constructed using Max/MSP. A SE design begins as many DMIs do in Max: synthesis parameters or prerecorded samples are assigned mappings to a physical controller. In this case, rather simple mappings of control parameters of a Buchla Twisted Waveform Generator program were instantiated on a Novation Launch Control XL.

The audio of the given sound source moves to a series of patches that generate trajectories in three dimensional space via calculations of the signal's spectral centroid. This Sound Entity Creator System (SEC) uses Max's jitter environment to limit possible motion to positions on the surface of one of several 3D wireframe shapes contained in Max's *jit.gl.gridshape* object. (This shape is not visualized. Instead, it is used only to produce a finite set of locations for the SE to occupy <fnref n="4" />. The spectral centroid is calculated thrice- the first two instances are translated into coordinates of cells found in the shape's jitter matrix. Each cell contains cartesian coordinates of a vertex on the wireframe shape on planes 1, 2, and 3. When a cell is selected, it outputs that coordinate. As the centroid values change, different cells output their coordinates, and when iterated over time, a trajectory is built. The third centroid value is converted to a value ranging from [0.,1.]. This acts as a scalar on the coordinate values, drawing the SE closer to the origin as the value approaches 0. These trajectories are then converted into three audio signals, one per dimension in space, and joined with the audible signal and any other control signals into a single multichannel audio file. Figure 2 shows a flow chart outlining this process.

This arrangement allows a composer or performer to engage with the control scheme of a SE to create a recording of both the audible output and the trajectories generated by SEC in real time, affording an instant and accurate connection between sound and trajectory as well as easy editing of both sound and spatial material in a DAW capable of supporting multichannel files such as Reaper. The audio files that represent the trajectories can easily be sampled into floating point values, scaled, and used to control a spatialization program via a protocol such as Open Sound Control.

The shape selected in SEC for *Pisces* was a torus, and a steady distance scalar of 1. was used. This resulted in trajectories that moved only on the surface of a torus centered about the origin in cartesian space.

*Pisces* is a meditation on subtle variations in timbre and the juxtaposition of various overtones produced through such variations in sonic and 3D space. The SEs, dubbed *Aphrodite* and *Eros*, have fundamental frequencies a perfect fifth apart. Ten distinct timbral states were found to be of compositional interest, and the piece moves from one state to another, with each SE moving to a different location on the surface of the torus as the change occurs. One slider on the Launch Control XL generates these changes through direct control of several parameters on the Buchla, and one is assigned to amplitude. Figure 3 shows the user interface of *Aphrodite*. Each small circle represents a selected timbral state. The slider moves a circular selector smoothly between these choices. Simple traditional musical notation was used to represent these states: ten notes on the clef were arbitrarily selected to represent them in order to state the location of the selector, the time to stay on a particular state, and the time spent moving between the states.

<footnote n="2">²houseofthegoldenthread.com/pisces</footnote>

<footnote n="3">³The justifications for using space and movement in electronic music has a long history, and the decision to tie movement to some computable attribute of the sound is a contribution to this ongoing conversation [7].</footnote>

<footnote n="4">⁴This method of limiting potential motion was inspired in</footnote>
<figure><img src="image_6.png" /><figcaption>Figure 2: A block diagram showing the signal flow in the SEC system.</figcaption></figure>

### 3.2.2 Visual and web implementation

The visualization of Aphrodite and Eros was achieved through the use of Three.js. Using various Three.js geometries and GLSL shaders, the SEs are presented as flowing, sphereoid shapes with various curves that flow outward and inward. Parametric curves with offsets are responsible for this imagery. Variations are calculated in the shape and location of these curves as a function of time, giving the overall shape its flowing nature. Figure 4 shows one of the SEs as it exists in the virtual environment.

With the visual representation created, the next step involved animating the SEs. Figure 5 shows the process as a flow chart. First, the multichannel audio file is separated into individual tracks and converted to arrays of floating point values. We can see that in this particular multichannel file there are two additional control signals, amplitude and spectral centroid, that are represented in Figure 2 as 'misc'. These two values were calculated and recorded as the piece was played, becoming synchronized with the audible material and trajectories.

The audible material is played with a web audio player created with the positionalAudio object in Three.js. This object is attached to the visualization mesh and imposes spatial cues on the audio file associated with that mesh such as binaural filtering and amplitude attenuation as a function of distance from the listener.

The remaining arrays were read using the `useFrame()` function, which allowed us to update positional information and visual transformations of the shaders. Since the multichannel audio was encoded at a rate

<figure><img src="image_7.png" /><figcaption>Figure 3: The user interface of the SE Aphrodite.</figcaption></figure>
<figure><img src="image_5.png" /><figcaption>Figure 2: A block diagram showing the signal flow in the SEC system.</figcaption></figure>

### 3.2.2 Visual and web implementation

The visualization of Aphrodite and Eros was achieved through the use of Three.js. Using various Three.js geometries and GLSL shaders, the SEs are presented as flowing, spheroid shapes with various curves that flow outward and inward. Parametric curves with offsets are responsible for this imagery. Variations are calculated in the shape and location of these curves as a function of time, giving the overall shape its flowing nature. Figure 4 shows one of the SEs as it exists in the virtual environment.

With the visual representation created, the next step involved animating the SEs. Figure 5 shows the process as a flow chart. First, the multichannel audio file is separated into individual tracks and converted to arrays of floating point values. We can see that in this particular multichannel file there are two additional control signals, amplitude and spectral centroid, that are represented in Figure 2 as 'misc'. These two values were calculated and recorded as the piece was played, becoming synchronized with the audible material and trajectories.

The audible material is played with a web audio player created with the positionalAudio object in Three.js. This object is attached to the visualization mesh and imposes spatial cues on the audio file associated with that mesh such as binaural filtering and amplitude attenuation as a function of distance from the listener.

The remaining arrays were read using the `useFrame()` function, which allowed us to update positional information and visual transformations of the shaders. Since the multichannel audio was encoded at a rate

<figure><img src="image_6.png" /><figcaption>Figure 3: The user interface of the SE Aphrodite.</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Figure 4: The visual form of a SE in *Pisces*.</figcaption></figure>

of 44.1k *samples/second* and the frame rate used was 60 *frames/second*, the positional and visual parameters were able to update every 735 samples and ensured a direct synchronization between these values and the audible events. The coordinate data places the visualization mesh in space, triggering the *positionalAudio* filtering and attenuation, and the amplitude and centroid signals manipulate the opacity and color of the shaders, respectively.

The environment in which the SEs move was kept deliberately simple. A transparant platform in the center of a cavernous space allows a listener to move up to the origin of cartesian space about which the two SEs move. This enables them to experience the piece as it was composed: with the listener located statically at the center. This is the position from which the trajectories were selected, and offers the most direct experience of the juxtaposition of spatial location and timbral states. The user can of course choose to remain below the platform to experience the movement as a totality or change orientation while at the center.

## 3.3 Pisces and MMIs

In order to uncover any MMIs present in *Pisces* we will utilize the methodology proposed in the previous section *Magical Musical Interactions*.

Our first task is to investigate the nature of the individual elements of the work to see how closely they reproduce real-world objects and experiences. We will take each one in turn:

**environment:** The space the piece is performed in conforms to physical constraints of the real world. While the cavernous imagery and the transparent platform are fanciful, they are by no means removed from reality. Gravity is in effect for the audience member, and as a result, they move with typical physical constraints. We can conclude that the environment itself reproduces a real-world environment quite closely for the audience member.

**audience:** The audience member is bound by the physical constraints of the environment. They are granted six degrees of freedom. An avatar is not produced in the space, so an audience member moves without an observable body. While the lack of an avatar does not support reproduction, there is no musically relevant action such an avatar could produce

in this particular piece. The audience member therefore is partially reproduced. Their movement through the space is the only way in which to produce an interaction.

**performer:** There is no human performer in this piece. The performer's actions have been recorded and are played back via the SE. As per the description of the SEs above, there is an identity between performer and instrument, and this is certainly a deviation given the abstract representation of the SEs.

**instrument:** The two SEs deviate substantially from any real-world objects. They defy gravity by exerting some unknown force, phase in and out of existence, and are visually constituted in such a manner as to be totally disconnected from real-world forms that we encounter physically. Their movements imply agency and are presented as performer/instrument hybrids. The sounds they emit- low pitched, harmonically rich drones, only have a real-world analog in a synthesizer and loudspeaker. This sound's connection to such abstract forms is a real-world deviation.

We can plainly see that any potential MMIs lie in the nature of the SEs themselves. Since no real-time performer exists, the only possible MMI is located in the interaction between the instrument and the audience member. What relevant musical interactions exist between them? We can identify one:

As the audience member moves about the space, they encounter a sound emitting object that cannot exist in the real world. This encounter, realized through the audience member's movement and resultant exploration of different listening positions, qualifies as a MMI. The object emitting all relevant sonic content happens to be a creature that cannot exist in the real world, therefore, anyone viewing such a creature is engaging in an act beyond current real-world constraints.

We can see from this analysis that an encounter with an object not found in the real world can qualify as MMI if the object emits musically relevant sound. This particular MMI is made much more robust by the spatial nature of the composition. When an audience member changes their location, they are directly altering musical content: the connection between the SEs' timbral states and the trajectories are composed, crafted at the origin of cartesian space. Listening from a different perspective changes these relationships and allows the audience to directly influence them.

# 4. THE FUTURE POTENTIAL OF SOUND ENTITIES AND MAGICAL MUSICAL INTERACTIONS

The recasting of a VRMI from an inert instrument to be handled and played to an external being has many implications for the design of Audio-First virtual content. The following discussion presents the potential of SEs to increase MMIs through new paradigms of performer, composer, and audience engagement.

## 4.1 Performer Engagement With SEs

In *Pisces*, the SEs are effectively ‘played back’ for a single audience member. Once the performance space is entered, the SEs begin the piece and play through it until the conclusion. Interactivity is limited to investigating the scene. The
<figure><img src="image_7.png" /><figcaption>Figure 4: The visual form of a SE in *Pisces*.</figcaption></figure>

of 44.1k samples/second and the frame rate used was 60 frames/second, the positional and visual parameters were able to update every 735 samples and ensured a direct synchronization between these values and the audible events. The coordinate data places the visualization mesh in space, triggering the *positionalAudio* filtering and attenuation, and the amplitude and centroid signals manipulate the opacity and color of the shaders, respectively.

The environment in which the SEs move was kept deliberately simple. A transparent platform in the center of a cavernous space allows a listener to move up to the origin of cartesian space about which the two SEs move. This enables them to experience the piece as it was composed: with the listener located statically at the center. This is the position from which the trajectories were selected, and offers the most direct experience of the juxtaposition of spatial location and timbral states. The user can of course choose to remain below the platform to experience the movement as a totality or change orientation while at the center.

## 3.3 Pisces and MMIs

In order to uncover any MMIs present in *Pisces* we will utilize the methodology proposed in the previous section *Magical Musical Interactions*.

Our first task is to investigate the nature of the individual elements of the work to see how closely they reproduce real-world objects and experiences. We will take each one in turn:

**environment:** The space the piece is performed in conforms to physical constraints of the real world. While the cavernous imagery and the transparent platform are fanciful, they are by no means removed from reality. Gravity is in effect for the audience member, and as a result, they move with typical physical constraints. We can conclude that the environment itself reproduces a real-world environment quite closely for the audience member.

**audience:** The audience member is bound by the physical constraints of the environment. They are granted six degrees of freedom. An avatar is not produced in the space, so an audience member moves without an observable body. While the lack of an avatar does not support reproduction, there is no musically relevant action such an avatar could produce

in this particular piece. The audience member therefore is partially reproduced. Their movement through the space is the only way in which to produce an interaction.

**performer:** There is no human performer in this piece. The performer's actions have been recorded and are played back via the SE. As per the description of the SEs above, there is an identity between performer and instrument, and this is certainly a deviation given the abstract representation of the SEs.

**instrument:** The two SEs deviate substantially from any real-world objects. They defy gravity by exerting some unknown force, phase in and out of existence, and are visually constituted in such a manner as to be totally disconnected from real-world forms that we encounter physically. Their movements imply agency and are presented as performer/instrument hybrids. The sounds they emit- low pitched, harmonically rich drones, only have a real-world analog in a synthesizer and loudspeaker. This sound's connection to such abstract forms is a real-world deviation.

We can plainly see that any potential MMIs lie in the nature of the SEs themselves. Since no real-time performer exists, the only possible MMI is located in the interaction between the instrument and the audience member. What relevant musical interactions exist between them? We can identify one:

As the audience member moves about the space, they encounter a sound emitting object that cannot exist in the real world. This encounter, realized through the audience member's movement and resultant exploration of different listening positions, qualifies as a MMI. The object emitting all relevant sonic content happens to be a creature that cannot exist in the real world, therefore, anyone viewing such a creature is engaging in an act beyond current real-world constraints.

We can see from this analysis that an encounter with an object not found in the real world can qualify as MMI if the object emits musically relevant sound. This particular MMI is made much more robust by the spatial nature of the composition. When an audience member changes their location, they are directly altering musical content: the connection between the SEs' timbral states and the trajectories are composed, crafted at the origin of cartesian space. Listening from a different perspective changes these relationships and allows the audience to directly influence them.

# 4. THE FUTURE POTENTIAL OF SOUND ENTITIES AND MAGICAL MUSICAL INTERACTIONS

The recasting of a VRMI from an inert instrument to be handled and played to an external being has many implications for the design of Audio-First virtual content. The following discussion presents the potential of SEs to increase MMIs through new paradigms of performer, composer, and audience engagement.

## 4.1 Performer Engagement With SEs

In *Pisces*, the SEs are effectively 'played back' for a single audience member. Once the performance space is entered, the SEs begin the piece and play through it until the conclusion. Interactivity is limited to investigating the scene. The
<figure><img src="image_9.png" /><figcaption>Figure 5: The process of converting multichannel audio files to visual animation and spatialization in *Pisces*.</figcaption></figure>

recent release of Cycling '74's RNBO environment<fnref n="5" /> however, could help to rebuild this architecture for export to the web. This patching environment is very similar to Max/MSP and could be used to translate SEC to a web native sound engine. Another intriguing option is presented in the *Patch World* application for Meta Quest by the company Patch XR<fnref n="6" />. This VR environment is specifically designed to generate Audio-First content. Users can create their own VRMIs and virtual environments for performance. Both of these options would give composers and performers a chance to create with SEs as they exist in their totality: as audiovisual beings.

This affordance is not trivial: it would create a new class of composer/performer centric MMIs. Instead of the user interface shown in Figure 3, a performer or composer using *Aphrodite* could rely on its location in space and color changes along with its timbral transformations to 'play' the instrument. An analogy can be found in the great audio-visual work *Fantasia*. Figure 4 shows a representation of the famous scene in which Mickey Mouse steals the magician's cap, and becomes capable of animating and directing objects.

Hand tracking control schemes could easily be implemented to mimic such gestural control schemes via HMDs that include hand controllers, or a device like the leapmotion <fnref n="7" />. Virtual avatars could be seen controlling SEs with their hand motions, becoming VR digital sorcerers that move objects with a wave of their hand.

These hand motions themselves could be used to directly influence the position of a SE and cause the current sound-synthesis-to-motion pipeline to work in reverse: position and motion could control the generation of the sound emitted. Transformations of the shape, color, and size of a SE could

<figure><img src="image_10.png" /><figcaption>Figure 6: Mickey Mouse practicing telekinesis in *Fantasia*. "Fantasia 1940" by Landon Ruan licensed under CC BY-NC-ND 3.0 accessed at https://www.deviantart.com/ncillustration/art/Fantasia-1940-531641254</figcaption></figure>

<footnote n="5">⁵https://cycling74.com/products/rnbo</footnote>

<footnote n="6">⁶https://patchxr.com/</footnote>

<footnote n="7">⁷https://www.ultraleap.com/</footnote>
<figure><img src="image_8.png" /><figcaption>Figure 5: The process of converting multichannel audio files to visual animation and spatialization in *Pisces*.</figcaption></figure>

recent release of Cycling '74's RNBO environment<fnref n="5" /> however, could help to rebuild this architecture for export to the web. This patching environment is very similar to Max/MSP and could be used to translate SEC to a web native sound engine. Another intriguing option is presented in the *Patch World* application for Meta Quest by the company Patch XR<fnref n="6" />. This VR environment is specifically designed to generate Audio-First content. Users can create their own VRMIs and virtual environments for performance. Both of these options would give composers and performers a chance to create with SEs as they exist in their totality: as audiovisual beings.

This affordance is not trivial: it would create a new class of composer/performer centric MMIs. Instead of the user interface shown in Figure 3, a performer or composer using *Aphrodite* could rely on its location in space and color changes along with its timbral transformations to 'play' the instrument. An analogy can be found in the great audio-visual work *Fantasia*. Figure 4 shows a representation of the famous scene in which Mickey Mouse steals the magician's cap, and becomes capable of animating and directing objects.

Hand tracking control schemes could easily be implemented to mimic such gestural control schemes via HMDs that include hand controllers, or a device like the leapmotion <fnref n="7" />. Virtual avatars could be seen controlling SEs with their hand motions, becoming VR digital sorcerers that move objects with a wave of their hand.

These hand motions themselves could be used to directly influence the position of a SE and cause the current sound-synthesis-to-motion pipeline to work in reverse: position and motion could control the generation of the sound emitted. Transformations of the shape, color, and size of a SE could

<figure><img src="image_9.png" /><figcaption>Figure 6: Mickey Mouse practicing telekinesis in *Fantasia*. "Fantasia 1940" by Landon Ruan licensed under CC BY-NC-ND 3.0 accessed at https://www.deviantart.com/ncillustration/art/Fantasia-1940-531641254</figcaption></figure>

<footnote n="5">⁵https://cycling74.com/products/rnbo</footnote>

<footnote n="6">⁶https://patchxr.com/</footnote>

<footnote n="7">⁷https://www.ultraleap.com/</footnote>
also be used to alter sonic content, once again creating per-
formance interactions well beyond the current limits of real-
world technology.

This arrangement is a radical change in the conception
of a VRMI. If we could consider a group of virtually re-
produced violins, marimbas, and horns to be an instrumen-
tal ensemble, a collection of SEs would be something more
akin to a collection of creatures in a game like *Pokémon*,
where various beings with different skills are controlled by
the user. Collaborative performances situated in a meta-
verse environment could include multiple avatars with their
own SEs, engaged in magical acts of influence and control
of virtual creatures.

## 4.2 Composer Design of SEs and Audience In-teraction

Putting aside the obvious question of how SEs may look
and sound as per the composer's intent, we offer here an
additional possibility- the utilization of algorithmically con-
trolled SEs. While a performer may have direct control over
a SE during a performance, nothing prohibits the composer
from designing SEs that have a *reactive* nature. As certain
controllable SEs appear, so too may reactive SEs that move
in sympathy with those under direct control. As audience
members move into certain locations they may be triggered
into activation. Audience members may even have selective,
high level control of these SEs. Groups of audience members
may be able to trigger such SEs in a manner similar to per-
formers for a particular audible effect and thus influence the
development of a composition. When paired with the po-
tential for the abstract visual representations of SEs that we
find in *Pisces*, we can imagine a world replete with unique
and engaging MMIs through a melding of visual and sonic
control by the composer, the performer, and the audience.

5. CONCLUSION

As the metaverse at large continues to expand, the musi-
cal metaverse will grow with it as more artists seek to uti-
lize the novel approaches to audience connection and con-
tent creation it provides. The attention brought to MMIs in
Audio-First works like *Pisces* serves to remind creators that
many possibilities exist beyond the frontier of reproduction
of real-world events and objects like traditional concert hall
performances and virtual acoustic instruments. Composers
can move into a world where improbable and impossible situ-
ations are the norm, and new interpretations of space, sound,
and interactivity define the networked VR experience.

6. REFERENCES

[1] R. Ascott. Is there love in the telematic embrace? *Art Journal*, 49(3):241–247, 1990.

[2] J. Atherton and G. Wang. Doing vs. being: A philosophy of design for artful vr. *Journal of New Music Research*, 49(1):35–59, 2020.

[3] E. Bates. *The composition and performance of spatial music*. PhD thesis, Trinity College Dublin, 2009.

[4] C. Çakmak and R. Hamilton. Composing spatial music with web audio and webvr. In *Proc. Web Audio Conf.*, pages 1–5, 2019.

[5] A. Çamcı and R. Hamilton. Audio-first vr: New perspectives on musical experiences in virtual environments. *Journal of New Music Research*, 49(1):1–7, 2020.

[6] J. Harrison. Diffusion: theories and practices, with particular reference to the beast system. *econtact*, 2:1–9, 1999.

[7] E. R. Miranda and M. M. Wanderley. *New digital musical instruments: control and interaction beyond the keyboard*, volume 21. AR Editions, Inc., 2006.

[8] M. A. Moser and D. MacLeod. *Immersed in technology: Art and virtual environments*. Mit Press, 1996.

[9] H. Ning, H. Wang, Y. Lin, W. Wang, S. Dhelim, F. Farha, J. Ding, and M. Daneshmand. A survey on the metaverse: The state-of-the-art, technologies, applications, and challenges. *IEEE Internet of Things Journal*, 2023.

[10] M. Parveau and M. Adda. Toward a user-centric classification scheme for extended reality paradigms. *Journal of Ambient Intelligence and Humanized Computing*, 11:2237–2249, 2020.

[11] A. Roginska and P. Geluso. *Immersive sound: the art and science of binaural and multi-channel audio*. Taylor & Francis, 2017.

[12] V. Sathe, P. Gupta, K. Kaushik, S. Bhat, and S. Deshpande. Virtual reality websites (vr web). In *2017 International conference of Electronics, Communication and Aerospace Technology (ICECA)*, volume 1, pages 647–652. IEEE, 2017.

[13] S. Serafin, C. Erkut, J. Kojs, N. C. Nilsson, and R. Nordahl. Virtual reality musical instruments: State of the art, design principles, and future directions. *Computer Music Journal*, 40(3):22–40, 2016.

[14] D. Smalley. Spectromorphology: explaining sound-shapes. *Organised sound*, 2(2):107–126, 1997.

[15] L. Turchet. Musical metaverse: vision, opportunities, and challenges. *Personal and Ubiquitous Computing*, pages 1–17, 2023.

[16] L. Turchet, C. Fischione, G. Essl, D. Keller, and M. Barthet. Internet of musical things: Vision and challenges. *Ieee access*, 6:1994–62017, 2018.

[17] L. Turchet, R. Hamilton, and A. Çamci. Music in extended realities. *IEEE Access*, 9:15810–15832, 2021.
also be used to alter sonic content, once again creating per-
formance interactions well beyond the current limits of real-
world technology.

This arrangement is a radical change in the conception
of a VRMI. If we could consider a group of virtually re-
produced violins, marimbas, and horns to be an instrumen-
tal ensemble, a collection of SEs would be something more
akin to a collection of creatures in a game like *Pokémon*,
where various beings with different skills are controlled by
the user. Collaborative performances situated in a meta-
verse environment could include multiple avatars with their
own SEs, engaged in magical acts of influence and control
of virtual creatures.

## 4.2 Composer Design of SEs and Audience In-teraction

Putting aside the obvious question of how SEs may look
and sound as per the composer's intent, we offer here an
additional possibility- the utilization of algorithmically con-
trolled SEs. While a performer may have direct control over
a SE during a performance, nothing prohibits the composer
from designing SEs that have a *reactive* nature. As certain
controllable SEs appear, so too may reactive SEs that move
in sympathy with those under direct control. As audience
members move into certain locations they may be triggered
into activation. Audience members may even have selective,
high level control of these SEs. Groups of audience members
may be able to trigger such SEs in a manner similar to per-
formers for a particular audible effect and thus influence the
development of a composition. When paired with the po-
tential for the abstract visual representations of SEs that we
find in *Pisces*, we can imagine a world replete with unique
and engaging MMIs through a melding of visual and sonic
control by the composer, the performer, and the audience.

5. CONCLUSION

As the metaverse at large continues to expand, the musi-
cal metaverse will grow with it as more artists seek to uti-
lize the novel approaches to audience connection and con-
tent creation it provides. The attention brought to MMIs in
Audio-First works like *Pisces* serves to remind creators that
many possibilities exist beyond the frontier of reproduction
of real-world events and objects like traditional concert hall
performances and virtual acoustic instruments. Composers
can move into a world where improbable and impossible situ-
ations are the norm, and new interpretations of space, sound,
and interactivity define the networked VR experience.

6. REFERENCES

[1] R. Ascott. Is there love in the telematic embrace? *Art Journal*, 49(3):241–247, 1990.

[2] J. Atherton and G. Wang. Doing vs. being: A philosophy of design for artful vr. *Journal of New Music Research*, 49(1):35–59, 2020.

[3] E. Bates. *The composition and performance of spatial music*. PhD thesis, Trinity College Dublin, 2009.

[4] A. Cadambi. Lithe: An object-based audio-graph framework for spatial composition and sound design.

*Project Documentation Submitted in Partial Fulfillment for the degree of Master of Science in Media, Arts and Technology*, 2017.

[5] C. Çakmak and R. Hamilton. Composing spatial music with web audio and webvr. In *Proc. Web Audio Conf*, pages 1–5, 2019.

[6] A. Çamcı and R. Hamilton. Audio-first vr: New perspectives on musical experiences in virtual environments. *Journal of New Music Research*, 49(1):1–7, 2020.

[7] J. Harrison. Diffusion: theories and practices, with particular reference to the beast system. *econtact*, 2:1–9, 1999.

[8] E. R. Miranda and M. M. Wanderley. *New digital musical instruments: control and interaction beyond the keyboard*, volume 21. AR Editions, Inc., 2006.

[9] M. A. Moser and D. MacLeod. *Immersed in technology: Art and virtual environments*. Mit Press, 1996.

[10] H. Ning, H. Wang, Y. Lin, W. Wang, S. Dhelim, F. Farha, J. Ding, and M. Daneshmand. A survey on the metaverse: The state-of-the-art, technologies, applications, and challenges. *IEEE Internet of Things Journal*, 2023.

[11] M. Parveau and M. Adda. Toward a user-centric classification scheme for extended reality paradigms. *Journal of Ambient Intelligence and Humanized Computing*, 11:2237–2249, 2020.

[12] A. Roginska and P. Geluso. *Immersive sound: the art and science of binaural and multi-channel audio*. Taylor & Francis, 2017.

[13] V. Sathe, P. Gupta, K. Kaushik, S. Bhat, and S. Deshpande. Virtual reality websites (vr web). In *2017 International conference of Electronics, Communication and Aerospace Technology (ICECA)*, volume 1, pages 647–652. IEEE, 2017.

[14] S. Serafin, C. Erkut, J. Kojs, N. C. Nilsson, and R. Nordahl. Virtual reality musical instruments: State of the art, design principles, and future directions. *Computer Music Journal*, 40(3):22–40, 2016.

[15] D. Smalley. Spectromorphology: explaining sound-shapes. *Organised sound*, 2(2):107–126, 1997.

[16] L. Turchet. Musical metaverse: vision, opportunities, and challenges. *Personal and Ubiquitous Computing*, pages 1–17, 2023.

[17] L. Turchet, C. Fischione, G. Essl, D. Keller, and M. Barthet. Internet of musical things: Vision and challenges. *Ieee access*, 6:1994–62017, 2018.

[18] L. Turchet, R. Hamilton, and A. Çamci. Music in extended realities. *IEEE Access*, 9:15810–15832, 2021.