# Re(M)ind: Composition and Generative Sound Design of a Multimodal Opera

Jônatas Manzolli, Micael Antunes

► To cite this version:

Jônatas Manzolli, Micael Antunes. Re(M)ind: Composition and Generative Sound Design of a Multimodal Opera. Celestino Soddu and Enrica Colabella. Proceedings of the XXVII Generative Art conference, Domus Argenia Publisher, 2024, 9788896610473. hal-04894588

HAL Id: hal-04894588

https://hal.science/hal-04894588v1

Submitted on 20 Feb 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Re(M)ind: Composition and Generative Sound Design of a Multimodal Opera

Prof. Jônatas Manzolli, BMath, BMus, MMath, PhD.

*Interdisciplinary Nucleus for Sound Studies (NICS), Unicamp, Brazil*
*Centre for Informatics and Systems of the University of Coimbra (CISUC), Portugal*

https://orcid.org/0000-0003-4257-7118

e-mail: jotamanzo@gmail.com

Dr. Micael Antunes. BMus, MMus, PhD.

Aix-Marseille Univ., CNRS, PRISM, LIS, InCIAM, France.

<figure><img src="image_2.png" /></figure>

## Abstract

This paper explores the challenges encountered in the musical composition and generative sound design of *Re(M)ind* (2023), a multimodal opera creation. It focuses on achieving a delicate balance between artistic creativity and the technological solutions required to integrate musical elements with dynamic performer-machine interactions. The discussion will cover the aspects of composition, sound design and real-time performance that shape an immersive audience experience. Additionally, the paper addresses the technical complexities of incorporating generative

music with operatic elements, interactive dance, and bio signals in real time, all while maintaining a cohesive artistic narrative.

## 1. Introduction

The intersection of technology and artistic expression is a dynamic and evolving field within contemporary performance arts, with multimodal opera emerging as an exciting frontier. These intricate productions blend traditional operatic elements with cutting-edge technological innovations—including sound design, motion capture, and bio-signal processing—to create immersive, interactive experiences for audiences. This paper examines the creative and technical challenges encountered during the development of *Re(M)ind* (2023), a multimodal opera that merges generative sound design with real-time performer-machine interactions.

*Re(M)ind* draws inspiration from a lineage of innovative works, beginning with the inception of RoBoser by Verschure and Manzolli in 1998 [1]. This legacy includes pioneering projects such as *Ada*:
Intelligent Space (2002) [2], re(PER)curso (2007), and the *Multimodal Brain Orchestra* (2009) [3]. Together, these foundational works inform the original score of *Re(M)ind*, composed by Manzolli. The production also incorporates advanced technologies—including bio-signals, motion tracking, and EEG—developed by the SPECS Lab research team, further enhancing the immersive and interactive dimensions of the performance.

The world premiere of *Re(M)ind: A Multimodal Journey into Consciousness* took place on October 19 and October 21, 2023, at Theatre "C" at Radboud University, Nijmegen, in collaboration with the Donders Institute and Donders Citylab. This artwork invites audiences to explore the complexities of human consciousness through an innovative fusion of art and science. The concise libretto, structured in four acts and created by Anna Mura, Manzolli, and Verschure, delves into the enigmatic processes of the human mind, inviting reflection on the nuances of cognition and self-awareness. By blending music, dance, and interactive video, *Re(M)ind* creates a rich, multisensory experience.

## 2. Concepts, Components and Technologies

This section provides an overview of aspects forming the foundation of *Re(M)ind*'s multimodal opera.

### 2.1 Multimodal Opera

In [4], the authors argued that from its inception in late sixteenth-century Italy, opera has been a multimodal art form, with each mode contributing to its

complex interplay of musical, verbal, visual, and dramatic meanings. Building on recent studies of embodied cognition in film and multimedia, [5] examines audio-visual congruence as key to understanding the viewer's experience of opera in its remediated form as film. The composition of such multimodal artwork requires a delicate balance between artistic vision and technical execution.

### 2.2 Design of Sound Textures

The compositional strategy at the heart of this work is based on the concept of musical texture, which serves as a dynamic framework that mediates the creative process. Musical texture offers both qualitative and quantitative aspects to composition [6]. From a qualitative perspective, composers often use texture as a metaphor, weaving together layers of sound to evoke affective responses through different sound configurations [7, 8]. This metaphorical use of texture allows a deeper exploration of how auditory elements interact to shape sensations.

On the quantitative side, musical texture provides a means of organizing material by describing the number and relationships of its constituent elements. This quantitative description is essential for understanding how these elements interact over time and across frequency ranges and how this impacts perception. Here, concepts from *Gestalt* theory [9], *psychoacoustics* [10], and *auditory scene analysis* [11] are often applied to investigate how elements within a texture integrate or segregate perceptually.
<figure><img src="image_3.png" /><figcaption>Figure 1 – Conceptual space of musical texture</figcaption></figure>

Although this paper does not aim to delve into the theoretical aspects of musical texture, we outline a conceptual framework for its use in the interactive design of *Re(M)ind* in Figure 1. In this context, we define musical texture in terms of three elements: (1) density, which represents the number of elements within a given texture—including melodies, soundscapes, spoken voices, and other sonic materials; (2) diversity, indicating the variety of elements within the texture; and (3) interrelationships, describing the connections and interactions among the components. With these elements, we generate a wide array of musical textures by manipulating available sound sets within a performance. In *Re(M)ind*, we explore these components, their interactions, and their real-time control, as detailed in the following sections.

## 2.3 Motion Caption

The motion caption system in *Re(M)ind* used two main devices: a set of body sensors distributed over the dancer's body and accelerometers embedded in smartphones, which were worn by both the dancer and the percussionist during the performance.

The body sensors collected x and y position data of the dancer's body in space, from which movement descriptors were generated to mediate the interaction between the dancer's movements and the resulting sound responses.

The primary objective in selecting these descriptors was to foster a responsive relationship between the performers' movements and the sound textures perceived by the audience. To achieve this, four descriptors closely associated with movement perception were chosen: velocity, acceleration, and jerk (represented by the derivative of acceleration), as well as the magnitude of movement. Additionally, a descriptor linked to position, specifically the body angles, was incorporated. This data enabled a dual mapping approach, where movement could both trigger sound samples and control spatialization, as further discussed in Section 3.2.

## 2.4 Bio-signals

The physiological measurements used during the performance included an Electrodermal Activity (EDA) sensor attached to the mezzo-soprano and two electroencephalogram (EEG) headsets to measure electrical activity in the brain. We calculated the overall activity of the EEG signals, representing a cumulative
measure of brain activity, to capture the EEG dynamics of the two brain-performers as they were exposed to the *Re(M)ind* performance.

Thus, Electrodermal Activity (EDA) can serve as an objective indicator for assessing emotional arousal [17]. In *Re(M)ind*, we used two measures of arousal to control the Affective Synthesizer: one from the EDA of the active performer, reflecting emotional arousal, and the other from the EEG of two passive performers, capturing brainwave patterns associated with emotional engagement. These combined measures enabled dynamic control of the system, allowing it to respond to both immediate and subtle emotional states.

Both EDA and EEG data were transmitted to the system, where their values were normalized within the interval [0...1] to be used as control input for the Affective Synthesizer, as will be discussed in Section 3.3.

**3. Multimodal Integration**

The multimodal scenario integrates innovative tools and processes to create an immersive and interactive performance. We explore musical composition strategies that integrate generative sound design with operatic structures.

**3.1 Musical Composition & Sound Design**

The compositional process began during the "Donders Cognition, Brain and Technology" (DCBT2023) Summer School on Cognition, Brain, and

Technology at Radboud University, Nijmegen is structured around four short acts, each reflecting a distinct state of consciousness: (1) Awakening the Instincts, (2) Self, (3) Memory, and (4) Confluence of Self and Memory.

The composition strategies of *Re(M)ind* were intertwined with sound design. This approach stems from the central narrative idea that the human mind is like a river full of infinite possibilities and never-ending. At each stage of composition, a sound design strategy was employed, creating a soundscape that opens each act. Following this initial soundscape, each act continues with an aria, in which the lyrics sung by the mezzo-soprano explore a different aspect of the mind. To achieve this sound poetics, each of the opera's four short acts gradually immersed the audience in a river of sounds and images, activated by sensors and enhanced by spatialization elements. This composition aims to create an immersive experience that mirrors Heraclitus' view of the world as always "becoming" rather than "being" - reflection of the continuous process of expanding consciousness and embracing the ever-changing nature of existence.

The real-time elements of the musical composition were structured around the concept of musical texture. This approach relies on a strategy of shared control to dynamically shape sound textures in real time [12]. The elements contributing to the composition over time are grouped into three main categories, as illustrated in Figure 2: (1) Fixed-media electronics, which include "mind river echoes" and pre-composed electroacoustic soundscapes. (2)
Performance elements encompassing the singer, percussionist, and pianist. (3) Real-time electronics consisting of granular synthesis, affective synthesizer, and ambisonics spatial control, each modulated by shared control mechanisms.

<figure><img src="image_4.png" /><figcaption>Figure 2 – Sound flux of shared textural composition</figcaption></figure>

The concept of shared control [12] is designed to achieve two primary objectives: (1) to foster a sense of integration between movement and sound by establishing a clear causal relationship between the performers' actions—particularly the dancer's movements—and the evolving soundscape, thereby enhancing the audience's perception of their interplay; and (2) to enable high-level sound scene management by electroacoustic performers, ensuring a cohesive balance between the soundscape and spatialization.

<figure><img src="image_5.png" /><figcaption>Figure 3 - Shared control of musical texture parameters</figcaption></figure>

Control inputs are distributed to different performers and linked to their specific interpretive behaviours, performance strategies and tools. The two electroacoustic performers have primary control over the sonic scene, managing parameters such as spatialisation and synthesis. Additional controls are embedded throughout the performers, as described in sections 2.3 and 2.4. The mappings of these controls are visualised in Figure 3, which illustrates the interplay between each performer's controls and the sound texture parameters.

## 3.2 Dance Sonification

The primary objective of the dance sonification in *Re(M)ind* was to establish a responsive connection between body movement and the composition of musical textures, as mentioned above. To achieve this, the movement descriptors of *velocity*, *acceleration*, and *jerk* were employed as triggers for a sound instrument implemented within the *MoveGuitar* application [12]. These triggers also activated an FM synthesizer controlled by the electroacoustic performers.

In addition, the magnitude and angle descriptors were used to control the spatial distribution of both the dancer-triggered instruments and the pre-recorded sounds. This spatialisation was achieved by mapping the descriptors to the polar coordinates of the ambisonics system [13]. Specifically, the magnitude was mapped as a distance parameter from a central switch point within the *ambisonics* space, while the angle was the angular component within the polar
coordinates of the same space. This configuration enabled an immersive response, allowing sound textures to dynamically correlate with the performer's body movement, thereby enhancing the expressive connection between movement and auditory experience.

## 3.3 Affective Synthesizer

The Affective Synthesizer was designed based on the Circumplex model [14], which served as inspiration for the creation of an emotionally responsive control space. This approach facilitates the real-time display of a sound bank, which interacts with EEG headset data and electrodermal activity (EDA) sensor data, allowing for a responsive from the performers acting as control of the sound designed responses of the synthesizer. The synthesizer is built upon a sound bank derived from ADA, the intelligent space created in 2002 [2]. This sound bank consists of a dataset of labelled sound files representing different emotional states associated with ADA: joy, sadness and anger [2].

To structure the synthesizer, a two-dimensional space was established to display the sound files, illustrated by Figure 4. The left portion of this space is divided into ten segments: five for sad sounds and five for angry sounds. Conversely, the right side of the space is allocated into five segments for sound files representing joy. Each segment is associated to a Gaussian curve that controls the loudness of the respective sound file, enabling smooth transitions between all sounds. This design not only enhances the emotional based response of the synthesiser, but also allows for fluid interactions informed by the physiological

responses of the performers.

<figure><img src="image_6.png" /><figcaption>Figure 4 - Pseudo-circumplex space and the disposition of the sound bank</figcaption></figure>

## 3.4 Sound Diffusion

The spatialization system in Re(M)ind utilized an ambisonics framework for constructing a dynamic sound environment, implemented via the HOA library [13]. This approach allowed sound sources to be positioned within a two-dimensional space, creating an immersive and responsive sound experience. Two primary spatialisation strategies were employed: one for localised sources and another for a granular synthesis immersive environment.
In the first approach, six different localised sound sources were configured as follows: (1) Percussion, (2) MoveGuitar, (3) Affective Synthesizer, (4) Tapes, (5) Mind River Echoes 1, (6) Mind River Echoes 2, as shown in the HOA interface in Figure 5.

<figure><img src="image_7.png" /><figcaption>Figure 5 - HOA interface sound source management</figcaption></figure>

Sources 2, 5, and 6 were designed to respond to the performers' movements, following the strategies previously discussed. The other sources were controlled directly by the electroacoustic performer, using two techniques: (1) predefined spatial trajectories and (2) sinusoidal oscillators mapped to polar coordinates, which facilitated the creation of various circular and spiral trajectories. The parameters for these movements were adjusted in real time via MIDI controller knobs, allowing nuanced manipulation of spatial paths in real time.

The second spatialization strategy, dedicated to granular synthesis, treated the synthesis environment as an

immersive sound field within the ambisonics space. Implemented with HOA library presets, this approach allowed for adjustments in the spread of granular sounds into the sonic field, creating a sense of volume and space that was dynamically modulated. This design followed the principles outlined by Barry Truax [15], who emphasized the spatial distribution of sound grains in immersive contexts to enhance depth and presence. Through these strategies, the spatialization system offered a fluid and interactive soundscape that aligned closely with the performers' actions and the composition's immersive intent.

## 4. Discussion & Conclusion

Musical textural composition plays a crucial role in integrating diverse elements within a multimodal opera, acting as a connective tissue that binds together the sonic, visual, and performative aspects.

<figure><img src="image_8.png" /></figure>
<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /><figcaption>Figure 6 – *Re(M)ind* performance portraits in three photos.</figcaption></figure>

In *Re(M)ind*, the use of musical textures serves as an expansive and flexible canvas, allowing for the seamless incorporation of other artistic and technological artifacts. Unlike traditional melodic or harmonic structures, textural music emphasizes layers of sound that interact dynamically with external inputs, such as bio-signals or movement data. This approach creates a sonic environment where various components—interactive dance, generative music, and real-time processing—can coexist and influence each other without disrupting the overall coherence of the performance.

The openness of sound textures fosters a space for continuous interaction, adaptation, and transformation, making them an ideal medium for multimodal operatic works that rely on the fluid integration of multiple creative and technological elements. This compositional method not only enriches the auditory experience but also supports the holistic integration of the various modalities at play.

In advancing textural composition, our work anticipates the integration of deep learning tools—specifically, variational autoencoders (VAEs)—in future performances to explore new dimensions of control over the textural space [16]. One of the key advantages of using VAEs in this context is their ability to capture high-level features in texture data within a lower dimensional probabilistic latent space. This allows smooth transitions between textures by interpolating within this space, offering an approach that could inspire spatial representations to generate and evolve textures in real-time performance. This provides a way to link physical gestures with textural transformation, creating an embodied interaction between the performer and the sound field.

The development of *Re(M)ind* showcases the potential of integrating real-time sound synthesis, motion capture, and bio-signal processing to enhance both performer expression and audience immersion. These innovations open new avenues for interactive performance, where technology and art seamlessly converge to create dynamic, evolving narratives.
Looking ahead, the continued incorporation of generative techniques into opera presents a promising pathway for artistic evolution. As technological tools become increasingly sophisticated, the ability to integrate real-time data into performance will further expand interactive possibilities. The use of textural music composition, as seen in *Re(M)ind*, is particularly significant in this regard, as its open and adaptive nature allows for the fluid integration of diverse elements such as bio-signals, movement, and spatial audio. This approach not only enriches the musical landscape but also creates a flexible framework that can accommodate future advancements, including AI-driven systems and immersive sound environments.

## Acknowledgment

*Re(M)ind* is a multimodal opera created by Manzolli with contributions from a team of performers and researchers. The narrative, scenography, and choreography were led by Anna Mura, Jônatas Manzolli, and Paul Verschure. Micael Antunes and Manzolli developed the textural musical elements described in this article, with performances by Helvio Mendes on percussion, Stephanie Firnkes as mezzo-soprano, Bert Kappen on piano, and Aida Guirro Salinas as movement artist. The SPECS team managed bio-signal capture and processing, while Hector Lopez Carral provided interactive graphics. Real-time EEG and motion capture (using Xsens technology) were conducted by Tarandeep Singh and Sajad Kahali.

Micael Antunes acknowledge InCIAM institute, through which he has received funding from the Excellence Initiative of

Aix-Marseille University – A*Midex, a French “Investissements d’Avenir programme” AMX-19-IET-005, and Manzolli thanks CNPq, project 308695/2022-4 Brazil, for their support.

## References

[1] J. Manzolli and P. F. M. J. Verschure, "Roboser: A Real-World Composition System," *Computer Music Journal*, vol. 29, no. 3, pp. 55–74, Sep. 2005.

[2] K. C. Wassermann, Kynan Eng, P. F. M. J. Verschure, and J. Manzolli, "Live soundscape composition based on synthetic emotions," *IEEE Multimedia*, vol. 10, no. 4, pp. 82–90, Oct. 2003.

[3] S. Le Groux, J. Manzolli, and P. F. M. J. Verschure, "Disembodied and Collaborative Musical Interaction in the Multimodal Brain Orchestra," in *Proceedings of NIME2010*, Sidney, Australia, 2010.

[4] R. E. Page, Ed., *New perspectives on narrative and multimodality*. in Routledge studies in multimodality, no. 1. New York: Routledge, 2010.

[5] Y. U. Everett, "Opera as Film: Multimodal Narrative and Embodiment," in *The Oxford Handbook of Music and the Body*, Y. Kim and S. L. Gilman, Eds., Oxford University Press, 2019, pp. 398–417. doi: 10.1093/oxfordhb/9780190636234.013.4

[6] W. Berry, Structural functions in music. in Dover books on music, music history. New York: Dover, 1987.

[7] J. Noble, "Perceptual and Semantic Dimensions of Sound Mass" Doctroal thesis, Schulich School of Music McGill University, Montreal, 2018.

[8] M. Antunes da Silva, "Explorando Texturas Sonoras: Análise e Composição com Suporte de Descritores de Áudio e Modelos Computacionais," Tese de Doutorado, Universidade de Campinas, Campinas,
Brasil, 2024.

[9] E. Cambouropoulos and C. Tsougras, "Auditory Streams in Ligeti's Continuum: A Theoretical and Perceptual Approach," p. 19.

[10] M.Antunes, G. F. Do Espirito Santo, J. Manzolli, and M. Queiroz, "A Psychoacoustic-Based Methodology for Sound Mass Music Analysis," in Music in the Al Era, vol. 13770, M. Aramaki, K. Hirata, T. Kitahara, R. Kronland-Martinet, and S. Ystad, Eds., in Lecture Notes in Computer Science, vol. 13770., Cham: Springer International Publishing, 2023, pp. 267-281. doi: 10.1007/978-3-031-35382-6_21.

[11] C. Douglas, J. Noble, and S. McAdams, "Auditory Scene Analysis and the Perception of Sound Mass in Ligeti's Continuum," Music Perception, vol. 33, no. 3, pp. 287-305, Feb. 2016, doi: 10.1525/mp.2016.33.3.287.

[12] G. Zanchetta, M. Antunes, D. Rossetti, and J. Manzolli, "Composing Interactions: Co-creative Generation Of Sound And Movement In Multimodal Architectures," in Actes Des Journées D'informatique Musicale 2024, Marseille, France, 2024.

[13] A. Sèdes, P. Guillot, and E. Paris, "The HOA library, review and prospects," in International Computer Music Conference| Sound and Music Computing, 2014, pp. 855-860. Accessed: Nov. 05, 2024. [Online]. Available: https://hal.science/hal-01196453/

[14] A. Evmenenko and D. S. Teixeira, "The circumplex model of affect in physical activity contexts: a systematic review," International Journal of Sport and Exercise Psychology, vol. 20, no. 1, pp. 168-201, Jan. 2022, doi: 10.1080/1612197X.2020.1854818.

[15] B. Truax, "Composing with real-time granular sound," Perspectives of New Music, pp. 120-134, 1990.

[16] Z. Wang, D. Wang, Y. Zhang, and G. Xia, "Learning Interpretable Representation for Controllable Polyphonic Music Generation," Aug. 17, 2020, arXiv: arXiv:2008.07122. doi: 10.48550/arXiv.2008.07122.

[17] P. Correia, P. Morais, C. Quintão, C. Quaresma, and R. Vigário, "Assessing the Emotional Reaction to Negative Pictures Through Electrodermal Activity Data," in Advances in Human Factors and System Interactions, vol. 265, I. L. Nunes, Ed., in Lecture Notes in Networks and Systems, vol. 265., Cham: Springer International Publishing, 2021, pp. 116-124. doi: 10.1007/978-3-030-79816-1_15.