# GDPD / IVMI-BUILDER: A Libre Software Framework for Extended Reality Musical Instruments and Sonic Installations

Florent Berthaut

► To cite this version:

Florent Berthaut. GDPD / IVMI-BUILDER: A Libre Software Framework for Extended Reality Musical Instruments and Sonic Installations. Proceedings of the 19th Linux Audio Conference, Jun 2025, Lyon, France. hal-05095979

HAL Id: hal-05095979

https://hal.science/hal-05095979v1

Submitted on 3 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# GDPD / IVMI-BUILDER: A LIBRE SOFTWARE FRAMEWORK FOR EXTENDED REALITY MUSICAL INSTRUMENTS AND IMMERSIVE SONIC INSTALLATIONS

Florent Berthaut*

CRISTAL, CNRS, Univ. Lille, France

florent.berthaut@univ-lille.fr

## ABSTRACT

In this paper, we present *GdPd* and *IVMI-builder*, two components of a libre software framework for designing audiovisual applications, immersive sonic installations and Extended Reality Musical Instruments. It is composed of addons for the Godot game engine and abstractions for the Pure Data programming language. It enables fast prototyping and deployment on multiple devices, including Mixed and Virtual Reality Head-Mounted Displays. We first present the architecture components and workflow, including live editing, networked applications and deployment. We then evaluate it through the example of re-implementing a set of existing instruments and examining the design of an immersive musical installation / instrument.

## 1. INTRODUCTION

With the generalisation of somewhat affordable Mixed and Virtual Reality (now generally combined as Extended Reality / XR) equipment, e.g. mobile devices, Head-Mounted Displays (HMD), 3D tracking system, an increasing number of Extended Reality Musical Instruments (XRMI) and Immersive Sonic Installations are being developed [1, 2]. However, prototyping immersive musical instruments and sonic installations remains complex, as it involves designing and testing the visual elements of the 3D scene, their behaviour and the interaction techniques, the sound processes and the mappings between graphics and sound. In order to facilitate the exploration of novel expressive techniques, all these aspects should be easily and continuously editable during the design phase. In addition, these applications can benefit from free and open-source software frameworks for design, which can be enriched with new interaction and visualisation techniques by the community, as research and technology progress.

In this paper, we introduce *GdPd* and *IVMI-builder*, two components of a libre software framework for designing audiovisual applications, XRMIs and Immersive Sonic Installations, based on the Godot game engine and the Pure Data [3] visual programming language. It facilitates live-editing of both the visual and sound aspects of the instruments while adapting to different workflows. In particular, it allows one to switch between separated and integrated sound synthesis without changes in the code, enabling various design processes and application architectures. It also integrates a set of 3D interaction techniques which can be extended, and networking capabilities. It has already been used in a number of projects that have led to scientific publications [4, 5] and artistic productions, one of which [6] is described in Section 3.2. We present the framework in Section 2 and evaluate it in Section 3 by reimplementing existing instruments and analysing the design process of an immersive sonic installation.

### 1.1. Related work

Many frameworks have been created for the development of audiovisual applications. These include text and visual programming languages such as Pure Data, Open Frameworks, Max/MSP

[7], Processing, VVVV. While they may provide both audio and 3D rendering capabilities, they often lack dedicated XR and graphical development features that are present in game engines such as Unreal, Unity or Godot, which design of 3D interactions.

In order to take advantage of the design opportunities offered by game engines, commercial spatial audio plugins such as Wwise have also been developed, along with plugins that allow one to closely integrate audio programming languages, such as Chunity [8], or uPD<fnref n="1" />, or to connect external audio programming software using OpenSoundControl messages, such as with UdkOSC [9], OSC-XR [10] or Myu [11].

Dedicated tools that take into account the variety of 3D display devices also exist, one particular example being Inviso [12], which does not rely on a game engine, but rather on a web programming framework. Finally, some tools directly integrate patching capabilities within the virtual environments through libpd such as PdXR [13] or through Web Audio Modules [14].

However, all these examples are at least partially non-free (e.g., based on Unity or Unreal) and/or limit access either to advanced audio or 3D programming features. In contrast, our framework rely on the Godot game engine (MIT licence) and Pure Data (BSD licence), and provides both advanced 3D editing, scripting and XR support but also a flexible workflow for audiovisual programming.

## 2. GDPD AND IVMI-BUILDER

Our framework relies on the Godot<fnref n="2" /> game engine for gestural input and visual rendering, and on the Pure Data<fnref n="3" /> visual programming language for sound synthesis.

*GdPd* [15] provides classes for Godot and abstractions for Pure Data which are symmetrical and enable exchange of data between the two software, e.g., control data from Godot which is then mapped to sound parameters in Pure Data, and feedback data from audio analysis in Pure Data mapped to visual feedback in Godot. As shown in Figure 1, the exchange is done either through OpenSoundControl messages, in which case the audio is processed and output by PureData, or through internal Pure Data messages, in which case the audio is processed and output inside the Godot application binary. No change in the code or patch is needed to switch between these modes, enabling various design workflows and application architecture, as demonstrated in the next sections. *GdPd* can be used to develop interactive application that combines 2D or 3D graphics and sound synthesis: video games, musical composition or expression software, sonic art installations and so on. *GdPd* is available at https://gitlab.univ-lille.fr/ivmi/gdpd.

*IVMI-builder* [16] builds upon *GdPd* to offer a framework for the design of XRMIs and immersive sonic installations, which, in addition to sound synthesis through Pure Data patches, facilitates the configuration of Extended Reality displays and of networked instruments, and provides access to 3D interaction and visualisation techniques. *IVMI-builder* is available at https://gitlab.univ-lille.fr/ivmi/ivmi-builder.

<footnote>* This work was supported by the IRCICA, the Région Hauts de France, the Fédération de Recherche SCV</footnote>

<footnote n="1">¹https://github.com/Magicolo/uPD</footnote>

<footnote n="2">²https://godotengine.org/</footnote>

<footnote n="3">³https://puredata.info/</footnote>
<figure><img src="image_2.png" /><figcaption>Figure 3: IVMI-builder integration in Godot and Pure Data : Inspector properties of the IvmiScene (top left), IvmiScene and IvmiNode scripting (top right), Usage in Pure Data (bottom)</figcaption></figure>

be easily added, set and their changes retrieved in GDscript. By default, properties have their *listen* flag active, meaning that all changes are sent to Pure Data.

### 2.2.2. Immersion and 3D Interaction techniques

Godot relies on OpenXR<fnref n="5" /> for immersive rendering, which provides access to most virtual and mixed reality HMDs, but also mobile devices for augmented reality applications. In addition to vendor developed runtimes for OpenXR, the Monado <fnref n="6" /> open-source implementation gives access to open hardware and obsolete devices. In order to facilitate the exploration of novel expression possibilities, as proposed by Berthaut [19] we provide a growing set of interaction techniques directly in the IVMI-builder addon. These include navigation techniques such as flying or teleportation, selection/manipulation techniques such as virtual ray and virtual hand, and control techniques such as 3D buttons, knobs, 1D/2D/3D sliders and more advanced controls such as Tunnels [20]. Some packed techniques are also provided which integrate navigation and selection/manipulation techniques in a single scene. These techniques are built as IvmiNodes with properties such as a value in controls, a speed or closest object in navigation, or the selected or grabbed objects in manipulation/selection techniques.

### 2.2.3. Networked instruments and installations

Many XRMI and sonic installations involve multiple networked devices [21]. IVMI-builder inherits from Godot network capabilities, which rely on the enet library for reliable UDP communication.

Each instance can be defined as a server, client or peer (which turns into a server in case none has been found. For clients, the server can be given (with their IP address and port) or discovered on the local network using multicast messages. To facilitate network instruments design, the method that sets

IvmiProperties is defined as a remote procedure call, meaning that each change in a property is synchronised between all connected instances. This can be deactivated, for properties which are local to a device. For example, mixed-reality HMDs sound spatialisation requires head to sound sources relative position which does not need to be shared with other instances. IvmiProperties not connected to anything in the Pure Data patch can also be used to change the application status (e.g., different collaboration modes or installation phases).

## 3. EVALUATION

In this section, we evaluate our framework by showing how it can be used to recreate existing XRMIs and by analysing the design process of an immersive sonic installation.

### 3.1. Remaking Existing XR Instruments

To demonstrate the use of our framework, we rebuilt three XR-MIs from the literature. They illustrate the capabilities of our framework regarding design workflow, application architecture, mappings and interaction techniques. These re-implementations are integrated as examples in IVMI-builder.

<figure><img src="image_3.png" /><figcaption>Figure 4: XRMI remake : a) Drile, b) Versum, c) VrGrains</figcaption></figure>

#### 3.1.1. Drile - immersive live-looping

The first instrument, Drile, was developed by Berthaut et al. [20]. It allows musicians to build hierarchical live-looping structures using visual representations of musical trees and graphical sliders. In our remake, depicted in Figure 4.A, we used the HOMER selection and manipulation technique and we re-implemented the Tunnels graphical slider designed in Drile. The live-looping mechanism was simplified by allowing one to

<footnote n="5">⁵https://www.khronos.org/openxr/</footnote>

<footnote n="6">⁶https://monado.dev/</footnote>
record and playback the movements of both the virtual sound objects and of the tunnels. Passing the virtual sound objects through the tunnels will modify properties associated to both graphical parameters (height, colour, transparency) and sound parameters (gain, pitch, filter cutoff frequency). The virtual sounds also have a *trigger* property to play the associated sound sample, which is controlled by pressing a button on a handheld controller, and a *loudness* property to provide visual feedback on the activity of each virtual sound object.

### 3.1.2. Versum - 3D musical navigation

The second instrument, Versum, was developed by Tarik Barry [22]. It consists in flying-based navigation in a virtual environment with orbiting sound sources. In our remake, depicted in Figure 4.B, we use a *Flying* navigation technique that relies on the orientation of the handheld controller. The environment is composed of spheres which are associated with pulses and lines associated with additive synthesis pads. For all sound sources, a *distance to camera* property is updated at each frame and the value sent to Pure Data in order to control the gain of each sound.

### 3.1.3. VRgrains - immersive concatenative synthesis

The third instrument, VRgrains, was designed by Zappi et al. [23], is a virtual reality interface for concatenative synthesis. In our remake, depicted in Figure 4.C, we use the TimbreID [24] library for Pure Data in order to extract sound grains in a sound recording and create associated *IvmiNodes* in the scene automatically for each of them. Their vertical and horizontal positions are defined according to the audio features extracted for each grain. We then use a virtual hand 3D selection technique to trigger the sounds, sequenced at a fixed rate in Pure Data.

## 3.2. Design of an immersive sonic installation

In this section, we demonstrate how our framework adapts to the various steps of the design of an immersive sonic installation, *Reverberations*, which was presented at the *IEEE VR 2025 XR Gallery* [6], and created in collaboration with a musician.

### 3.2.1. Overall approach

It consists in the exploration of a virtual building superimposed on the physical space using mixed-reality HMDs. It is collaborative, with a maximum of three participants. A captured frame from one participant's point of view is shown in Figure 6.

Four virtual sound sources, containing either instrumental or voice recordings, are placed inside the building and activated / changed in sequence. The building itself evolves over time, with all walls randomly stretching one by one. In doing so, the perception of the sources is altered. They are rendered using binaural spatialisation (with distance attenuation, panning and front-back filtering) and five reverberation (with the *rev3object*) driven by the distance between the user and the source and by audio raycasting in five directions (front, back, left, right, up). Moving towards or away from the sources changes the legibility of the sounds and the words, as do changes in the geometry of the building, leading to a meditative experience over our understanding and perception of sound.

After a few minutes, a world-in-miniature appears in the centre of the building, with sliders that allow participants to stretch the building themselves, and trigger to change the sound sources (from voice to instrument to silent). The overall experience lasts 7 minutes.

### 3.2.2. Design process

The system was designed as a networked application with one server running on a laptop and three Meta Quest 3 HMDs connected on a wireless network.

<figure><img src="image_4.png" /><figcaption>Figure 5: Reverberations was designed in three successive steps, taking advantage of the flexible workflow enabled by our framework.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 6: Reverberations allows up to three participants to explore and manipulate an immersive sonic architecture. Here one participant is using a World-In-Miniature metaphor to extend one of the walls of the virtual building, altering the reverberation of virtual sound sources (one of which is the green sphere seen in the background)</figcaption></figure>

Properties include for each source their position and distance relative to the user's head, their mode (instrument, voice, silent), and for each user the distance to walls in the 5 directions, retrieved through raycasting.

The design process followed three main phases, illustrated in Figure 5, which were facilitated by our framework, as we moved from initial tests to final deployment. The three phases were:

* Architecture tests on a single laptop and the *OpenSound-Control* mode: building the overall installation software architecture with temporal sequences, ensuring that all properties are transmitted and that the sound synthesis and spatialisation works.

* Immersive tests with an HMD, laptop and bluetooth headphones in the *OpenSoundControl* mode: refining interaction and spatialisation, especially the 3D navigation and manipulation of the world-in-miniature

* Final tests and deployment with HMD only in the *Audio* mode: testing the collaboration with multiple HMDs, refining spatialisation for the audio on the HMDs.
### 3.3. Discussion and Limitations

GdPd and IVMI-builder facilitated and accelerated both the recreation of instruments from the literature and the design of the immersive installation, thanks to the flexibility of the architecture and workflow, the live-editing features, and the integrated interaction techniques, OpenXR compatibility and networking capabilities.

However, we also noticed some limitations of the current version of the framework. The first is the design of collaborative applications. In the context of mixed-reality instruments for example, the configuration and calibration of both the virtual environment and of the users could be made faster and simpler by integrating a few additional features in the networking components. The second is the absence of handling of live update of patches on the HMD which might even facilitate the design of multi devices applications. For example, fine-tuning the patches and updating each HMD in the design of *Reverberations* took a lot of time. The third is the focus on HMDs, partly due to the use of OpenXR for which it is the main goal. The framework should take into account other immersive platforms: CAVEs, stereoscopic screens, spatial augmented reality, mobile augmented reality.

### 4. CONCLUSION

In this paper we presented a libre software framework for designing Extended Reality Musical Instruments, Immersive Sonic Installations and audiovisual applications. Our goals with this framework is to ensure the longevity of immersive musical and sonic applications, in particular avoiding the obsolescence of XR technologies, following the philosophy of other tools such as the Monado open-source OpenXR runtime<fnref n="7" />.

Our future work includes: 1) adding visualisation and interaction techniques from research in XR (redirected walking, 3D drawing and sculpting), 2) extending the framework to support Spatial Augmented Reality and stereoscopic displays.

### 5. REFERENCES

[1] Luca Turchet, Rob Hamilton, and Anil Çamci, “Music in extended realities,” *IEEE Access*, vol. 9, pp. 15810–15832, 2021.

[2] Stefania Serafin, Cumhur Erkut, Juraj Kojs, Niels C Nilsson, and Rolf Nordahl, “Virtual reality musical instruments: State of the art, design principles, and future directions,” *Computer Music Journal*, vol. 40, no. 3, pp. 22–40, 2016.

[3] M Puckette, “Pure data,” in *Proceedings of International Computer Music Conference 1997*, 1997.

[4] Esther Gruy and Florent Berthaut, “MagneTip: Reintroducing a Physical Interaction Loop for 3D Musical Drawing in Extended Reality,” in *New Interfaces for Musical Expression*, Utrecht (Netherlands), Sept. 2024.

[5] Pierrick Uro, Florent Berthaut, Thomas Pietrzak, and Marcelo Wanderley, “Decoupling physical and virtual spaces for new collaboration strategies in co-located mixed reality instruments,” in *to appear in New Interfaces for Musical Expression*, 2025.

[6] Sébastien Beaumont and Florent Berthaut, “Reverberations,” in *IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops*, 2025, pp. 1694–1695.

[7] Miller Puckette, “Max at seventeen,” *Computer Music Journal*, vol. 26, no. 4, pp. 31–43, 2002.

[8] Jack Atherton and Ge Wang, “Chunity: Integrated audiovisual programming in unity,” in *Proceedings of the international conference on new interfaces for musical expression*, 2018, pp. 102–107.

[9] Robert Hamilton, “Udkosc: An immersive musical environment,” in *Proceedings of the International Computer Music Conference*, 2011.

[10] David Johnson, Daniela Damian, and George Tzanetakis, “Osc-xr: A toolkit for extended reality immersive music interfaces,” in *Proceedings of Sound and Music Computing Conference*, 2019, pp. 202–209.

[11] I Bukvic and Ji-Sun Kim, “μ max-unity3d interoperability toolkit,” in *Proceedings of the International Computer Music Conference*, 2009.

[12] Anil Çamci, Kristine Lee, Cody J Roberts, and Angus G Forbes, “Inviso: a cross-platform user interface for creating virtual sonic environments,” in *Proceedings of the ACM Symposium on User Interface Software and Technology*. ACM, 2017, pp. 507–518.

[13] Damian Dziwis, “Pdxr-the evolution of pure data into the metaverse,” in *Proceeding of the International Computer Music Conference (ICMC)*, 2023, pp. 1–8.

[14] Michel Buffa, Dorian Girard, and Ayoub Hofr, “Using web audio modules for immersive audio collaboration in the musical metaverse,” in *2024 IEEE 5th International Symposium on the Internet of Sounds (IS2)*. IEEE, 2024, pp. 1–10.

[15] Florent Berthaut, GdPd, 2020, https://gitlab.univ-lille.fr/ivmi/gdpd, https://hal.science/hal-04398764.

[16] Florent Berthaut and Théophile Cosse, “IVMI-builder,” 2024, https://gitlab.univ-lille.fr/ivmi/ivmi-builder, https://hal.science/hal-04398716.

[17] Peter Brinkmann, Peter Kirn, Richard Lawler, Chris McCormick, Martin Roth, and Hans-Christoph Steiner, “Embbedding pure data with libpd,” in *Proceedings of the Pure Data Convention*. Citeseer, 2011, vol. 291.

[18] Gary P Scavone, “Rtaudio: A cross-platform c++ class for realtime audio input/output,” in *Proceedings of the International Computer Music Conference*, 2002.

[19] Florent Berthaut, “3D interaction techniques for musical expression,” *Journal of New Music Research*, 2020.

[20] Florent Berthaut, Myriam Desainte-Catherine, and Martin Hachet, “Drile: An Immersive Environment for hierarchical live-looping,” in *New Interface for Musical Expression*, Sydney, Australia, June 2010, p. page 192.

[21] Liang Men and Nick Bryan-Kinns, “Lemo: exploring virtual space for collaborative creativity,” in *Proceedings of the 2019 Conference on Creativity and Cognition*, 2019, pp. 71–82.

[22] Tarik Barri, “Versum: Audiovisual composing in 3d,” in *Proceedings of the International Conference on New Interfaces for Musical Expression*. June 2009, pp. 264–265, Zenodo.

[23] Victor Zappi, Dario Mazzanti, Andrea Brogni, and Darwin Caldwell, “Concatenative synthesis unit navigation and dynamic rearrangement in vrgrains,” in *Proceedings of the sound & music computing conference*, 2012, pp. 403–408.

[24] William Brent, “A timbre analysis and classification toolkit for pure data,” in *Proceedings of the International Computer Music Conference*, 2010, vol. 2010, pp. 224–229.

<footnote n="7">⁷https://monado.dev/</footnote>