<figure><img src="image_1.png" /></figure>

HAL
open science

# Proceedings of the 19th Linux Audio Conference

Romain Michon, Pierre Lecomte

► To cite this version:

Romain Michon, Pierre Lecomte. Proceedings of the 19th Linux Audio Conference. Proceedings of the 19th Linux Audio Conference, Jun 2025, Lyon, France. 2025, 978-2-9597911-2-3. hal-05194352

HAL Id: hal-05194352

https://hal.science/hal-05194352v1

Submitted on 31 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

# Proceedings of the
19th Linux Audio Conference (LAC-25)

June 25th -28th, 2025 – Lyon (France)

Inria // GRAME-CNCM // INSA Lyon // AFIM
<figure><img src="image_4.png" /></figure>
<figure><img src="image_5.png" /></figure>

LAC 2025

Proceedings of the 19th Linux Audio Conference

June 25th-28th, 2025

Romain Michon and Pierre Lecomte, eds.
LAC-25 is organized by:

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

GRAME
CNCM, LYON

INSA

INSTITUT NATIONAL
DES SCIENCES
APPLIQUÉES
LYON

afim

Association Francophone d'Informatique Musicale

With the support of:

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

Université Claude Bernard

<figure><img src="image_10.png" /></figure>

Lyon 1

<figure><img src="image_11.png" /></figure>

LE BRAS
COMMUNICATIONS

<figure><img src="image_12.png" /></figure>

LYON

<figure><img src="image_13.png" /></figure>

RESOTECH
INGENIERIE / SPECTACLE / EVENEMENT

Proceedings of the 19th Linux Audio Conference

Romain Michon and Pierre Lecomte, Editors

ISBN: 978-2-9597911-2-3

EAN: 9782959791123

Website: https://jimlac25.inria.fr/

Bibtex:

@proceedings{25LACConf,
Editor = {Romain Michon and Pierre Lecomte},
Organization = {Inria, GRAME-CNCM, and INSA Lyon},
Publisher = {GRAME-CNCM},
Title = {Proceedings of the 19th Linux Audio Conference},
Year = {2025}}

These proceedings, and all the papers included in it, are an open-access publication distributed under the terms of the Creative Commons Attribution 4.0 Unported License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author(s) and source are credited. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.

<figure><img src="image_14.png" /></figure>

creative
commons

Typesetting by Romain Michon using LATEX.
# Table of Contents

<table><tr><td>LAC-25 Preface</td><td>vi</td></tr><tr><td>Organizational Committee, Authors, and Presenters</td><td>vii</td></tr><tr><td>LAC-25 Paper Session 1</td><td>1</td></tr><tr><td>A Libre Software Framework for Extended Reality Musical Instruments and Sonic Installations – Florent Berthaut</td><td>2</td></tr><tr><td>A Modular Ecosystem for Audio-Visual Augmented Reality Performances – Henrik Von Coler</td><td>7</td></tr><tr><td>algojuggledrums - Creating Rhythms from Juggling Patterns – Albert Gräf and Sophia Renz</td><td>11</td></tr><tr><td>LAC-25 Paper Session 2</td><td>16</td></tr><tr><td>AUD-SUR: An Audio Analyzer Assistant for Audio Surveillance Applications – Lam Pham, Martin Boyer, Schindler Schindler, Silvia Poletti, Marcel Hasenbalg, David Fischinger, Phat Lam, and Dat Trans</td><td>17</td></tr><tr><td>AudioReach: An End-to-End Solution – Patrick Lai</td><td>24</td></tr><tr><td>Bridging Musical Gesture and Computational Representation: Designing Gesture Composition Framework – Markidis Marco Matteo</td><td>30</td></tr><tr><td>LAC-25 Paper Session 3</td><td>38</td></tr><tr><td>BuGUI: A Widget System for Button Grid Controllers – Thibaud Keller, Victor Lazzarini, Gordon Delap, Jean-Michel Celerier</td><td>39</td></tr><tr><td>Composing Resilient Software: Architectural Evolution and Collaborative Governance in JackTrip's Open-Source Journey – Sven Thielen, Chris Chafe</td><td>46</td></tr><tr><td>CSOUND 7: Bare Metal and Co-Processing Hardwares – Victor Lazzarini, Aman Jagwani</td><td>52</td></tr><tr><td>LAC-25 Paper Session 4</td><td>58</td></tr><tr><td>Enhancing Interactive Communication with the OVBOX: New Features and Applications in Hearing Research – Giso Grimm, Volker Hohmann</td><td>59</td></tr><tr><td>LAC-25 Paper Session 5</td><td>65</td></tr><tr><td>Inside Andrea Molino's La vérité, pas toute: The Creation of Swarm Sounds – Nicola Bernardini, Alvise Vidolin, Luca Richelli, Alessandro Fiordelmondo, Mattia Pizzato</td><td>66</td></tr><tr><td>Linux Audio on Android – Victor Zappi</td><td>71</td></tr><tr><td>MFP: A graphical patcher extensible in Python and Faust – Bill Gribble</td><td>78</td></tr><tr><td>Pd-Lua Signals and Graphics – Albert Gräf, Timothy Schoen, Benjamin Wesch</td><td>85</td></tr><tr><td>LAC-25 Paper Session 6</td><td>92</td></tr><tr><td>Raptor, the Random Arpeggiator Version 7 – Albert Gräf</td><td>93</td></tr><tr><td>Real-Time Musical Instruments Recognition for Scenography Purposes – Hugues Kadi, Mathis Devidal, Romain Michon, Johan Erbani</td><td>98</td></tr><tr><td>LAC-25 Paper Session 7</td><td>106</td></tr><tr><td>Spatial Room Impulse Response Extrapolation Using the Image Source Method – Zhenxian Li, Thomas McKenzie</td><td>107</td></tr><tr><td>Tape Simulation – Fons Adriaensen</td><td>113</td></tr><tr><td>LAC-25 Paper Session 8</td><td>118</td></tr><tr><td>WebPdL2Ork: Bringing the Linux Laptop Orchestra to the Browser – William Furgerson, Ivica Bukvic, Justin Kerobo, Bradley Davis</td><td>119</td></tr><tr><td>Zita-at2 Autotune – Fons Adriaensen</td><td>125</td></tr><tr><td>Presentation of Workshops</td><td>129</td></tr><tr><td>Presentation of Demos</td><td>130</td></tr></table>
# LAC-25 Preface

By Romain Michon, Pierre Lecomte, and Stéphane Letz.

After four years of absence in the “post-covid world,” Linux Audio Conference (LAC) came back on June 25-28, 2025 in Lyon (France) for its nineteenth edition! It joined forces with the *Journées d'Informatique Musicale* (French Computer Music conference) which preceded it on June 23-25, 2025. Those who attended these two events were able to enjoy no less than six days of conference in a row, as well as six concerts (including four for LAC-25), and various social events. The scientific portion of LAC-25 was hosted at the *Marie Curie library of INSA Lyon* (engineering school). Concerts took place outside on the *INSA Lyon* campus, at the Astrée Theater of *Lyon 1 University*, and at *Le Sucre* for the concluding live coding evening. In total, twenty four oral presentation, four demos, three workshops, and one keynote were given, and twenty three music pieces were performed.

Some said that the Covid period marked the beginning of a new era where conferences and other events of this kind would increasingly happen online. With more than eighty participants in total and a rather limited online attendance (five people in average), LAC-25 can be considered as a “big edition,” which rejoices us. From a purely scientific and technical standpoint, online conferences and scientific publishing probably do as well as in-person events. But it lacks the “human aspect” – which is essential for the artistic portion of LAC – and our ability to make new friends (and see old ones), start new collaborations, and discover new things. Right from the beginning, LAC has always been a place where you see people randomly seating together in front of a computer in the corner of a room to check out the latest breaking developments in an open source software, to code new features, implement new ideas that emerged from an intense discussion at a coffee break or over a beer the night before, etc.

In a world where Artificial Intelligence (AI) is increasing the pace at which intellectual property is being produced by large companies, open source software has never made more sense. It's nice to see than LAC remains such a vibrant community with familiar faces and many new ones.

Now that LAC is reborn, let's hope they'll be many more editions of this great and unique event. Long live Linux Audio Conference!
# Organizational Committee, Authors, and Presenters

## Organizational Committee

Romain Michon (Inria, France) – General Chair

Pierre Lecomte (École Centrale de Lyon, France) – Paper Chair

John Granzow (University of Michigan, USA) – Music Chair

Stéphane Letz (GRAME - Centre National de Création Musicale, France) – Technical Chair

## Technical Staff

Clément Auvergne

Jean-Cyrille Burdet

Irène Hontang

Jeanne Ribeau

## Volunteers

Théo Cantaloube

Axel Chypre

Pierre Cochard

Loukas Duque

Rémi Guillotte

Simon Jacquin

Louis Ledoux

Matteo Olivo

Benjamin Quiédeville

Thomas Rushton

## Authors and Presenters

Fons Adriaensen

Florent Berthaut

Martin Boyer

Michel Buffa

Ivica Bukvic

Jean-Michaël Celerier

Chris Chafe

Alexander Chalikiopoulos

Bradley Davis

Matthew Davison

Gordon Delap

Mathis Devidal

Johan Erbani

Alessandro Fiordelmondo

David Fischinger

William Furgerson

Robin Gareus

Will Godfrey

Albert Gräf

Bill Gribble

Giso Grimm

Marcel Hasenbalg

Volker Hohmann

Aman Jagwani

Hugues Kadi

Thibaud Keller

Justin Kerobo

Patrick Lai

Phat Lam

Victor Lazzarini

Zhenxian Li

Fernando Lopez-Lezcano

Marco Matteo Markidis

Thomas Mayer

Thomas McKenzie

Romain Michon

Lam Pham

Mattia Pizzato
Silvia Poletti

Maxime Popoff

Sophia Renz

Luca Richelli

Schindler Schindler

Timothy Schoen

Carla Tapparo

Sven Thielen

Nils Tonnätt

Dat Trans

Alvise Vidolin

Henrik Von Coler

Benjamin Wesch

Julian Wolff

Teodor Wozniak

Victor Zappi

## Performers and Artists

AFALFL

Bubobubobubo

Char Stiles

Alessandro De Cecco

Stefano Catena

Kerem Ergener

Flopine

Fronssons

Massimo Fragala

g_code

Zoe (yi-Cheng) Lin

Fernando Lopez-Lezcano

Marco Matteo Markidis

Ralt144MI

Pérégrine

Niklas Reppel

Diemo Schwarz

Nicolas Souchal

Henrik Von Coler

## Reviewers

Fons Adriaensen

Constantin Basica

Nicola Bernardini

Florent Berthaut

Ivica Bukvic

Alex Chechile

Jatin Chowdhury

Goetz Dipper

Robin Gareus

Stefano Giacomelli

Albert Gräf

Irène Hontang

Molly Jones

Christoph Kuhr

Nolan Lem

Stéphane Letz

Fernando Lopez-Lezcano

Douglas Mccausland

Eric Maestri

Romain Michon

Caroline Miller

Chandra Bose Naripeddy Subhash

Won Park Joo

Laurent Pottier

Tanguy Risset

Dirk Roosenburg
Mayank Sanganeria

Scott Smallwood

Florian Paul Schmidt

Giuseppe Silvi

Julian Villegas

Henrik Von Coler

Jonathan Woithe

Julian Wolff

Stefano Zambon

Julie Zhu
<figure><img src="image_15.png" /></figure>
LAC-25 Paper Session 1
# GDPD / IVMI-BUILDER: A LIBRE SOFTWARE FRAMEWORK FOR EXTENDED REALITY MUSICAL INSTRUMENTS AND IMMERSIVE SONIC INSTALLATIONS

Florent Berthaut*

CRISTAL, CNRS, Univ. Lille, France

florent.berthaut@univ-lille.fr

## ABSTRACT

In this paper, we present *GdPd* and *IVMI-builder*, two components of a libre software framework for designing audiovisual applications, immersive sonic installations and Extended Reality Musical Instruments. It is composed of addons for the Godot game engine and abstractions for the Pure Data programming language. It enables fast prototyping and deployment on multiple devices, including Mixed and Virtual Reality Head-Mounted Displays. We first present the architecture components and workflow, including live editing, networked applications and deployment. We then evaluate it through the example of re-implementing a set of existing instruments and examining the design of an immersive musical installation / instrument.

## 1. INTRODUCTION

With the generalisation of somewhat affordable Mixed and Virtual Reality (now generally combined as Extended Reality / XR) equipment, e.g. mobile devices, Head-Mounted Displays (HMD), 3D tracking system, an increasing number of Extended Reality Musical Instruments (XRMI) and Immersive Sonic Installations are being developed [1, 2]. However, prototyping immersive musical instruments and sonic installations remains complex, as it involves designing and testing the visual elements of the 3D scene, their behaviour and the interaction techniques, the sound processes and the mappings between graphics and sound. In order to facilitate the exploration of novel expressive techniques, all these aspects should be easily and continuously editable during the design phase. In addition, these applications can benefit from free and open-source software frameworks for design, which can be enriched with new interaction and visualisation techniques by the community, as research and technology progress.

In this paper, we introduce *GdPd* and *IVMI-builder*, two components of a libre software framework for designing audiovisual applications, XRMIs and Immersive Sonic Installations, based on the Godot game engine and the Pure Data [3] visual programming language. It facilitates live-editing of both the visual and sound aspects of the instruments while adapting to different workflows. In particular, it allows one to switch between separated and integrated sound synthesis without changes in the code, enabling various design processes and application architectures. It also integrates a set of 3D interaction techniques which can be extended, and networking capabilities. It has already been used in a number of projects that have led to scientific publications [4, 5] and artistic productions, one of which [6] is described in Section 3.2. We present the framework in Section 2 and evaluate it in Section 3 by reimplementing existing instruments and analysing the design process of an immersive sonic installation.

### 1.1. Related work

Many frameworks have been created for the development of audiovisual applications. These include text and visual programming languages such as Pure Data, Open Frameworks, Max/MSP

[7], Processing, VVVV. While they may provide both audio and 3D rendering capabilities, they often lack dedicated XR and graphical development features that are present in game engines such as Unreal, Unity or Godot, which design of 3D interactions.

In order to take advantage of the design opportunities offered by game engines, commercial spatial audio plugins such as Wwise have also been developed, along with plugins that allow one to closely integrate audio programming languages, such as Chunity [8], or uPD<fnref n="1" />, or to connect external audio programming software using OpenSoundControl messages, such as with UdkOSC [9], OSC-XR [10] or Myu [11].

Dedicated tools that take into account the variety of 3D display devices also exist, one particular example being Inviso [12], which does not rely on a game engine, but rather on a web programming framework. Finally, some tools directly integrate patching capabilities within the virtual environments through libpd such as PdXR [13] or through Web Audio Modules [14].

However, all these examples are at least partially non-free (e.g., based on Unity or Unreal) and/or limit access either to advanced audio or 3D programming features. In contrast, our framework rely on the Godot game engine (MIT licence) and Pure Data (BSD licence), and provides both advanced 3D editing, scripting and XR support but also a flexible workflow for audiovisual programming.

## 2. GDPD AND IVMI-BUILDER

Our framework relies on the Godot<fnref n="2" /> game engine for gestural input and visual rendering, and on the Pure Data<fnref n="3" /> visual programming language for sound synthesis.

*GdPd* [15] provides classes for Godot and abstractions for Pure Data which are symmetrical and enable exchange of data between the two software, e.g., control data from Godot which is then mapped to sound parameters in Pure Data, and feedback data from audio analysis in Pure Data mapped to visual feedback in Godot. As shown in Figure 1, the exchange is done either through OpenSoundControl messages, in which case the audio is processed and output by PureData, or through internal Pure Data messages, in which case the audio is processed and output inside the Godot application binary. No change in the code or patch is needed to switch between these modes, enabling various design workflows and application architecture, as demonstrated in the next sections. *GdPd* can be used to develop interactive application that combines 2D or 3D graphics and sound synthesis: video games, musical composition or expression software, sonic art installations and so on. *GdPd* is available at https://gitlab.univ-lille.fr/ivmi/gdpd.

*IVMI-builder* [16] builds upon *GdPd* to offer a framework for the design of XRMIs and immersive sonic installations, which, in addition to sound synthesis through Pure Data patches, facilitates the configuration of Extended Reality displays and of networked instruments, and provides access to 3D interaction and visualisation techniques. *IVMI-builder* is available at https://gitlab.univ-lille.fr/ivmi/ivmi-builder.

<footnote>* This work was supported by the IRCICA, the Région Hauts de France, the Fédération de Recherche SCV</footnote>

<footnote n="1">¹https://github.com/Magicolo/uPD</footnote>

<footnote n="2">²https://godotengine.org/</footnote>

<footnote n="3">³https://puredata.info/</footnote>
<figure><img src="image_16.png" /><figcaption>Figure 3: IVMI-builder integration in Godot and Pure Data : Inspector properties of the IvmiScene (top left), IvmiScene and IvmiNode scripting (top right), Usage in Pure Data (bottom)</figcaption></figure>

be easily added, set and their changes retrieved in GDscript. By default, properties have their *listen* flag active, meaning that all changes are sent to Pure Data.

### 2.2.2. Immersion and 3D Interaction techniques

Godot relies on OpenXR<fnref n="5" /> for immersive rendering, which provides access to most virtual and mixed reality HMDs, but also mobile devices for augmented reality applications. In addition to vendor developed runtimes for OpenXR, the Monado <fnref n="6" /> open-source implementation gives access to open hardware and obsolete devices. In order to facilitate the exploration of novel expression possibilities, as proposed by Berthaut [19] we provide a growing set of interaction techniques directly in the IVMI-builder addon. These include navigation techniques such as flying or teleportation, selection/manipulation techniques such as virtual ray and virtual hand, and control techniques such as 3D buttons, knobs, 1D/2D/3D sliders and more advanced controls such as Tunnels [20]. Some packed techniques are also provided which integrate navigation and selection/manipulation techniques in a single scene. These techniques are built as IvmiNodes with properties such as a value in controls, a speed or closest object in navigation, or the selected or grabbed objects in manipulation/selection techniques.

### 2.2.3. Networked instruments and installations

Many XRMI and sonic installations involve multiple networked devices [21]. IVMI-builder inherits from Godot network capabilities, which rely on the enet library for reliable UDP communication.

Each instance can be defined as a server, client or peer (which turns into a server in case none has been found. For clients, the server can be given (with their IP address and port) or discovered on the local network using multicast messages. To facilitate network instruments design, the method that sets

IvmiProperties is defined as a remote procedure call, meaning that each change in a property is synchronised between all connected instances. This can be deactivated, for properties which are local to a device. For example, mixed-reality HMDs sound spatialisation requires head to sound sources relative position which does not need to be shared with other instances. IvmiProperties not connected to anything in the Pure Data patch can also be used to change the application status (e.g., different collaboration modes or installation phases).

## 3. EVALUATION

In this section, we evaluate our framework by showing how it can be used to recreate existing XRMIs and by analysing the design process of an immersive sonic installation.

### 3.1. Remaking Existing XR Instruments

To demonstrate the use of our framework, we rebuilt three XR-MIs from the literature. They illustrate the capabilities of our framework regarding design workflow, application architecture, mappings and interaction techniques. These re-implementations are integrated as examples in IVMI-builder.

<figure><img src="image_17.png" /><figcaption>Figure 4: XRMI remake : a) Drile, b) Versum, c) VrGrains</figcaption></figure>

#### 3.1.1. Drile - immersive live-looping

The first instrument, Drile, was developed by Berthaut et al. [20]. It allows musicians to build hierarchical live-looping structures using visual representations of musical trees and graphical sliders. In our remake, depicted in Figure 4.A, we used the HOMER selection and manipulation technique and we re-implemented the Tunnels graphical slider designed in Drile. The live-looping mechanism was simplified by allowing one to

<footnote n="5">⁵https://www.khronos.org/openxr/</footnote>

<footnote n="6">⁶https://monado.dev/</footnote>
record and playback the movements of both the virtual sound objects and of the tunnels. Passing the virtual sound objects through the tunnels will modify properties associated to both graphical parameters (height, colour, transparency) and sound parameters (gain, pitch, filter cutoff frequency). The virtual sounds also have a *trigger* property to play the associated sound sample, which is controlled by pressing a button on a handheld controller, and a *loudness* property to provide visual feedback on the activity of each virtual sound object.

### 3.1.2. Versum - 3D musical navigation

The second instrument, Versum, was developed by Tarik Barry [22]. It consists in flying-based navigation in a virtual environment with orbiting sound sources. In our remake, depicted in Figure 4.B, we use a *Flying* navigation technique that relies on the orientation of the handheld controller. The environment is composed of spheres which are associated with pulses and lines associated with additive synthesis pads. For all sound sources, a *distance to camera* property is updated at each frame and the value sent to Pure Data in order to control the gain of each sound.

### 3.1.3. VRgrains - immersive concatenative synthesis

The third instrument, VRgrains, was designed by Zappi et al. [23], is a virtual reality interface for concatenative synthesis. In our remake, depicted in Figure 4.C, we use the TimbreID [24] library for Pure Data in order to extract sound grains in a sound recording and create associated *IvmiNodes* in the scene automatically for each of them. Their vertical and horizontal positions are defined according to the audio features extracted for each grain. We then use a virtual hand 3D selection technique to trigger the sounds, sequenced at a fixed rate in Pure Data.

## 3.2. Design of an immersive sonic installation

In this section, we demonstrate how our framework adapts to the various steps of the design of an immersive sonic installation, *Reverberations*, which was presented at the *IEEE VR 2025 XR Gallery* [6], and created in collaboration with a musician.

### 3.2.1. Overall approach

It consists in the exploration of a virtual building superimposed on the physical space using mixed-reality HMDs. It is collaborative, with a maximum of three participants. A captured frame from one participant's point of view is shown in Figure 6.

Four virtual sound sources, containing either instrumental or voice recordings, are placed inside the building and activated / changed in sequence. The building itself evolves over time, with all walls randomly stretching one by one. In doing so, the perception of the sources is altered. They are rendered using binaural spatialisation (with distance attenuation, panning and front-back filtering) and five reverberation (with the *rev3object*) driven by the distance between the user and the source and by audio raycasting in five directions (front, back, left, right, up). Moving towards or away from the sources changes the legibility of the sounds and the words, as do changes in the geometry of the building, leading to a meditative experience over our understanding and perception of sound.

After a few minutes, a world-in-miniature appears in the centre of the building, with sliders that allow participants to stretch the building themselves, and trigger to change the sound sources (from voice to instrument to silent). The overall experience lasts 7 minutes.

### 3.2.2. Design process

The system was designed as a networked application with one server running on a laptop and three Meta Quest 3 HMDs connected on a wireless network.

<figure><img src="image_18.png" /><figcaption>Figure 5: Reverberations was designed in three successive steps, taking advantage of the flexible workflow enabled by our framework.</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>Figure 6: Reverberations allows up to three participants to explore and manipulate an immersive sonic architecture. Here one participant is using a World-In-Miniature metaphor to extend one of the walls of the virtual building, altering the reverberation of virtual sound sources (one of which is the green sphere seen in the background)</figcaption></figure>

Properties include for each source their position and distance relative to the user's head, their mode (instrument, voice, silent), and for each user the distance to walls in the 5 directions, retrieved through raycasting.

The design process followed three main phases, illustrated in Figure 5, which were facilitated by our framework, as we moved from initial tests to final deployment. The three phases were:

* Architecture tests on a single laptop and the *OpenSound-Control* mode: building the overall installation software architecture with temporal sequences, ensuring that all properties are transmitted and that the sound synthesis and spatialisation works.

* Immersive tests with an HMD, laptop and bluetooth headphones in the *OpenSoundControl* mode: refining interaction and spatialisation, especially the 3D navigation and manipulation of the world-in-miniature

* Final tests and deployment with HMD only in the *Audio* mode: testing the collaboration with multiple HMDs, refining spatialisation for the audio on the HMDs.
## 3.3. Discussion and Limitations

GdPd and IVMI-builder facilitated and accelerated both the recreation of instruments from the literature and the design of the immersive installation, thanks to the flexibility of the architecture and workflow, the live-editing features, and the integrated interaction techniques, OpenXR compatibility and networking capabilities.

However, we also noticed some limitations of the current version of the framework. The first is the design of collaborative applications. In the context of mixed-reality instruments for example, the configuration and calibration of both the virtual environment and of the users could be made faster and simpler by integrating a few additional features in the networking components. The second is the absence of handling of live update of patches on the HMD which might even facilitate the design of multi devices applications. For example, fine-tuning the patches and updating each HMD in the design of *Reverberations* took a lot of time. The third is the focus on HMDs, partly due to the use of OpenXR for which it is the main goal. The framework should take into account other immersive platforms: CAVEs, stereoscopic screens, spatial augmented reality, mobile augmented reality.

## 4. CONCLUSION

In this paper we presented a libre software framework for designing Extended Reality Musical Instruments, Immersive Sonic Installations and audiovisual applications. Our goals with this framework is to ensure the longevity of immersive musical and sonic applications, in particular avoiding the obsolescence of XR technologies, following the philosophy of other tools such as the Monado open-source OpenXR runtime<fnref n="7" />.

Our future work includes: 1) adding visualisation and interaction techniques from research in XR (redirected walking, 3D drawing and sculpting), 2) extending the framework to support Spatial Augmented Reality and stereoscopic displays.

## 5. REFERENCES

[1] Luca Turchet, Rob Hamilton, and Anil Çamci, “Music in extended realities,” *IEEE Access*, vol. 9, pp. 15810–15832, 2021.

[2] Stefania Serafin, Cumhur Erkut, Juraj Kojs, Niels C Nilsson, and Rolf Nordahl, “Virtual reality musical instruments: State of the art, design principles, and future directions,” *Computer Music Journal*, vol. 40, no. 3, pp. 22–40, 2016.

[3] M Puckette, “Pure data,” in *Proceedings of International Computer Music Conference 1997*, 1997.

[4] Esther Gruy and Florent Berthaut, “MagneTip: Reintroducing a Physical Interaction Loop for 3D Musical Drawing in Extended Reality,” in *New Interfaces for Musical Expression*, Utrecht (Netherlands), Sept. 2024.

[5] Pierrick Uro, Florent Berthaut, Thomas Pietrzak, and Marcelo Wanderley, “Decoupling physical and virtual spaces for new collaboration strategies in co-located mixed reality instruments,” in *to appear in New Interfaces for Musical Expression*, 2025.

[6] Sébastien Beaumont and Florent Berthaut, “Reverberations,” in *IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops*, 2025, pp. 1694–1695.

[7] Miller Puckette, “Max at seventeen,” *Computer Music Journal*, vol. 26, no. 4, pp. 31–43, 2002.

[8] Jack Atherton and Ge Wang, “Chunity: Integrated audiovisual programming in unity,” in *Proceedings of the international conference on new interfaces for musical expression*, 2018, pp. 102–107.

[9] Robert Hamilton, “Udkosc: An immersive musical environment,” in *Proceedings of the International Computer Music Conference*, 2011.

[10] David Johnson, Daniela Damian, and George Tzanetakis, “Osc-xr: A toolkit for extended reality immersive music interfaces,” in *Proceedings of Sound and Music Computing Conference*, 2019, pp. 202–209.

[11] I Bukvic and Ji-Sun Kim, “μ max-unity3d interoperability toolkit,” in *Proceedings of the International Computer Music Conference*, 2009.

[12] Anil Çamci, Kristine Lee, Cody J Roberts, and Angus G Forbes, “Inviso: a cross-platform user interface for creating virtual sonic environments,” in *Proceedings of the ACM Symposium on User Interface Software and Technology*. ACM, 2017, pp. 507–518.

[13] Damian Dziwis, “Pdxr-the evolution of pure data into the metaverse,” in *Proceeding of the International Computer Music Conference (ICMC)*, 2023, pp. 1–8.

[14] Michel Buffa, Dorian Girard, and Ayoub Hofr, “Using web audio modules for immersive audio collaboration in the musical metaverse,” in *2024 IEEE 5th International Symposium on the Internet of Sounds (IS2)*. IEEE, 2024, pp. 1–10.

[15] Florent Berthaut, GdPd, 2020, https://gitlab.univ-lille.fr/ivmi/gdpd, https://hal.science/hal-04398764.

[16] Florent Berthaut and Théophile Cosse, “IVMI-builder,” 2024, https://gitlab.univ-lille.fr/ivmi/ivmi-builder, https://hal.science/hal-04398716.

[17] Peter Brinkmann, Peter Kirn, Richard Lawler, Chris McCormick, Martin Roth, and Hans-Christoph Steiner, “Embbedding pure data with libpd,” in *Proceedings of the Pure Data Convention*. Citeseer, 2011, vol. 291.

[18] Gary P Scavone, “Rtaudio: A cross-platform c++ class for realtime audio input/output,” in *Proceedings of the International Computer Music Conference*, 2002.

[19] Florent Berthaut, “3D interaction techniques for musical expression,” *Journal of New Music Research*, 2020.

[20] Florent Berthaut, Myriam Desainte-Catherine, and Martin Hachet, “Drile: An Immersive Environment for hierarchical live-looping,” in *New Interface for Musical Expression*, Sydney, Australia, June 2010, p. page 192.

[21] Liang Men and Nick Bryan-Kinns, “Lemo: exploring virtual space for collaborative creativity,” in *Proceedings of the 2019 Conference on Creativity and Cognition*, 2019, pp. 71–82.

[22] Tarik Barri, “Versum: Audiovisual composing in 3d,” in *Proceedings of the International Conference on New Interfaces for Musical Expression*. June 2009, pp. 264–265, Zenodo.

[23] Victor Zappi, Dario Mazzanti, Andrea Brogni, and Darwin Caldwell, “Concatenative synthesis unit navigation and dynamic rearrangement in vrgrains,” in *Proceedings of the sound & music computing conference*, 2012, pp. 403–408.

[24] William Brent, “A timbre analysis and classification toolkit for pure data,” in *Proceedings of the International Computer Music Conference*, 2010, vol. 2010, pp. 224–229.

<footnote n="7">⁷https://monado.dev/</footnote>
# A MODULAR ECOSYSTEM FOR
## AUDIO-VISUAL AUGMENTED REALITY PERFORMANCES

Henrik von Coler

School of Music
Georgia Institute of Technology, USA
hvc@gatech.edu

## ABSTRACT

The modular approach of Linux audio and multimedia systems allows the combination of different software components into an ecosystem with maximum flexibility. This paper describes such a Linux-based system for audio-visual performances with Augmented Reality (AR) interfaces. OSC messages and video feeds are sent from one or multiple Meta Quest 3 devices to a Linux machine. This machine implements a spatialization system in SuperCollider, which creates Ambisonics or binaural output with dedicated spatial sound effects. In addition, a video stream of the devices' first person view is streamed for the audience, using the Android Debugging Bridge and Pure Data. With this combination, the system allows the manipulation of the video stream through the audio input. The proposed approach can be run on a simple machine at reasonable latencies while being robust enough for large-scale live performances.

## 1. INTRODUCTION

Extended Reality (XR) can be incorporated in live music performances and music production in many ways. Mobile Augmented Reality (AR) allows participatory performances [1] - the audience can interact with the events on stage. Virtual Reality (VR) can be used to realize immersive telematic concerts [2]. Head-mounted displays (HMD) can create virtual and augmented experiences for both the audience and the performers [3]. This technology makes it possible to use AR interfaces next to physical instruments to create a hybrid performance situation [4].

The flexibility of Linux-based multimedia systems allows the combination of various free and open source components. Often times, advanced tasks - like multichannel sound spatialization - can be run on commodity hardware, when properly configured [5]. Although experimental and professional tools and environments - like Pure Data or Max/MSP - can be used to create self-contained audio-visual experiences, additional components are necessary to include XR hardware, like head-mounted displays.

The system presented in this paper is an audio-visual AR performance environment. Performers or musicians wear a HMD - more precisely a Meta Quest 3 - on stage to explore new possibilities in Human Computer Interaction (HCI) during a musical performance. Two principles are central to the proposed approach:

I **Control**: A core function of the system is to offer means for expressive control of computer music systems through AR interfaces. These interfaces can be placed next to physical instruments, thus creating a hybrid setup for new possibilities.

II **Transparency**: Another central aspect of the proposed system is the ability to stream the first person view of the musicians to the audience. Especially in electronic music performances, this can enhance the connection between performers and audience.

The system described in this paper has initially been deployed in human-computer interaction (HCI) research [4] to evaluate

the general usability of AR interfaces next to physical instruments. In its recent version, the system is used for a live electroacoustic performance with double bass and modular synth [6]. Both musicians are equipped with a HMD to control digital audio effects and stream their first person view.

Aspects of user experience and the creative possibilities have been described in the above cited publications in detail. This paper aims at giving a detailed introduction into the workflow and tools used in this modular AR performance system, highlighting best practices, pitfalls and strengths. It should allow readers to replicate, change and expand on this approach.

## 2. TECHNOLOGY

The components and basic signal flow of the proposed system are visualized in Figure 1. Audio and video processes are shown in separate blocks. They can either run on the same physical machine or on two individual computers. One or multiple Meta Quest 3 send OSC data to SuperCollider [7], where it is used for audio effects and spatialization and forwarded Pure Data [8] for visual processing. The Ambisonics channels from SuperCollider are sent to an external decoder via JACK [9].

Crucial steps and aspects are explained in the remainder of this section. Section 2.1 contains details on the video part of the system, while the audio part is covered in Section 2.2. All scripts and configuration files for the 2CUBES performance, using two AR headsets, can be found in the dedicated repository<fnref n="1" />.

### 2.1. Video

A key aspect of the proposed system is the ability to make the first person view of one or multiple performers visible to the audience, with the additional option to manipulate the video in real-time. The official options to show the camera feed from the Meta Quest 3 headset include<fnref n="2" />:

* Cast to the Meta Horizon app. This only works for Android and IOS receivers and is not suitable.

* Cast to a browser via WiFi (oculus.com/casting). This method relies on an internet access to establish a connection between the glasses and the receiver. For a real-time performance system used at different venues, this is not acceptable.

Since none of the official methods meets the requirements of this project, multiple software components are combined to stream video from the Meta Quest to a projector via a Linux machine. The key dependencies involved are *Android Debug Bridge (adb)*, *scrcpy*, *Video for Linux 2 (V4L2)*, and *Gem in Pure Data (Pd)*. Each tool plays a crucial role in capturing, processing, and outputting the video stream:

1. **adb**: Establishes communication between the Meta Quest and the Linux PC, enabling screen capture via scrcpy.

<footnote n="1">¹https://github.com/L42i/2CUBES</footnote>

<footnote n="2">²https://www.meta.com/help/quest/192719842695017/</footnote>
<figure><img src="image_20.png" /><figcaption>Figure 1: Components of the audio-visual system with signal flow.</figcaption></figure>

2. `scrcpy`: Uses adb to mirror the Meta Quest display onto the PC, providing a real-time video feed.

3. **V4L2 and v4l2loopback**: Creates a virtual video device, allowing the `scrcpy` stream to be accessed as a camera input.

4. **Gem (Pd)**: Processes the video stream for artistic manipulation and projection.

### 2.1.1. The Android Debug Bridge (adb)

The Android Debug Bridge (adb) [10] is a versatile command-line tool, designed to allow communication between a development machine and an Android device or emulator. As part of the Android Software Development Kit (SDK), adb is essential for deploying applications, debugging, and accessing the Android system at a low level. It is used to optimize performance by adjusting system parameters and automating testing procedures through scripting.

The Android Debug Bridge operates through a client-server architecture, consisting of three components: the `adb` client, which runs on the development machine; the `adb` daemon, which runs on the target Android device; and the `adb` server, which manages communication between the two. `adb` provides critical functionalities such as shell access for executing commands on the device, file transfer capabilities, and port forwarding for debugging networked applications.

As part of the proposed system, `adb` establishes the connection between the HMD and the Linux machine. Each Android device is associated with an Android ID, a unique 14-digit alphanumeric string. Throughout this document, the placeholder `0123456789ABCD` will be used as a placeholder for an arbitrary ID. Android devices, respectively Meta Quests, connected via USB can be listed with:

```
$ adb devices
```

For a newly connected Android device, this command gives the following output:

```
List of devices attached
0123456789ABCD no permissions (missing udev
rules? user is in the plugdev group); see [
http://developer.android.com/tools/device.
html]
```

In this case, the Android device needs to be authorized, which is done on the device itself. Right after connecting to the computer via USB, the Quest will show a prompt in a popup window:

USB Detected
Click on this notification to allow
connected device to access files.

After clicking on this prompt, the user gets the choice to **Deny, Allow or Always allow from this computer**. Even when selecting the latter option, a single click confirmation is necessary every time the device is reconnected to the computer via USB. The output of `adb devices` for an authorized device will give:

```
List of devices attached
0123456789ABCD device
```

### 2.1.2. The v4l2loopback Module

**Video for Linux 2 (V4L2)** [11] is the Linux kernel's multimedia framework for handling video capture and output devices. It provides a standardized API for accessing video hardware, supporting webcam(s), TV tuners, and other video input sources. V4L2 offers extensive control over video formats, frame rates, and device parameters, making it a powerful tool for real-time video processing. In this project, V4L2 is essential for capturing and processing live video feeds from the Meta Quest. It allows seamless integration of the camera streams into arbitrary applications, ensuring low-latency performance critical for live music environments. Additionally, V4L2 works in conjunction with modules like `v4l2loopback` to facilitate virtual video devices for software-based video manipulation.

The **v4l2loopback** [12] kernel module provides a virtual video device that allows users to create loop-back video streams in Linux. It is commonly used to route video output from software applications to virtual camera devices, making it accessible to other applications that require camera input. For the proposed video solution, `v4l2loopback` enables seamless integration of the AR visuals by allowing software-generated imagery to be treated as a camera feed. This is particularly useful for real-time video processing, streaming, and interactive visual performances. The module supports multiple virtual devices, configurable frame formats, and resolution settings, making it highly adaptable for different performance environments.
and is this not easy to use, especially when decoders are needed for changing performance venues. While the ATK allows the design of Ambisonics decoders with fewer steps, completely inside SuperCollider, the use of external decoder software offers even more flexibility. The standalone version of AllRADecoder from the IEM plugin suite [18] has a flexible and intuitive way of designing decoders with a graphic interface. As a JACK client it can be fed with the encoded Ambisonics stream from SuperCollider.

### 2.2.3. Audio Connection Management

JACK connections are managed with `aj-snapshot` [19]. In this workflow, connections have to be set manually for a new project, respectively a new configuration, once. They are then stored in an XML file, when taking a snapshot. Once connections are stored, `aj-snapshot` can be launched in daemon mode to auto-connect what has been stored. The auto-connect option in `jackd` has to be disabled for this behavior (using the `-a` flag).

## 3. SYSTEM PERFORMANCE AND HANDLING

First performances with the described system were realized on two machines: one for the audio and one for the video part. In later stages, a single machine was used for both domains: A GEEKOM MiniPC with a 12th Gen Intel(R) Core(TM) i7-12650H.

Audio relies on plain JACK, after deactivating PipeWire. This was found to be more reliable in terms of connection handling. A JACK server is booted with a real-time priority of 95 and a buffer size of 256 samples (n=2) at 48 kHz. The resulting latency allows for smooth real-time interaction without significant xruns. Different audio interfaces have been used, each performing well with these server settings.

The gain in flexibility with the modular software system comes at the price of an increase in management effort. Several processes have to be launched in the right sequence and might need to be changed during runtime. Two `tmux` [20] sessions are used to cope with this - one for the audio part and one for the video part. All components, including `jackd` are launched in individual tiles. Each program can be relaunched smoothly for development and debugging, with `aj-snapshot` taking care of connections. This approach has proven to be highly reliable and deterministic.

## 4. CONCLUSIONS

Overall, the proposed audio-visual system can be considered a professional tool for various applications in research and creative practice. It is important to note that this is just one possible solution. Any component may be replaced to improve certain aspects. The audio part of this system has been applied and tested repeatedly. It is reliable, efficient and delivers high quality results. Although PipeWire has been excluded at this stage of the project, in might well be a solution for a complete audio-visual routing system.

A major drawback of the proposed video solution is the USB connection for streaming, especially since the AR interface only uses WiFi to send OSC. With high quality cables of 20 feet length, this issue is negligible for many instruments. In addition, it extends the battery life. Without battery add on, the HMDs can be drained quickly when rehearsing, followed by a performance. However, for free movement and the aesthetics of the performance, the cable should be replaced with a wireless option to stream the video.

While there are alternative ways to design a similar system, also using only a single piece of software, the modular approach is able to combine different tools, each with their individual strength. The use of command line tools like `tmux` might scare

unexperienced users off in the first place. However, once mastered, this paradigm offers advanced opportunities for control and maintenance.

## 5. REFERENCES

[1] Anna N. Nagele, Valentin Bauer, Patrick G. T. Healey, Joshua D. Reiss, Henry Cooke, Tim Cowlishaw, Chris Baume, and Chris Pike, "Interactive audio augmented reality in participatory performance," *Frontiers in Virtual Reality*, vol. 1, 2021.

[2] Damian Dziwis and Henrik von Coler, "The entanglement: Volumetric music performances in a virtual metaverse environment," *Journal of Network Music and Arts*, vol. 5, no. 1, pp. 3, 2023.

[3] Luca Turchet, Rob Hamilton, and Anil Çamci, "Music in extended realities," *IEEE Access*, vol. 9, pp. 15810–15832, 2021.

[4] Hyunkyung Shin and Henrik von Coler, "Arcube: Hybrid spatial interaction for immersive audio," in *Proceedings of the 2024 ACM Symposium on Spatial User Interaction*. 2024, Association for Computing Machinery.

[5] Fernando Lopez-Lezcano, "Towards open 3d sound diffusion systems," in *Proceedings of the International Computer Music Conference (ICMC)*, 2014.

[6] Hyunkyung Shin and Henrik von Coler, "2CUBES: A Multi-User Augmented Reality Performance," in *Proceedings of the International Computer Music Conference (ICMC)*, 2025.

[7] James McCartney, "SuperCollider: a new real time synthesis language," in *Proceedings of the International Computer Music Conference (ICMC)*, 1996.

[8] Miller S. Puckette, "Pure Data," in *Proceedings of the International Computer Music Conference (ICMC)*, 1996.

[9] Davies, Paul, "JACK API," http://www.jackaudio.org/, 2020, Accessed: 2025-03-31.

[10] Android Developers, "Android debug bridge (adb)," https://developer.android.com/tools/adb, 2025, Accessed:2025-03-27.

[11] Linux Kernel Organization, *Video for Linux 2 (V4L2) API Documentation*, 2024, Accessed: 2025-03-30.

[12] Vadim Umrikhin and Contributors, "v4l2loopback - a kernel module to create v4l2 loopback devices," 2024, Accessed: 2025-03-30.

[13] Romain Vimont, "scrcpy (screen copy)," https://github.com/Genymobile/scrcpy, 2025, Accessed: 2025-03-30.

[14] IOhannes M. Zmölnig, "Gem - graphics environment for multimedia," https://github.com/umlaeute/Gem, 2025, Accessed:2025-03-30.

[15] Rui Nuno Capela, *QjackCtl*, 2025, Accessed: 2025-03-31.

[16] Florian Grond and Pierre Lecomte, "Higher order ambisonics for supercollider," in *Proceedings of the Linux Audio Conference*, 2017.

[17] Olivier Warusfel, "Atk for supercollider," https://github.com/ambisonictoolkit/atk-sc3, 2025, Accessed: 2025-03-31.

[18] Institute of Electronic Music and Acoustics (IEM), "IEM Plug-in Suite," https://plugins.iem.at/, 2025, Accessed: 2025-03-30.

[19] Lieven Moors and Jari Suominen, "Aj snapshot: A tool for capturing and analyzing data," https://aj-snapshot.sourceforge.io/, 2012, Accessed: 2025-03-30.

[20] Nicholas Marriott, "tmux," https://github.com/tmux/tmux, 2025, Accessed: 2025-03-30.
# ALGOJUGGLEDRUMS - CREATING RHYTHMS FROM JUGGLING PATTERNS

Albert Gräf

Computer Music Dept.

Johannes Gutenberg University (JGU) Mainz, Germany

aggraef@gmail.com

Sophia Renz

Art History

Johannes Gutenberg University (JGU) Mainz, Germany

sophia.renz03@gmail.com

## ABSTRACT

The paper introduces “algojuggledrums”, a new algorithm for translating juggling patterns into musical loops which can be played back at a given tempo. We briefly explain the underlying notions related to juggling, discuss the main routine of the algorithm, and present an implementation which encompasses a Pd patch and an external written in Lua.

## 1. INTRODUCTION

Apart from some early forerunners in the 1930s and the 1950s, electronic drum machines came into their own in the 1960s with equipment by Wurlitzer, Seeburg, and Korg. Digital grooveboxes, their modern counterpart, have taken their place in music production and beat making since the ground-breaking work of Roger Linn in the 1980s which led to the creation of the LM-1 (1980), the LinnDrum (1982), and the AKAI MPC 60 (1988). Modern versions of the MPC and similar devices by other manufacturers are still prevalent in hip hop music, but they are also used in many other genres. While early drum machines let the musician choose from a number of presets for different common rhythms, modern grooveboxes let you program beats by employing a combination of MIDI sequencing and sampling techniques.

An alternative, lesser-known approach has been the algorithmic generation of rhythms which can be found, e.g., in Clarence Barlow’s pioneering work on rhythmic “indispensabilities” [2], as well as in Godfried T. Toussaint’s “Euclidean rhythms” which have become quite popular due to their implementations in both hard- and software [14]. Also worth mentioning here is the art of “change ringing” which can be described in terms of permutation groups [8]. In contrast to grooveboxes, which are programmed like MIDI sequencers, these algorithmic approaches are based on mathematical models which allow rhythmic sequences to be generated in a more or less automatic fashion.

This is also the approach taken in this paper. The novel aspect of the method presented here is that we employ so-called “siteswap” patterns, a numerical notation system that provides a simplified, theoretical representation of specific juggling tricks. While juggling is more of a sport or a visual art form, like music it also involves abstract and repeating patterns which are often performed in alignment with an underlying *beat*. By mapping siteswap patterns onto musical and rhythmic structures, we explore an alternative to motion-based performance tracking. The use of juggling movements as a control interface for electronic music has already been investigated in several prior projects [3, 13, 15, 16]. In contrast, our approach enables the generation and control of a wide range of sequences by basically adjusting siteswap parameters – without necessarily requiring the patterns to be physically jugglable.

The paper summarizes and builds on the Master Thesis (in German) by one of the authors (Sophia Renz). If you’d like to dive deeper, the thesis is available online for your perusal [12].

## 2. SITESWAP PATTERNS

Our algorithm employs a simplified model of periodic juggling patterns known as “siteswap” patterns [11]. These are characterized by the following properties:

* The objects (let’s call them balls) are thrown at a constant beat, i.e., at equidistant points in time.

* At every beat, at most one ball gets caught, and that same ball immediately gets thrown again.

* When thrown, each ball stays in the air for a prescribed number of beats.

* Patterns are periodic, i.e., they repeat indefinitely.

Thus, the balls are caught and thrown sequentially at discrete points in time. In practice, most jugglers use both hands, thus alternating between the left and the right hand for each throw.

Under these conditions the juggling pattern can be denoted as a simple list of nonnegative numbers $a_1, a_2, ..., a_n$ which repeats indefinitely. Here each “throw” $a_i > 0$ denotes the number of beats the ball remains in the air until it gets caught and thrown again; a zero value $a_i = 0$ means that no ball gets caught or thrown at this point in the sequence. For instance, the pattern 5,0,1 indicates an infinite juggling sequence 5,0,1,5,0,1,..., where we first throw ball #1 at the first beat so that it stays in the air for 5 beats; in the second beat no balls get thrown; in the third beat ball #2 gets thrown for just 1 beat; in the fourth beat ball #2 lands and gets thrown for another 5 beats; in the fifth beat no balls get thrown; in the sixth beat ball #1 lands again and gets thrown for 1 beat; and finally, in the seventh beat, ball #1 gets thrown for 5 beats again, and the entire sequence repeats from this point on. The resulting sequence can be visualized in a so-called juggling diagram, cf. Fig. 1.

<figure><img src="image_21.png" /><figcaption>Figure 1: Juggling diagram [10, p. 3].</figcaption></figure>

Not every list of nonnegative numbers is a valid siteswap pattern, however. For a quick check, we can calculate the average throw $m = (a_1 + ... + a_n)/n$. It turns out that this must be an integer which in fact gives the number of balls required to realize the juggling sequence. For instance, our previous example 5,0,1 has an average throw of $(5+1)/3 = 2$, which is an integer. In fact, as we’ve already seen, 2 balls are needed to realize this sequence. In contrast, the sequence 5,4,2 is not a valid siteswap pattern, because $5+4+2 = 11$ and $11/3$ is not an integer. But even if a sequence has an integer average throw value, it may still not be valid, because it may cause *collisions*, i.e., different balls landing at the same time. E.g., consider the sequence 5,4,3 which has an average throw of $12/3 = 4$ and thus satisfies the average throw criterion. If we throw a ball at beat 1 then it will land again at beat $1+5=6$, because the sequence starts with a throw of 5. But the second ball to be thrown at beat 2 has a throw of 4 and will thus land at the same beat $2+4=6$, causing both balls to land at the same time, which is not permitted in a siteswap pattern.
There is an algebraic criterion which lets us decide easily whether any given sequence denotes a siteswap pattern. It involves constructing a test sequence $b_i = (a_i + i - 1) \mod n$ ($i = 1, ..., n$). The sequence is a siteswap pattern if and only if the numbers $b_i$ are all different [10]. Our sequence 5,4,3 yields the test sequence 5+0,4+1,3+2 which gives 2,2,2 (mod 3), thus 5,4,3 is not a siteswap pattern. On the other hand, the similar sequence 5,3,4 yields the test sequence 2,1,0 and thus is a siteswap pattern.

Siteswap patterns have many interesting mathematical properties, and there are algorithms to actually construct such patterns. These are beyond the scope of this paper, so we refer the reader to Polster's survey and his book instead [10, 11].

But let us briefly discuss the property that this kind of juggling patterns derives its name from. Consider a valid siteswap pattern $a_1, a_2, ..., a_n$ and two beat indices $1 \le j < k \le n$ such that $a_j \ge d = k - j$. Then it is always possible to construct a new valid siteswap pattern $b_1, b_2, ..., b_n$ such that $b_i = a_i$ for all $i \ne j, k, b_j = a_k + d$, and $b_k = a_j - d$ which effectively swaps the landing sites of the balls thrown at beats $j$ and $k$. This is called a *site swap*.

Note that a site swap is always possible in the case of two adjacent positions $k = j + 1$ if position $j$ is not empty, i.e., $a_j > 0$. In this case, $b_j = a_{j+1} + 1$ and $b_{j+1} = a_j - 1$. It is this special case which is discussed in [10, p. 6]. For instance, consider the siteswap pattern 5,3,4. By performing a swap between the two adjacent sites $j = 2$ and $k = 3$, we obtain a new siteswap pattern 5,5,2. If we swap the same two sites again, we go back to our original pattern 5,3,4. This also works in the general case, which means that the site swap operation is its own inverse (mathematicians call this an *involution*).

### 3. ANALYZING PATTERNS

Unfortunately, the algebraic criterion from the previous section doesn't provide us with an actual juggling sequence. Thus we still need an algorithm which lets us analyze a given pattern, determine whether it is a valid siteswap pattern, and construct some representation of the resulting juggling sequence. The Lua function listed in Appendix A does this. It takes a pattern (a list a.k.a. Lua table of nonnegative integer values) as input, checks for all possible error conditions, and returns two Lua tables *pat* and *objs* as result (unless it runs into an error condition, in which case it raises an exception with a trace of the computation in the error message for diagnostic purposes).

There are a lot of corner cases in the algorithm, and we won't go into all the gory details here, but note that we first check how many objects (balls) we need (making sure that the average throw is an integer), then we go through the objects one by one, filling in each throw in the *pat* table as if we were drawing a juggling diagram. We finally do another pass of the constructed *objs* table to make sure that we fill in any missing *pat* entries, and add the zero entries. If we encounter any collisions during the construction, we bail out with an exception. (The algorithm employs Kikito's *inspect* (https://github.com/kikito/inspect.lua) for creating textual representations of intermediate table results for the traces, so that we also get some useful diagnostics.)

The end result consists of two tables. The *pat* table is a finite representation of the actual juggling pattern. It is indexed by beats (counting from 1). The length of this table (the number of beats) will always be an exact multiple of the length of the input pattern. Each non-empty entry is a pair $\{i, k\}$ where $i$ is an object (ball) index running from 1 to the average throw (an integer), and $k$ is the throw, i.e., the number of beats object $i$ stays in the air. An entry can also be $\{0, 0\}$ to indicate a beat where no ball is thrown, corresponding to a zero entry in the input pattern.

The second table *objs* is used for internal bookkeeping during execution of the algorithm, but also provides useful information

about each object (ball), so we include it in the output. The *objs* table is indexed by numbers running from 1 to the number of objects (balls) used in the juggling sequence. For each object $i$, it contains a list with a complete cycle of all throws object $i$ goes through during the juggling sequence. This is also known as an *orbit* in the mathematical theory of juggling.

For instance, let's consider the pattern 5,3,4. The average throw of $12/3 = 4$ tells us that we need 4 balls to juggle this pattern. The resulting *pat* table gives a single cycle of the juggling sequence, which looks as follows:

```
{ 1, 5 } { 2, 3 } { 3, 4 }
{ 4, 5 } { 2, 3 } { 1, 4 }
{ 3, 5 } { 2, 3 } { 4, 4 }
```

Thus the sequence starts by throwing object 1 for a duration of 5 beats. In the second beat, object 2 gets thrown for 3 beats. In the third beat, we throw object 3 for 4 beats. The next three beats repeat the same pattern 5,3,4, but with the objects 4, 2, and 1, and finally beats 7-9 repeat the same pattern 5,3,4 again, but with objects 3, 2, and 4. This completes the cycle, and the entire sequence repeats when object 1 lands again in the next beat, making this a 9 beat sequence. Note that in general it will take several repetitions of the siteswap pattern to make the entire juggling sequence repeat; in this example, we needed 3 instances of the original pattern.

The *objs* table shows us the cycle of throws each individual object goes through while running through one cycle of the juggling sequence. In this example, it looks as follows:

```
{ 5, 4 } { 3 } { 4, 5 } { 5, 4 }
```

This readily tells us that object 2 always gets thrown for a duration of 3 beats, while all the other objects alternate between throws of 4 and 5 during each cycle of the juggling sequence. It's not really clear how that information might be used for musical purposes, but it's there if you need it. E.g., it might be used to draw a visual representation of the sequence such as a juggling diagram.

### 4. TURNING SITESWAP PATTERNS INTO MUSIC

Let us finally discuss how we can use the pattern data computed with the algorithm from the previous section to drive a rhythm machine. One might question whether this makes any sense at all. After all, juggling is primarily a visual experience, how are we supposed to turn this into something musically interesting? But our juggling sequences are repetitive and, as music psychologist Diana Deutsch once remarked, almost any sequence of sonic events becomes musical once you put it on repeat [6]. Also, as we have seen, the input pattern may be repeated several times with different objects (balls), which should give us some amount of interesting variation in the musical rendering.

Here is one way to turn the *pat* data output by our algorithm into MIDI. We have two numbers on each beat to work with: the object number $i$ and the throw number $k$. It's not hard to imagine that we can map objects to MIDI note numbers and throws to velocities, giving us a single MIDI note to play for each beat. This data can then be fed into some kind of synthesizer or sampler device to hear the sequence. For instance, here is a simple scheme that works quite well in practice:

* Our MIDI setup consists of two velocity values minvel and maxvel in the range 0 to 127, and a list of MIDI note numbers $n_1, n_2, ...$ that we cycle through in order to assign a MIDI note to each object.

* Map object $i > 0$ to the MIDI note number $n_i$. If the number of objects exceeds the number of MIDI notes in the list, we can just wrap around to the beginning, cycling as many times through the note list as needed to map a note number to each object.
* Map a throw value of *k* > 0 to the velocity (k-minthrow)/(maxthrow-minthrow)*(maxvel-minvel), where minthrow and maxthrow are the minimum and maximum throw values > 0 in the pattern, respectively. This gives us a linear interpolation of the velocity corresponding to *k* ranging from minvel to maxvel. This value then gets rounded to the nearest integer.

For instance, consider the pattern 5,3,4 (with minthrow = 3 and maxthrow = 5) from above. Using minvel = 80, maxvel = 120, and the MIDI notes 35 42 47 48 as the MIDI setup, this will produce the following note output in the corresponding 9-beat loop:

<table><thead><tr><td>beat</td><td>object, throw</td><td>MIDI note</td></tr></thead><tbody><tr><td>1</td><td>1, 5</td><td>35 120</td></tr><tr><td>2</td><td>2, 3</td><td>42 80</td></tr><tr><td>3</td><td>3, 4</td><td>47 100</td></tr><tr><td>4</td><td>4, 5</td><td>48 120</td></tr><tr><td>5</td><td>2, 3</td><td>42 80</td></tr><tr><td>6</td><td>1, 4</td><td>35 100</td></tr><tr><td>7</td><td>3, 5</td><td>47 120</td></tr><tr><td>8</td><td>2, 3</td><td>42 80</td></tr><tr><td>9</td><td>4, 4</td><td>48 100</td></tr></tbody></table>

## 5. PD PATCH

We have implemented these ideas as a simple Pd patch, which can be found in the accompanying materials [7], cf. Fig. 2.

<figure><img src="image_22.png" /><figcaption>Figure 2: algojuggle demo patch.</figcaption></figure>

The central component in this patch is the algojuggle object which maintains all the requisite data and cycles through the pattern with each bang message. The object is written in Lua and also incorporates the analyze_pattern algorithm from Appendix A. Note that in order to have this object recognized in Pd, you need to have the pd-lua loader extension installed [1] (please check the cited link for installation instructions).

The siteswap pattern is input as a list on the first inlet; the object then uses analyze_pattern to check that the input is a valid siteswap pattern and construct the actual juggling sequence. The MIDI setup, consisting of minvel, maxvel, and the list of note numbers as described in the previous section, gets passed on the second inlet. Upon receiving a bang message, the

object outputs the MIDI note number and velocity of the next note in the pattern on the first two outlets, which get handed over to Pd's makenote to produce the stream of MIDI note ons and offs which are output with the noteout object on MIDI channel 10 (the MIDI drumkit channel) by default. To hear the notes, you will thus have to pipe Pd's MIDI output to some GM-compatible MIDI synthesizer such as qsynth.

For each bang, algojuggle also outputs the current position in the pattern along with the object number and throw value on the third outlet, so that you can watch the object cycle through the juggling sequence. It's conceivable that in the future this data might also be used to drive a real-time animation similar to what Jack Boyce's Juggling Lab provides online [5].

The bang messages are generated with Pd's metro object. This can be turned on and off with the toggle directly above the metro object, and the tempo can be controlled with the other GUI elements at the top of the patch, using either beats per minute (bpm) or millisecond (msec) values. Alternatively, you can also leave the metro object turned off, and step through the sequence manually by clicking the bang GUI object below the metro object, which can be useful for debugging purposes.

When loading the patch, the algojuggle object is already initialized with the data shown in the screenshot, i.e., 5 3 4 as the siteswap pattern and a MIDI setup with minvel = 80, maxvel = 120, and the MIDI notes 35 42 47 48 (in the General MIDI drumkit, these are the note numbers for “Acoustic Bass Drum”, “Closed Hi-Hat”, “Low Mid Tom”, and “High Mid Tom”, respectively). Thus you can just engage the metro toggle and the patch will start playing notes immediately. You can also try changing the output MIDI channel to 1 to generate a little melody instead, and adjust the notes to your liking.

Finally, the algojuggle object also implements the site swap operation described in Section 2. This can be used at any time, also during playback, which gives you a quick way to manufacture the pattern. To these ends, the first inlet accepts a message of the form swap *i* *j*, where *i* < *j* are two indices in the original siteswap pattern. These must be such that the distance *d* = *j* − *i* is at most the throw value at position *i*, otherwise you will get an error message. If the pair constitutes a valid site swap, the resulting siteswap pattern is output on the fourth outlet, which in our patch is wired to the list GUI element for the input pattern which immediately feeds it back into algojuggle's first inlet.

Note that to keep things simple, here we have only described the most basic version of the patch. Work has also started on an extended version of the patch (algojuggle-ex.pd) which includes preliminary MIDI controller support for live improvisation, MIDI clock sync to synchronize with DAW programs or external grooveboxes, and the possibility to run multiple instances of the extended patch in concert. Please check the README file in the repository [7] for details. The repository also includes some listening samples in both MIDI and audio (WAV) format.

## 6. CONCLUSION

After listening to the sequences produced with our patch, we hope that you can agree that it is possible to obtain interesting musical results using the method sketched out in this paper. The output can be quite surprising at times when experimenting with different input patterns, which means that our method can be used to supply ideas for compositional purposes. It might even be a good tool for live performances if you prepare a collection of interesting siteswap patterns beforehand.

On the other hand, this also means that it can be hard to control the kind of musical sequences that you get with this approach. You will have to experiment. Our implementation currently doesn't include a siteswap pattern generator, so you'd want to scour the web for lists of siteswap patterns (see, e.g., Allen Knutson's Siteswap FAQ [9]), or rely on existing siteswap generator and visualization programs such as Juggling Lab [4].
At this point it is still unclear whether it is possible to characterize the musical properties of sequences generated from siteswap patterns in a way that makes it easier to specifically design such patterns for musical purposes. On the positive side, this opens up opportunities for future research and musical experimentation. Here are some specific avenues for improving the algorithm and its implementation:

* Work on integrating a library or generator algorithm for musically interesting siteswap patterns. This will require an analysis of the musical results of the siteswap pattern approach and comparison with other generative methods.

* Enhance the MIDI sequence generation with techniques found in other beatmaking devices, such as loops, note repeats (ratchets), probabilistic variations, etc.

* Design a control scheme for expressive live improvisation, by interfacing to input controllers readily available for such purposes, such as DJ and pad controllers, sensor input, etc.

## Acknowledgements

The authors would like to thank the anonymous reviewers for their helpful comments and suggestions.

## 7. REFERENCES

[1] pd-lua. https://agraef.github.io/pd-lua/.

[2] C. Barlow. *On Musiquantics*. Number 51 in Musikinformatik & Medientechnik. JGU, Mainz, 2012.

[3] T. Bovermann, J. Groten, A. de Campo, and G. Eckel. Juggling Sounds. In *Proceedings of the 2nd International Workshop on Interactive Sonification*, York, UK, 2007.

[4] J. Boyce. Juggling Lab. https://jugglinglab.org/.

[5] J. Boyce. Juggling Lab GIF server. https://jugglinglab.org/html/animinfo.html.

[6] D. Deutsch. Behaves So Strangely, Aug. 2010. https://www.radiolab.org/podcast/91513-behaves-so-strangely.

[7] A. Gräf. algojuggle-demo, Mar. 2025. https://github.com/agraef/algojuggle-demo.

[8] L. Harkleroad. *The Math Behind the Music*. Cambridge University Press, Cambridge, 2006.

[9] A. Knutson. Siteswap FAQ. http://www.juggling.org/help/siteswap/faq.html.

[10] B. Polster. Juggling Survey (The Mathematics of Juggling). https://www.qedcat.com/articles/juggling survey.pdf.

[11] B. Polster. *The Mathematics of Juggling*. Springer, New York, NY, 2003.

[12] S. Renz. Algorithmische Rhythmen – Von Jonglier patterns zu Klangsequenzen. Master’s thesis, Johannes Gutenberg University, Mainz, 2024. https://gitlab.rlp.net/srenz/algojugglerhythm.

[13] G. Silveira. Using Juggling as a Controller for Computer-Generated Music: an Overview of the Creation of an Interactive System Between Juggling and Electronic Music. Bachelor’s thesis, Universidade Federal de Pelotas, 2015.

[14] G. T. Toussaint. *The Geometry of Musical Rhythm: What Makes a "Good" Rhythm Good?* Chapman and Hall/CRC, Boca Raton London New York, 2nd edition, 2019.

[15] A. Wagenaar. Juggling as a controller of electronic music. Master’s thesis, Conservatory of Amsterdam, 2009. https://www.arthurwagenaar.nl/wp-content/uploads/Juggling-as-a-controller-of-electronic-music.pdf.

[16] A. Willier and C. Marque. Juggling Gestures Analysis for Music Control. In I. Wachsmuth and T. Sowa, editors, *Gesture and Sign Language in Human-Computer Interaction*, pages 296–306, Berlin, Heidelberg, 2002. Springer.

## A. LUA CODE

```lua
-- helper function to cycle through the
-- elements of a list
function cycle(list, i)
  i = (i-1) % #list + 1
  return list[i]
end

-- analyze a siteswap pattern
function analyze_pattern pattern
  local n = #pattern
  local sum = 0
  for _ , k in ipairs (pattern) do
    sum = sum + k
  end
  local nobjs = sum/n
  -- nobjs must be integer, otherwise the
  -- pattern isn't valid
  if math.floor(nobjs) ~= nobjs then
    error("invalid pattern, average throw must
          be integer, but got " .. nobjs)
  end
  local objs = {}
  local pat = {}
  local obj = {} -- object for each position
  local start = {} -- actual start position
  local max_pat = 0 -- maximum index in pat
  local trace = "" -- execution trace
  for i = 1, nobjs do
    local last_i = i
    while obj[ last_i ] or cycle (pattern, last_i)
      <= 0 do
        -- look for the next free position
        last_i = last_i + 1
      end
      start[i] = last_i
      local last_k = cycle (pattern, last_i)
      local seq = { last_k }
      -- set of indices (mod n) already visited
      local i_set = { [last_i % n] = true }
      while true do
        trace = trace ..
        string.format("\n%#d: pos: %d (== %d),
                      step: %d, seq: %s", i, last_i,
                      (last_i-1) % n + 1, last_k,
                      inspect (seq))
        if pat[ last_i ] then
          -- we already have a previous object at
          -- this position, bail out
          error (string.format("collision: #%d-#%d
                                    at pos %d%s", i, pat[ last_i ][1],
                                                last_i, trace))
        end
        if last_i > max_pat then
          max_pat = last_i
        end
        pat[ last_i ] = {i, last_k}
        obj[ last_i ] = i
        last_i = last_i + last_k
        last_k = cycle (pattern, last_i)
        if i_set[ last_i % n ] then
          -- position already visited, we're done
          -- but mark the position as occupied
          obj[ last_i ] = i
          break
        elseif #seq > n then
          -- if we come here, something is broken
          error("this can't happen, please check
                the algorithm!" .. trace)
        end
        table.insert (seq, last_k)
        i_set[ last_i % n ] = true
      end
```
```perl
        objs[i] = seq
    end
-- go through the objs table once again, fill
-- in missing pat entries
if max_pat % n =~ 0 then
    -- make sure that we extend the pattern to a
    -- full cycle at the end
    max_pat = max_pat + n - max_pat % n
end
for i = 1, nobjs do
    local last_i = start[i]
    local j = 1
    local seq = objs[i]
    while last_i <= max_pat do
        local last_k = cycle(seq, j)
        if not pat last_i then
            pat last_i = {i, last_k}
            trace = trace ..
            string.format("\n#%d: pat[%d]: %s", i,
                last_i, inspect pat last_i))
        elsif pat last_i][1] =~ i then
            trace = trace ..
            string.format("\n#%d: pat[%d]: %s", i,
                last_i, inspect({i, last_k}))
            -- collision, bail out
            error(string.format("collision: #d-#d
                at pos %d%s", i, pat last_i][1],
                last_i, trace))
        end
        last_i = last_i + last_k
        j = j+1
    end
end
-- fill in empty positions (0 = "no object")
for i = 1, max_pat do
    if not pat[i] then
        pat[i] = {0,0}
    end
end
return pat, objs
end
```
LAC-25 Paper Session 2
# AUD-SUR: AN AUDIO ANALYZER ASSISTANT FOR AUDIO SURVEILLANCE APPLICATIONS

Lam Pham¹*, Phat Lam²*, Dat Tran³, Alexander Schindler¹, Silvia Poletti¹, Marcel Hasenbalg¹, David Fischinger¹, Martin Boyer¹

¹ Austrian Institute of Technology, Vienna, Austria
{lam.pham, alexander.schindler, silvia-poletti, hasenbalg.marcel, fischinger.david, martin.boyer}@ait.ac.at

² Ho Chi Minh City University of Technology, Ho Chi Minh City, Vietnam

phat.lamhcmutddk21@hcmut.edu.vn

³ FPT University, Ho Chi Minh City, Vietnam

datttse170375@fpt.edu.vn

## ABSTRACT

In this paper, we present an audio analyzer assistant tool designed for a wide range of audio-based surveillance applications (This work is a part of our defame Fakes and EUCINF projects). The proposed tool, referred to as Aud-Sur, comprises two main phases: Audio Analysis and Audio Retrieval, respectively. In the first phase, multiple open-source audio models are leveraged to extract information from input audio recording uploaded by a user. In the second phase, the user interacts with the Aud-Sur tool via a natural language question-and-answer manner, powered by a large language model (LLM), to retrieve the information extracted from the processed audio file. The Aud-Sur tool was deployed using Docker on a microservices-based architecture design. By leveraging open-source audio models for information extraction, LLM for audio information retrieval, and a microservices-based deployment approach, the proposed Aud-Sur tool offers a highly extensible and adaptable framework that can integrate more audio tasks, and be widely shared within the audio community for further development.

*Items*— audio, surveillance, pre-trained model, large language model, retrieval.

## 1. INTRODUCTION

With advancements in neural network architectures and deep learning techniques, various audio-based analysis tasks have been proposed by the research community such as Audio Captioning, Audio Event Detection, Speech-to-Text, Speech Emotion Detection, etc. However, these tasks are often proposed for specific purposes and evaluated on certain benchmark datasets. For instance, audio captioning, audio event detection, speech to text, speech emotion have been evaluated using the large benchmark datasets such as Clotho [1], Audioset [2], Librispeech [3], and IMOCAP [4], respectively. Recently, pretrained audio models leveraging self-supervised learning strategies have been proposed, demonstrating potential across different audio tasks [5, 6, 7]. However, these models are designed and verified on certain relevant audio tasks (i.e., mainly human speech analysis), whereas a wide range of audio processing tasks requires different audio engineering features. For example, while Mel filter banks are suitable for Speech-To-Text task, linear filters are more effective for Deepfake Speech Detection [8]. Notably, in Acoustic Scene Classification (ASC), a combination of multiple spectrograms of Gammatonegram (GAM), Constant-Q Transform (CQT), and Mel-spectrogram (MEL) has been shown to enhance model performance [9]. Furthermore, these pre-trained audio models present the high computational complexity and require the fine-tuning process for downstream audio tasks.

In real-life audio-based applications, multiple audio tasks have to be integrated into complex systems rather than relying on a standalone task. For example, a call center support tool

<figure><img src="image_23.png" /><figcaption>Figure 1: The high-level architecture of proposed Aud-Sur tool</figcaption></figure>

may need to integrate Speech Denoise, Eco Reduction, Speech-To-Text, Speaker Identification, etc. Therefore, the ability to seamlessly integrate new audio tasks is essential for real-world audio-based applications. In the context of audio surveillance, these applications focus on audio event detection and audio tracking [10]. Existing audio surveillance systems are often designed for specific tasks such as anomaly violence detection [11], detection of children abuse [12], or riot context detection [9, 13]. However, this task-specific approach limits the ability to integrate or extend new audio tasks in the future. Furthermore, publications on audio surveillance often focus on model development rather than proposing a deployment process, resulting in a lack of comprehensive solutions [14]. To address the challenge of deploying a comprehensive solution for audio surveillance that not only focuses on model development but also provides a user-friendly practical usage, we propose:

* We leverage state-of-the-art open-source audio models, each excel in a specific task. This approach allows seamless upgrades, where an outdated model can be replaced with a more advanced one without requiring major system modifications.

* Unlike conventional audio surveillance systems that focus only on Audio Event/Scene Detection, our tool integrates a broader range of tasks such as Speech-to-Text, Speech Emotion Detection, Audio Captioning, Language Detection to provide richer context and deeper situational awareness.

* We use a large language model (LLM) to enable natural and human-like interaction between our proposed system and the user. This enhances accessibility and makes the tool more user-friendly for both technical and non-technical users.

* To enhance scalability and adaptability, we design the proposed system using a microservices-based architecture and Docker technique. This approach allows seamless updates,

<footnote>* Main and equal contribution into the paper.</footnote>
<figure><img src="image_24.png" /><figcaption>Figure 2: Logical processing flow in the Audio Analysis Phase (AAP)</figcaption></figure>

parallel execution, and easy extending and integration of new models in the future.

## 2. HIGH-LEVEL ARCHITECTURE

Figure 1 presents the high-level architecture of the proposed Aud-Sur tool. In overall, the tool comprises three main components. The first component is the User Interface (UI). The UI serves as an entry point where users interact with the tool by uploading an audio file and submitting a query or question, typically in natural language, to request specific information about the audio. The second component is the Audio Analysis Phase (AAP) which processes the uploaded audio recording by performing various audio tasks. The AAP component extracts task-structured results representing multiple aspects of the audio input. For each audio task, the results are exported in a JSON file, which is then aggregated and standardized to ensure consistency across factors such as time boundaries and missing values in the JSON results from other audio tasks. Finally, the Audio Retrieval Phase (ARP) utilizes the standardized results to interpret the extracted information and retrieve relevant insights based on the user's question, formulating a corresponding response in natural language.

### 2.1. Audio Analysis Phase (AAP)

The Audio Analysis Phase (AAP) aims to extract structured information representing different aspects of input audio recording. The extracted data serves as the main reference resource for generating responses in the consequent Audio Retrieval Phase (ARP). Since the quality of the extracted information directly impacts the accuracy and relevance of the final response, ensuring high-quality results in this phase is crucial. Therefore, instead of relying on a single model to perform multiple tasks, we employ multiple individual models, each of which was optimized for a specific audio task and achieved high performance on benchmark datasets.

To describe the APP in detail, we present Figure 2, which provides a comprehensive flowchart for processing input audio. In particular, at the top of the flow chart (Figure 2A), the APP process begins with the Preprocessing module, which performs multiple steps such as resampling, filtering, and changing representation format (e.g., raw audio, spectrogram, MFCC) to ensure

<table><caption>Table 1: Individual audio models used in the AAP component</caption><thead><tr><td>Models</td><td>Tasks</td><td>Licenses</td></tr></thead><tbody><tr><td>Whisper [5]</td><td>Speech to Text (S2T)<br/>Language Detection (LD)</td><td>MIT</td></tr><tr><td>mBart [15]</td><td>English Translation (ET)</td><td>MIT</td></tr><tr><td>PANN [16]</td><td>Acoustic Event Detection (AED)</td><td>MIT</td></tr><tr><td>pyannote [17]</td><td>Speaker Count (SC)</td><td>MIT</td></tr><tr><td>ECAPA-TDNN [18]</td><td>Voice Gender Detection (VGD)</td><td>MIT</td></tr><tr><td>SpeechBrain [19]</td><td>Speaker Diarization (SD)</td><td>Apache license 2.0</td></tr><tr><td>Wav2vec2.0 [19, 6]</td><td>Speech Emotion Recognition (SER)</td><td>Apache license 2.0</td></tr><tr><td>KD [20]</td><td>Audio Captioning (AC)</td><td>Apache license 2.0</td></tr><tr><td>AIT-ASC [9, 21]</td><td>Acoustic Scene Classification (ASC)</td><td><strong>Our development</strong></td></tr><tr><td>AIT-DSD [22, 23, 8]</td><td>Deepfake Speech Detection (DSD)</td><td><strong>Our development</strong></td></tr></tbody></table>

the alignment between the input audio and each of pre-defined single models. Then the preprocessed audio is fed into multiple audio models to extract audio information. To assure a more organized and effective extraction of relevant information from an audio recording, we categorize multiple audio models in this phase into three main groups, referred to as the sound-based tasks (Figure 2B), speech-based tasks (Figure 2C), and summary-based tasks (Figure 2D). These three task groups analyze the input in parallel. Notably, for speech-based tasks, a preliminary speaker count module [17] is first leveraged to detect the presence of human speech. If at least one speaker is identified, the system continues speech-based tasks processing; otherwise, these tasks are skipped to optimize computational efficiency.

For each task group, the detailed process is presented in Figure 2B, Figure 2C, and Figure 2D, respectively. The first group of sound-based tasks focuses on analyzing the background environment and non-speech elements within the recording. They include specific audio tasks of Acoustic Scene Classification (ASC) [9, 21] (e.g., recordings into one of multiple pre-defined acoustic scene classes such as 'airport', 'park', 'outdoor', 'in door', etc.), Acoustic Event Detection (AED) [16] (e.g., recognizes specific auditory events such as 'sirens', 'alarms', or 'footsteps', etc). As the tool is applied for surveillance applications, we highly focus on detecting events indicating violent contexts such as 'gun', 'fireworks', 'screaming', 'explosion', etc, by adding violent/alarm tags on the task results for the next ARP processing. As Figure 2B describes, we employ sound-based tasks using 10-second segments instead of input recording for greater flexibility in handling variations in background environ-
<figure><img src="image_25.png" /><figcaption>Figure 3: Zero-shot Prompting with Filtering & Response control (ZPFR)</figcaption></figure>

ments and sound events. Then, the segment-based results of these two models are gathered and collectively referred to as 'Acoustic information'. Notably, before finalizing the AED results, we apply an additional refinement step, in which we assign a 'violent' tag to any segments containing at least one event indicative of violent behaviors. Among the 500+ class labels in AudioSet [2], which the AED model was trained on, we manually selected a subset of class events associated with violence or riots and used them as label references for this tagging process. This serves as additional auditory cues in the next ARP to better enhance situational awareness for the main purpose of audio surveillance.

The second group, referred to as speech-based tasks, is designed to extract information related to human speech. These incorporate Speech-To-Text (S2T) [5] (e.g., transcribing spoken content into text), Speaker Diarization (SD) [19] (e.g., segmenting audio based on personal identity or simply identifies 'who spoke when'), Speech Emotion Recognition [19, 6] (e.g., determining the speaker's emotional state) and Language Detection (LD) [5] (e.g., detecting the spoken language). The flow to process results in this task group is described in Figure 2C. First, a Speech-To-Text model (e.g., Whisper) is utilized to generate text transcriptions, which are segmented based on their respective time boundaries. Since Whisper is a general-purpose model, we derive language identification results directly from the language tag in its multitask sequence tokens. Using the time boundaries provided by Speech-To-Text model, we further divide the audio input into segments, which then serve as inputs for the three following models: Speech Emotion Recognition (SER), Speaker Diarization (SD), and Voice Gender Detection (VGD). In other words, these models process audio segments defined by Whisper's segmentation. Assuming Whisper's time boundary estimation is sufficiently accurate, this approach of segmentation ensures that speaker-related attributes such as identity, gender, and emotion are captured precisely at the moment of speech, reducing errors associated with multiple speakers overlapping in a single segment when using fixed-length segments that could affect the performances of the next three single models. The results of single models in the speech-based tasks are also aggregated, referred to as 'human speech information'.

The final group is referred to as the summary-based tasks which provides a high-level understanding of the audio content. The tasks include Audio Captioning (AC) [20], which generates textual descriptions summarizing the content of an audio recording, and Deepfake Speech Detection (DSD) [22, 23], which determines whether a given speech sample has been artificially manipulated or is authentic. As illustrated in Figure 2D, models in this category process input audio based on fixed-length segments. The Audio Captioning (AC) model operates on 10-second segments to accommodate potential shifts in the overall content of the recording, while the DSD model processes 4-second segments, which are sufficient to capture distinctive artifacts in

<figure><img src="image_26.png" /><figcaption>Figure 4: Chain-of-thought (CoT) prompting</figcaption></figure>

human speech. The final output of the DSD model is obtained through majority voting on the segment-wise results. Again, the outputs from this task group are then gathered and collectively referred to as 'other information'.

Given JSON file results from task groups, these JSON files are then aggregated and standardized. The Standardize module aims to re-structure the final information of the audio recordings. In our work, we employ a timeline-centric standardization approach that arranges the results based on a shared timeline, where single task's outputs are aligned according to their occurrence throughout the audio recording. By structuring the AAP into such content-based categories, our proposed tool ensures a systematic and scalable approach to extracting meaningful insights from recordings. This modular framework allows for greater expansion and management (i.e., new tasks or improved models can be categorized and integrated without re-designing the entire system).

## 2.2. Audio Retrieval Phase (ARP)

The Audio Retrieval Phase (ARP) plays a crucial role in bridging the gap between audio analysis results from the AAP and a human-interpretable response. Given the final information JSON output from the AAP component, which may be fragmented or lack context, the ARP leverages a Large Language Model (LLM) to process and contextualize the information and then transfer it into a coherent and informative response tailored to the user's query. Since the application is designed for surveillance purposes, the ARP needs to address several challenges including irrelevant queries/questions (e.g., A user asks about topics unrelated to the audio content, leading to off-topic or unhelpful responses), LLM Hallucinations (e.g., the LLM might generate inaccurate or misleading information despite being provided the extract audio content), etc. To ensure reliable and contextually relevant responses, we explore two prompting techniques:

**Zero-shot Prompting with Filtering & Response Control (ZPFR):** Zero-shot prompting is a technique in which an AI model is given a task or question without any prior examples or specific training, relying solely on its pre-existing knowledge and prompt to generate a response [24]. While effective in many cases, zero-shot prompting alone may not guarantee precision or adherence to specific guidelines. To improve response quality, we integrate filtering rules into the system prompt to pre-process user queries and eliminate irrelevant inputs. Additionally, response control rules are applied to guide the model in generating outputs that align with pre-defined objectives. An example of such type of prompting is illustrated in Figure 3.

**Chain-of-Thought Prompting:** Chain-of-thought (CoT) prompting [25] is a technique that significantly improves the reasoning capabilities of large language models (LLMs). Instead of simply asking for a final answer, CoT prompting encourages the model to break down complex problems into a series of intermediate steps, mimicking human-like reasoning. Based on this
<figure><img src="image_27.png" /><figcaption>Figure 5: Deployment of our proposed AUD-SUR tool as multiple services leveraging Docker</figcaption></figure>

<figure><img src="image_28.png" /><figcaption>Figure 6: The user interface (UI) of our proposed Aud-Sur tool</figcaption></figure>

<figure><img src="image_29.png" /><figcaption>Figure 7: Our proposed Aud-Sur tool operates as an interactive assistant</figcaption></figure>

mechanism, we try to guide the LLM toward various reasoning steps to implicitly handling irrelevant queries and reducing hallucination above before generating final responses. An example of this type of prompt can be described as Figure 4.

# 3. DEPLOYMENT

## 3.1. Module Implementation

The deployment of our proposed Aud-Sur tool comprehends the design of a User Interface (UI), independent audio models used in the AAP component, and LLM in the ARP component. First, User Interface (UI) is implemented by Streamlit platform [26]. Regarding audio models in the AAP component, the Table 1 presents the pre-trained audio models along with their respective licenses for specific audio tasks. To showcase the feasibility of expanding the scope of the proposed AAP and share further contributions in the context of audio surveillance with the research community, we integrate the APP with our self-deployed models of Acoustic Scene classification and Deepfake Speech Detection, along with open-source models for other audio tasks. All audio models utilize deep neural networks and are implemented using the PyTorch framework. Regarding the ARP component, we use two approaches to leverage the operation of Large Language Model (LLM). The first approach involves interacting with the

LLM via an API, while in the second one, we run the LLM locally, providing greater control and data privacy. In both cases, we utilize LLaMA [27] as our LLM with the following configurations: a temperature of 0.3, a maximum token limit of 256 to constrain responses, and a top-p value of 0.5 for nucleus sampling.

## 3.2. Deployment Environment

We utilize Docker and FastAPI to deploy our proposed Aud-Sur tool following a client-server architecture as illustrated in Figure 5. The client side consists of a user interface (UI). This UI allows users to interact with the system and submit queries. The queries and response results are processed with the support from prompting designs and a Large Language Model (LLM). The server side is structured as multiple independent services, with each service responsible for a predefined group of tasks, as mentioned in Section 2. In other words, each service includes some audio components which use the same Python package libraries and Conda dependencies. Each service also includes an independent API which is used to communicate via predefined ports and collectively generate results for the client side.

Regarding Docker containerization, described at the bottom of Figure 5, we first construct a base Docker image that includes common dependencies (e.g., Conda environments, GPU-enabled
<figure><img src="image_30.png" /><figcaption>Figure 8: Compare LLM response using ZPFR prompting and CoT prompting. The ‘record_1.wav’ involves a confidential reconnaissance unit report between two German speakers.</figcaption></figure>

inference installations, and essential libraries). Building upon
this base image, each service is packaged into a separate Docker
image and run independently in its own Docker container. The
services interact via API calls and results are transferred to the
client through port connections. By following this approach, we
achieve a modular, scalable and efficient deployment process,
ensuring independent execution of all system functionalities.

4. DEMONSTRATION AND DISCUSSION

This section illustrates a live demonstration of our proposed Aud-
Sur tool. Figure 6 presents the User Interface, where users can
upload an audio file in ‘.wav’ or ‘.mp3’ format and submit text-
based queries or questions to inquire information related to the
uploaded audio. The tool processes each query/question by ana-
lyzing the input, extracting relevant information, and then gen-
erating a corresponding response. The interaction between the
user and the system is depicted in Figure 7. Unlike conventional
methods that rely on a single model’s input-output pipeline, the
integration of a LLM in the ARP component of our proposed
Aud-Sur tool enables the tool function in a chatbot-like manner
that facilitates a more dynamic and context-aware approach to
information retrieval. This approach overcomes the limitations
of standalone audio models, particularly in cases where a user’s
query requires information from multiple models or consists of
multiple sub-questions. For instance, in Figure 7, the user’s third
query "What are the speakers in this recording talking about?
How do they feel when talking to each other?" requires results
from both Speech-to-Text and Speech Emotion Detection. Lever-
aging multiple-model approach combined with the support of
LLM, our Aud-Sur tool could combines information from multi-
ple sources to generate a comprehensive response. Additionally,
with prompting support, our system can filter out queries that
are irrelevant to the audio content. As illustrated in Figure 7, the
system politely rejects off-topic questions (highlighted in red)
and instead prompts the user with more contextually appropriate
questions.

To assess the impact of different prompting techniques, we

evaluate our tool’s responses using ZPFR and CoT prompting
on the same set of user queries. Given that our tool is designed
for surveillance applications, we specifically analyze its ability to
monitor and detect audio indicators related to potential threats,
such as extreme audio events, riots, violent sounds, pre-crime
activities, etc. For these purposes, we conduct demonstrations
using two audio samples that contain diverse information rel-
evant to our objectives. Specifically, the first sample, named
‘record 1.wav’, features a confidential reconnaissance unit report
conversation between two German speakers, aiming at evaluat-
ing our system’s ability to extract sensitive intelligence-related
information in human speech. The second one, referred to as
‘record 2.wav’, is created by combining three different audio
recordings with different background scenes, with a riot occur-
ring in the first 15 seconds. The second recording is designed to
test the system’s effectiveness in detecting security threats and
handle context shifting.

As shown in Figure 8 and Figure 9, our Aud-Sur tool success-
fully extracts relevant information across varied audio environ-
ments. The analysis also shows differences in the results using
the two prompting techniques. In particular, ZPFR method tends
to provide more natural, controlled and compact responses. In
contrast, CoT prompting produces more structured and detail-
oriented, due to its predefined reasoning steps. These differences
highlight the importance of optimizing a prompting strategy to
enhance accuracy, reliability, and suitability of LLM-driven real-
world audio surveillance applications.

5. CONCLUSION

We have presented Aud-Sur, an audio analyzer assistant tool, for
audio surveillance applications<fnref n="1" /> (i.e., The tool can be directly
used by authorities for early detecting potential crimes). It lever-
ages various audio models for information extraction and inte-
grates a large language model (LLM) for natural language-based
audio information retrieval. With the integration of multiple open-

<footnote n="1">¹The tool was finished and planed for a demo or an access in the LAC
conference duration.</footnote>
<figure><img src="image_31.png" /><figcaption>Figure 9: Comparison of LLM responses using ZSFR prompting and CoT prompting. The ‘record 2.wav’ file is created by combining three separate audio recordings, with the first 15 seconds depicting a riot scene.</figcaption></figure>

source audio models and the use of microservices and Docker for deployment, our system enables seamless integration of new audio models/tasks, making the Aud-Sur tool highly adaptable and suitable for a wide range of audio surveillance applications. Inspired by surveillance applications and leveraging open source models, we further analyze and research on video in the future work, in which both audio and image information are explored.

## 6. ACKNOWLEDGMENTS

EUCINF project is co-funded by European Union under grant agreement N°101121418. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the granting authority can be held responsible for them.

Defame Fakes is funded by the Austrian security research programme KIRAS of the Federal Ministry of Finance (BMF).

<figure><img src="image_32.png" /></figure>

FFG
Forschung wirst

KIRAS
Bibliotheken-forschung

<figure><img src="image_33.png" /></figure>

Bundesministerium
Finanzen

## 7. REFERENCES

[1] Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen, “Clotho: an audio captioning dataset,” in Proc. ICASSP, 2020, pp. 736–740.

[2] Jort F. Gemmeke et al., “Audio set: An ontology and human-labeled dataset for audio events,” in Proc. ICASSP, 2017, pp. 776–780.

[3] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur, “Librispeech: An asr corpus based on public domain audio books,” in Proc. ICASSP, 2015, pp. 5206–5210.

[4] Carlos Busso et al., “Iemocap: Interactive emotional dyadic motion capture database,” Language resources and evaluation, vol. 42, pp. 335–359, 2008.

[5] Alec Radford et al., “Robust speech recognition via large-scale weak supervision,” in Proc. ICML, 2023, pp. 28492–28518.

[6] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli, “wav2vec 2.0: a framework for self-supervised learning of speech representations,” in Proc. NIPS, 2020.

[7] Sanyuan Chen et al., “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022.

[8] Lam Pham, Phat Lam, Dat Tran, Hieu Tang, Tin Nguyen, Alexander Schindler, Florian Skopik, Alexander Polonsky, and Hai Canh Vu, “A comprehensive survey with critical analysis for deepfake speech detection,” Computer Science Review, vol. 57, pp. 100757, 2025.

[9] Lam Pham, Dat Ngo, Dusan Salovic, Anahid Jalali, Alexander Schindler, Phu X. Nguyen, Khoa Tran, and Hai Canh Vu, “Lightweight deep neural networks for acoustic scene classification and an effective visualization for presenting sound scene contexts,” Applied Acoustics, vol. 211, pp. 109489, 2023.

[10] Marco Crocco, Marco Cristani, Andrea Trucco, and Vittorio Murino, “Audio surveillance: A systematic review,” ACM Comput. Surv., vol. 48, no. 4, 2016.

[11] Jungpil Shin, Abu Saleh Musa Miah, Yuta Kaneko, Najmul Hassan, Hyoun-Sup Lee, and Si-Woong Jang, “Multimodal attention-enhanced feature fusion-based weakly supervised anomaly violence detection,” IEEE Open Journal of the Computer Society, vol. 6, pp. 129–140, 2025.

[12] Jiuqi Yan, Yingxian Chen, and W.W.T. Fok, “Detection of children abuse by voice and audio classification by short-time fourier transform machine learning implemented on nvidia edge gpu device,” in Proc. PRAI, 2023, pp. 893–897.

[13] Lam Pham, Tin Nguyen, Phat Lam, Hieu Tang, and Alexander Schindler, “Toolchain for comprehensive audio/video
analysis using deep learning based multimodal approach: Use case of riot or violent context detection,” in Proc. CBMI, 2024, pp. 1–4.

[14] Dalila Durães, Francisco S Marcondes, Filipe Gonçalves, Joaquim Fonseca, José Machado, and Paulo Novais, “Detection violent behaviors: a survey,” in Ambient Intelligence–Software and Applications: 11th International Symposium on Ambient Intelligence. Springer, 2021, pp. 106–116.

[15] Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer, “Multilingual denoising pre-training for neural machine translation,” *Transactions of the Association for Computational Linguistics*, vol. 8, pp. 726–742, 2020.

[16] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 28, pp. 2880–2894, 2020.

[17] Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill, “pyannote.audio: neural building blocks for speaker diarization,” 2019.

[18] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck, “Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,” in *Proc. INTERSPEECH*, 2020, pp. 3830–3834.

[19] Mirco Ravanelli et al., “SpeechBrain: A general-purpose speech toolkit,” 2021, arXiv:2106.04624.

[20] Xuenan Xu, Haohe Liu, Mengyue Wu, Wenwu Wang, and Mark D Plumbley, “Efficient audio captioning with encoder-level knowledge distillation,” in *Proc. INTERSPEECH*, 2024, pp. 1160–1164.

[21] Lam Pham, Khoa Tran, Dat Ngo, Hieu Tang, Son Phan, and Alexander Schindler, “Wider or deeper neural network architecture for acoustic scene classification with mismatched recording devices,” in *Proc. ACM MM Asia*, 2022, pp. 1–5.

[22] Lam Pham, Dat Tran, Florian Skopik, Alexander Schindler, Silvia Poletti, David Fischinger, and Martin Boyer, “DIN-CTS: Low-complexity depthwise-inception neural network with contrastive training strategy for deepfake speech detection,” *arXiv preprint arXiv:2502.20225*, 2025.

[23] Lam Pham, Phat Lam, Truong Nguyen, Huyen Nguyen, and Alexander Schindler, “Deepfake audio detection using spectrogram-based feature and ensemble of deep learning models,” in *Proc. IS2*, 2024, pp. 1–5.

[24] Yinheng Li, “A practical survey on zero-shot prompt design for in-context learning,” in *Proceedings of the Conference Recent Advances in Natural Language Processing - Large Language Models for Natural Language Processings*, 2023, p. 641–647.

[25] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in *Proceedings of the 36th International Conference on Neural Information Processing Systems*, 2022.

[26] “Streamlit platform,” https://streamlit.io/, 2021.

[27] Hugo Touvron et al., “Llama: Open and efficient foundation language models,” 2023.
# AUDIOREACH™: AN END-TO-END AUDIO SOLUTION

Patrick Lai

Principal Software Engineer
Qualcomm Technologies, Inc., USA
plai@qti.qualcomm.com

## ABSTRACT

The AudioReach solution developed by Qualcomm Technologies, Inc. addresses significant obstacles in the audio development landscape. This paper explores the comprehensive capabilities provided by AudioReach, including off-target development, graphical design, testing, and deployment. The framework supports heterogenous or distributed processing and offers a unified environment for developing audio use cases across various processors and platforms. By open-sourcing AudioReach[1], Qualcomm Technologies aims to foster a community-driven project that enhances audio development efficiency and lowers costs for vendors, algorithm and system use case designers, tuning engineers, and system integrators.

## 1. INTRODUCTION

Qualcomm Technologies' diverse product portfolio spans both ends of the computing spectrum, from high performance to low cost (minimal power, cycle, and memory footprint). To support a broad range of audio and voice use cases across diverse SoCs, platforms, and product devices, a scalable and adaptable solution is essential. To facilitate the integration of various deliverables, including algorithms, system software, and tuning data, a developer-friendly solution is essential to empower multidisciplinary developers to seamlessly combine their contributions into a cohesive final product. Qualcomm Technologies has developed a comprehensive SDK to address these needs. An open-source project has also been initiated, recognizing the need for a community sponsored solution to fully address gaps in the existing open-source software landscape. This paper explores various aspects of the AudioReach SDK, including key software components and constructs, the development workflow it enables, and other aspects and highlights of the open-source project.

## 2. OPEN SOURCE PROJECT

In examining the open-source landscape, it is evident that there is no single solution addressing all the following needs:

* A unified, integrated environment for developing algorithms, signal flow graphs with interconnected processing modules, and system use cases.

* A comprehensive infrastructure for configuring, monitoring, tuning, and optimizing system use cases.

* A framework supporting distributed processing across heterogeneous CPU, MUC, and DSP cores or SoCs.

* Independent control of use cases in low-power and high-performance subsystems on the same device.

Later sections of the paper will dive into detail of AudioReach SDK so it would become apparent how SDK would address the gaps. Meanwhile, AudioReach has commercialized on many product lines including wearables, automobiles, handsets, Extended Reality, and computing devices which are built on sophisticated SoCs with both high performance and low power subsystems running different open-source and proprietary operating systems. The supported use cases on these devices include,

but are not limited to, typical audio playback, recording, circuit-switched calls, low-power voice UI functionalities, and always-on audio sensors. With wide product deployment and a rich set of capabilities, Qualcomm Technologies believes the community can benefit from the AudioReach SDK and has decided to open-source the solution as a community sponsored project. While detailed information about the project can be found in the project site, this paper highlights three key aspects:

* Given the portability of AudioReach, the project is not limited to Qualcomm® platforms but has also been (and will be) actively ported to non-Qualcomm platforms as part of the evolving project roadmap.

* The project has chosen the BSD-3-Clause Clear license to benefit the broader ecosystem.

* The project aims to be community-driven and seeks to establish a steering committee to collectively define the project's roadmap. To demonstrate our commitment to involving the community, Qualcomm Technologies is overhauling internal processes to facilitate public development.

AudioReach SDK is available as part of Qualcomm® Linux® Yocto and Ubuntu distribution. As for non-Qualcomm platforms, Yocto meta layer is available to enable AudioReach on Raspberry Pi while work is in progress to port to SoCs with Xtensa DSP. Please note that the current contribution is focused on Linux due to the open-source nature of Linux ecosystem. The scope of contributions will be broadened once AudioReach support is extended to other open-source operating systems such as Zephyr.

AudioReach open-source project is still in its infancy. While embedded software components and AudioReach Creator (GUI-based use case & tuning configuration tool) have been built up, figure 1 illustrates the proposed roadmap that includes plans to support additional third-party hardware, build up off-target development capabilities, integration with MathWorks tools, and release open-source OS-agnostic version of AudioReach Creator (ARC). Today, public/free version of ARC is distributed as Windows executable through Qualcomm developer network.

## 3. AUDIOREACH SDK

The AudioReach SDK includes software components for host and embedded devices, enables developers to design, simulate, and tune signal flow graphs in real-time. The solution supports interoperability with standard APIs and frameworks such as PulseAudio[3], and PipeWire[4]. AudioReach is highly portable, allowing developers to enjoy the same environment and workflow across different hardware and OS platforms.

### 3.1. Design Objectives

Prior to delving into the architecture and components of the SDK, it is essential to comprehend the primary design objectives that guide the AudioReach architecture and implementation.

#### Leveragability

* Portability across different SoC / HW vendor platforms using same development environment and workflow.
<figure><img src="image_34.png" /><figcaption>Figure 1: AudioReach roadmap</figcaption></figure>

* Interoperability with OS ecosystem software and tools

## Scalability

* Modularity that allows the same solution to scale up or down to with capabilities and limitations (e.g., cycle or memory constrained environments).

* Distributed processing across multiple heterogeneous cores and SoCs

## Extensibility

* Data driven customization minimizing coding and integration effort

* Extensible interfaces, use case logic, components, and signal processing framework elements

## Capability

* Common development environment and tooling across entire Single tool to enable full development workflow

## 3.2. High-level Architecture and Key Components

AudioReach is comprised of platform independent and platform dependent (adaptation layer) software spanning across host and embedded device as illustrated in Figure 2. AudioReach Creator (ARC), formerly known as Qualcomm® Audio Calibration Tool (ACT), is PC-based application exposing a GUI to the audio system designer for connecting processing modules into audio signal flow graphs, configuring, and storing graphs in an audio calibration database (ACDB). This database can be loaded into the device at run-time and used to configure signal flow graphs deployed across various audio and voice use cases. Furthermore, this database can be updated at run-time and used, in addition to ARC, by tuning and use case design engineers to tune, configure, and monitor audio and voice processing blocks in real-time. These capabilities are referred to as “real-time tuning and calibration (RTC)” and “integrated resource monitoring (IRM)”.

The AudioReach software components running on the embedded device in the Platform Adaptation Layer (PAL), graph services (ARGS), and AudioReach Processing Engine (ARE). The ARE includes a Signal Processing Framework (SPF) along with reference algorithms as shown in the diagram below. Depending on the SoC design, the ARE usually runs in the DSP subsystem running RTOS while audio platform software typically resides on the application subsystem running a high-level operating system such as Linux. Communication between ARGs and ARE is facilitated using an IPC protocol called Generic Packet Router (GPR).

Typically, applications developed for ubiquitous software platforms use standardized APIs and multimedia/audio frameworks. To align with these standardized APIs, a platform adaptation layer (PAL) is used to translate platform-specific APIs and constructs to ARGs APIs and constructs. In addition, ARGs and ARE is built on top of a platform and OS abstraction layer to allow the AudioReach solution to scale across different OS and hardware platforms.

<figure><img src="image_35.png" /><figcaption>Figure 2: Subsystem view</figcaption></figure>

## 3.3. High-level Software Flow

ARGS is the conduit between ACDB and the ARE. It manages the opening and closing of use case signal flow graphs and exchange of real-time audio and voice signal data, meta-data, and control / configuration data. As previously stated, ACDB stores audio signal flow graph connection information and calibration data. When a use case is launched (client requests a graph to be opened), ARGs retrieves and downloads the graph definition and calibration data from ACDB and delivers it to ARE using GPR. The ARE then constructs the signal flow graph using the processing module connection information provided and attaches that graph to applicable source and sink endpoints. The processing modules are attached or plugged into the ARE using a Common Audio Processing Interface (CAPI).
## 3.4. Fundamental concepts and Constructs

The AudioReach SDK is built on several core concepts and constructs, including module, audio graph, sub-graph, and containers as depicted in figure 3. Audio graph represents the interconnection of modules to achieve specific use cases, while sub-graphs are logical abstractions for groups of modules. Containers is the concept of ARE which allows system designers to group and execute audio processing modules together in a single software thread. In the graph definition, modules are assigned to run on specified CPU domains.

To load and configure signal flow graphs, AudioReach identifies graph, calibration data, and destination modules using a vector of keys and values (KV). Using ARC, the system designer associates a key-value vector to desired signal flow graph, which also attaches or tags calibration and other meta-data to modules within that graph.

<figure><img src="image_36.png" /><figcaption>Figure 3: Graph composition</figcaption></figure>

## 4. DEVELOPMENT WORKFLOW

Although off-target development capabilities are yet fully implemented as indicated in the project roadmap, AudioReach is envisioned to support a complete beginning to end workflow that includes off-target simulation and testing to on-target characterization, optimization, and verification. This off- to on-target development workflow facilitates a seamless, low effort migration to hardware platforms and devices running AudioReach, which reduces development cost and time-to-market for developers. As depicted in Figure 4, the typical development workflow using AudioReach includes the following basic steps while developers who already have IP in C can skip step 1 and 2:

1. Algorithm library development starts off-target, using standard algorithm design tools such as MATLAB/Simulink to design with a wealth of primitives, simulate, and automatically generate architecture optimized code

2. In System Composer, compose signal flow graph with algorithm libraries from Simulink, simulate, partition processing workload across cores based on simulation results, and auto generate CAPI-wrapped modules

3. A specific AudioReach development workspace is created in AudioReach Creator (ARC) by importing meta data from an existing product (ACDB, ARGs, ARE, and module configuration and header files) and/or netlist with AudioReach specific meta data from System Composer project if applicable

4. Import algorithm CAPI modules into Visual Studio Code and compile, using architecture-specific toolchain, as part of AudioReach simulation platform or loadable library which can be loaded into simulation platform dynamically

5. Use ARC to simulate, characterize, optimize, and validate use case designs including the algorithm modules on simulation platform before moving onto a given embedded device

6. Export use case data including graph configuration, calibration, tagging information out of ARC and into a target embedded device.

7. Once code and use case data is successfully imported into the target embedded device, ARC can be used to configure, tune, calibrate, and monitor the signal processing software in real-time. AudioReach also supports modifying the signal flow graph definitions (topology, parameters, et cetera) at run-time, obviating the need to rebuild (compile and link) the software and reboot the device.

<figure><img src="image_37.png" /><figcaption>Figure 4: Development workflow</figcaption></figure>

## 5. AUDIOREACH CREATOR

AudioReach Creator is a GUI based application that serves as the primary front-end for designers and developers to define, configure, optimize, monitor, and validate their system use cases. The key attributes of ARC include the following:

* Compose, configure, optimize, and store use case signal flow graph data into workspace and embedded audio configuration database (ACDB) files.

* Distribute sections of signal flow graph to different cores.

* Activate use cases in both off-target and on-target environments.

* Monitor and profile cycle and memory usage in real-time at different granularities (per module, per thread, per core).

* Coordinate loading of dynamically (run-time) modified signal flow graphs into embedded device without rebuilding or rebooting device.

* Dynamically tune audio and voice signal processing modules locally and remotely in both offline and real-time modes, including pushing modified parameters into persistent storage (ACDB) on the embedded device.

* Interoperate with multiple third-party tuning tools and applications using a published command line interface.

## 6. AUDIOREACH ENGINE

The AudioReach Engine (ARE) is a comprehensive signal processing framework (SPF) designed to manage a broad spectrum of audio and voice signal processing flows. It adopts a client-server model where the server executes algorithms and a client framework that bridges the high-level software stack and the server framework. Figure 5 illustrates high-level architecture of ARE. The platform and OS abstraction layer (POSAL) ensures the framework works on different processors and hardware architectures. The ARE consists of several functional blocks, including the Generic Packet Router (GPR), Audio Processing
Manager (APM), and Audio Module Data Base (AMDB). The GPR provides routing functionality for audio message packets across the ARE and ARGs. The APM is responsible for setting up and managing the use case graphs in the ARE. The AMDB provides the database of CAPI-wrapped topology and processing modules in the ARE.

ARE is a highly capable, flexible, and extensible framework. Signal processing modules are easily integrated into the framework using CAPI. The framework is modularized to support many different types of signal flow graphs, threading configurations, and end point types. ARE is equipped with an extensive range of capabilities, allowing the solution to span across a broad set of diverse products. As depicted in Figure 5, the following list of capabilities showcase the breadth and depth of the ARE solution:

* Supports fixed and floating point PCM data format, and various standard compressed data formats.

* Supports multiple scheduling modes and data delivery mechanisms.

* Supports multiple (independent or interdependent) instances of AREs across heterogeneous CPU, MCU, and DSP cores.

* Supports media format propagation across the processing modules in the signal flow graph.

* Supports metadata propagation across the signal flow graph (e.g., propagation of end-of-stream event).

* Supports dynamic loading of modules and entire signal flow graphs.

* Supports control link across modules to exchange messages, intents, or other meta-data.

* Supports partial tear down and incremental construction of signal flow graph which can be useful for device switch scenario.

* Supports flexible and configurable frame sizing across modules and containers.

* Supports inter-module and -container buffering to allow interconnection of modules designed for different frame sizes.

* Supports real-time calibration and processing data monitoring (signal, cycle, memory, etc.) or Integrated resource monitoring (IRM).

* Supports clock voting to optimize performance and power.

* Supports flexible placement of PCM data logging in signal flow graph.

## 7. AUDIOREACH GRAPH SERVICES

AudioReach graph services (ARGS) are a collection of cross-platform software libraries, as depicted in Figure 7, that provide the following services:

### Use Case Execution:

* Establish communication channel with AudioReach Engines (ARE)

* Set up and tear down audio graphs running on ARE for intended use cases

* Provide audio data interface to client to exchange audio data with ARE

* Provide control interface allowing client to apply control commands on intended modules in the audio graph

* Relay module event to client

* Apply calibration and other configuration data on the signal flow graph

* Monitor and recover from ARE faults and restart events

<figure><img src="image_38.png" /><figcaption>Figure 5: ARE high-level architecture</figcaption></figure>

### Development Workflow:

* Log audio samples or module-specific data at configurable points in the signal flow graph

* Perform real time tuning of audio processing modules running in ARE

* Download new calibration data to the device

* Monitor system and module resource in real-time

* Allow ARC to remotely trigger audio and voice use cases on the embedded device

AudioReach graph services, as depicted in Figure 6 below, include the following four cross-platform software components:

* Graph Service Library (GSL)

* ACDB Manager Library (AML)

* Generic Packet Router (GPR)

* Audio Tuning Service (ATS)

These components invoke utilities and obtain platform specific information provided by OS abstraction layer (OSAL) and platform wrapper in order to operate on the intended platform.

GSL serves as the entry point for a client to access functionalities provided by graph service and ARE. Through GSL APIs, a client can initiate graph setup, exchange audio data, and operate on the audio graph running on ARE. GSL queries AML using graph key vectors (GKV), calibration key vectors (CKV), and tag key vectors (TKV), which are passed by the client to retrieve graph definition, calibration, and tagged payloads. Then, GSL packetizes retrieved data from AML and invokes GPR to send commands and receive response from ARE. If modules running in the DSP can generate events, such as key word detection, GSL would relay the detection event back to client.

Apart from providing service to client, ARGs also plays a critical role in audio system development workflow. Audio tuning service works as bridge between ARC on host PC and rest of AudioReach software running on target. ATS dispatches commands from ARC to various service handlers which interfaces with rest of AudioReach software components to achieve development workflow use cases described in introduction section.
<figure><img src="image_39.png" /><figcaption>Figure 6: ARGs high-level architecture</figcaption></figure>

## 8. GENERIC PACKET ROUTER

The GPR infrastructure provides a common set of APIs across different platforms, software layers, and processor domains to enable them to communicate using asynchronous messages. The infrastructure is a memory optimized, platform agnostic layer. Figure 7 illustrates an example of the GPR infrastructure on different processor domains such as the application processor domain, DSP domain, and different OS platform such as Linux.

The GPR is responsible for routing messages and commands between the ARE and GSL processing domains in the AudioReach framework. It achieves this remote communication by abstracting out the underlying inter-processor communication (IPC) data link or transport layers, obviating need for system integrators implement platform specific solutions.

<figure><img src="image_40.png" /><figcaption>Figure 7: GPR on different processing domains</figcaption></figure>

## 9. AUDIOREACH FOR LINUX

AudioReach offers three architecture options for Linux as depicted in Figure 8. Option 1 follows the conventional architecture which Qualcomm Technologies has upstreamed its Au-

<figure><img src="image_41.png" /><figcaption>Figure 8: Linux support</figcaption></figure>

dioReach ALSA drivers into the Linux kernel under sound/soc/q-com/qdsp6. These drivers utilize ASOC topology support to give flexibility of building different use case topologies in DSP. Please note that option 1 supports limited use cases and is mainly targeted for developers who are accustomed to kernel sound card architecture.

Option 2 adheres to a conventional Linux audio userspace stack, in which PulseAudio or Pipewire operates on top of libALSA. Below this layer, the AudioReach SDK includes plugins that interact with ARGs residing in userspace instead of going into kernel sound card. The advantage of the AudioReach solution lies in its ability to offer a consistent development environment and software constructs, irrespective of the hardware and operating system platforms being targeted. By constructing this userspace architecture, it preserves the ecosystem API, thereby eliminating the necessity for existing applications to adapt to a new set of APIs, while simultaneously offering an enhanced feature set provided by AudioReach.

In cases where a product has concurrent use cases requiring dynamic reconfiguration as use cases are enabled and disabled, option 3 provides full stack support from PulseAudio/Pipewire and interfaces with the platform adaptation layer (PAL), utilizing plugins to manage a wide variety of use cases and sound devices efficiently.

To deploy option 2 or 3, one can find drivers and plugins for ALSA, PulseAudio within the AudioReach project while Pipewire plugins support is coming soon. We plan to engage with relevant ecosystem project communities to upstream these components. Our vision is to establish the AudioReach user-mode architecture as a viable alternative to the kernel architecture and integrate it directly into standard distributions.

## 10. CONCLUSIONS

The AudioReach SDK offers a solution for developing audio and voice applications across various platforms and devices. With the recent open-sourcing of AudioReach, the community now has access to a toolkit for creating feature-rich audio products. Though there is long road ahead to achieve envisioned development workflow, we hope audio developers like the project roadmap and can join the AudioReach community and contribute to its development. Please refer to AudioReach documentation site[2] to learn more about the project and SDK.

The Linux Foundation and The Linux Foundation logo design are registered trademarks of The Linux Foundation. Linux is a registered trademark of Linus Torvalds.
Qualcomm branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries.

Qualcomm Developer Network and Qualcomm Linux are programs of Qualcomm Technologies, Inc. and/or its subsidiaries.

Qualcomm is a trademark or registered trademark of Qualcomm Incorporated.

## 11. ACKNOWLEDGMENTS

Many thanks to my lead, Raymond Smith, and my colleague, Viktor Nikolic, for their contribution to this paper

Many thanks to my lead, Raymond Smith, for giving me the opportunity to start/lead AudioReach open-source project

Many thanks to Aditya Rathi, Annemarie Porter, Robin Gupta, Rohit Kumar who help me build up the project

Many thanks to all Qualcomm technologies audio developers who contribute to making of AudioReach SDK

## 12. REFERENCES

[1] "AudioReach", http://github.com/audioreach

[2] "AudioReach documentation", http://audioreach.github.io

[3] "PluseAudio Community", "PulseAudio", http://pulseaudio.org

[4] "Pipewire Community", "Pipewire", http://pipewire.org
# BRIDGING MUSICAL GESTURE AND COMPUTATIONAL REPRESENTATION:
DESIGNING GESTURE COMPOSITION FRAMEWORK

Marco Matteo Markidis

Independent Researcher

Parma, Italy

mm.markidis@gmail.com

## ABSTRACT

By designing the Gesture Composition Framework (GCF), the author aims to introduce a novel computer-based composition framework that treats musical gestures as first-class citizens in the compositional process, enabling composers to work at a higher level of abstraction than traditional signal-processing approaches. Unlike traditional frameworks that focus on synthesis parameters or signal flow, GCF represents gestures as complete musical ideas with temporal evolution, enabling a more intuitive mapping between musical intention and computational representation. This paper presents an embedded domain-specific language (EDSL) in C++ that leverages modern language features to provide both expressive power and compile-time safety. The framework's key innovations include a gesture-centric approach to composition, rich parameter evolution trajectories, pattern-based temporal structuring, and strict type validation through C++ concepts. Through quantitative analysis, we demonstrate how GCF maintains a significantly higher Conceptual Mapping Ratio indicating a closer alignment between musical concepts and implementation elements. These preliminary results confirm that GCF effectively has the potential to bridge the gap between traditional musical thinking and computational implementation, offering composers a powerful new tool that combines intuitive expression with technical precision.

## 1. INTRODUCTION

### 1.1. Computer-Based Composition Approaches

Electronic music composition has evolved through multiple paradigms since the early experiments with tape music in the 1950s. Contemporary approaches to scoring and composing computer-based electronic music can be broadly categorized into four main paradigms:

1. Text-based languages (CSound, SuperCollider, OpenMusic) that offer precise control through programming constructs but often require composers to think in terms of low-level synthesis parameters.

2. Visual programming environments (Pure Data, Max/MSP) that provide intuitive interfaces through graphical patching but may become unwieldy for complex compositional design.

3. Traditional notation extensions (Finale with NotePerformer, Bach library for Max) that augment conventional music notation with electronic music symbols, which maintain familiarity but struggle to represent the full complexity of lower-level processes (sound synthesis).

4. Digital Audio Workstations (DAWs) that offer timeline-based editing and parameter automation but tend to enforce a track-based mentality that may not align with more experimental compositional approaches.

Each approach offers distinct advantages but also imposes specific constraints. Text-based languages provide precision and

algorithmic capabilities but often hamper intuitive musical expression. Visual environments offer immediacy and experimental freedom but may lack rigorous structural validation. Traditional notation maintains connection with classical training but struggles to represent timbres and transformations. DAWs excel at production but may constrain compositional thinking to conventional structures.

### 1.2. Motivation and Scope

It is important to distinguish between synthesis frameworks and composition frameworks in computer-based music creation. Synthesis frameworks primarily focus on sound generation techniques, signal processing algorithms, and audio parameter manipulation. Composition frameworks, by contrast, emphasize the organization and structural representation of musical ideas, providing abstractions for expressing temporal relationships and musical intent. While GCF includes synthesis capabilities through audio nodes, its primary contribution lies in compositional representation and organization rather than sound generation techniques.

In the following paper, a 'gesture' refers to the smallest meaningful unit of musical expression from a compositional perspective—a computational primitive that encapsulates musical intent with temporal evolution<fnref n="1" />. This definition focuses on the compositional representation of complete musical ideas rather than physical movements or performance actions. A gesture in our framework represents a unified concept that includes duration, parameter trajectories, and relationships to sound-producing entities, enabling composers to work with musically meaningful units rather than individual synthesis parameters.

This paper introduces the Gesture Composition Framework (GCF), a computer-based composition framework designed to bridge the gap between musical thinking and computational implementation.

Current computer-based composition environments excel in their respective domains but present challenges when composers need to work at higher levels of abstraction. GCF treats musical gestures as first-class citizens in the compositional process, providing abstractions for composers who conceptualize music in terms of gestural units with temporal evolution. Also, GCF enables these composers to express temporal musical relationships through gesture-centric organization.

GCF serves as a computer-based composition framework, emphasizing the computational foundation while maintaining applicability beyond traditional computer music contexts. The framework's design accommodates various output modalities—from direct audio synthesis to potential integration with notation systems or control data for traditional instruments.

### 1.3. Paper Outline

The remainder of this paper reviews related work in computer-based composition tools and embedded DSLs (Section 2), pre-

<footnote n="1">¹This computational definition resonates with F. Nicolas's conception of gesture as "a particular kind of musical moment, which is generally clearly identifiable... because it contains a form and a cut-out that are obvious both on reading the score and on listening" (cited in [6], Appendix B.4.a, translated by Cadoz & Wanderley).</footnote>
sents the conceptual framework underlying GCF (Section 3), details the embedded DSL design and implementation (Section 4), evaluates the framework through quantitative analysis (Section 5), and concludes with contributions and future directions (Section 6).

# 2. RELATED WORK

## 2.1. Current Computer-Based Composition Systems

### 2.1.1. *Pure Data and Max/MSP*

Pure Data (Pd) and its commercial counterpart Max/MSP represent the paradigm of visual programming for electronic music. Developed by Miller Puckette [1], these environments allow composers, RIM and sonic artists to create music through graphical "patches" of interconnected objects. The visual nature provides immediate feedback and an intuitive understanding of signal flow, making these tools popular for both composition and live performance.

Puckette himself acknowledges in [2] that these visual programming environments were primarily designed as real-time scheduling and interconnection systems rather than comprehensive composition tools. He notes that Max paradigm environments are effective for connections and interconnectivity, and they focus on processes rather than data. Additionally, they do not offer any native concept of event. These limitations directly impact the expression of complex musical gestures, which often require sophisticated transformations of parameters over time.

### 2.1.2. CSound and Text-based Languages

CSound, developed by Barry Vercoe at MIT [3], exemplifies the text-based approach to electronic music composition, separating instrument definitions (orchestra) from event sequences (score). SuperCollider [4] extends this with a real-time, object-oriented approach.

These systems offer precision and algorithmic capabilities but often require composers to think in terms of technical implementation rather than musical gestures. As Lansky [5] observed, "the composer must become fluent in both the language of music and the language of the computer," creating a dual cognitive load with a consequent inherited loss of time. While excelling at sound synthesis definition, these systems provide less support for expressing complex parameter evolutions and temporal structures as unified entities.

## 2.2. Prior Academic Work on Gesture-based Composition

The concept of musical gesture has received considerable attention in music research. Cadoz and Wanderley [[6] compare various definitions of gesture, where some of these refer to musical gesture as "a temporal process where movements reference both action and meaning," highlighting its foundational role in musical expression.

Several systems have attempted to address gesture in electronic music: OM-Chroma [7] and DISSCO [8] both introduced hierarchical approaches to composition that include gesture-like constructs. OM-Chroma integrates sound synthesis with compositional structures, while DISSCO uses a hierarchical organization. However, both systems maintain separation between their gesture concepts and the actual synthesis process, creating workflow discontinuities.

Baalman [9] proposed WFSCollider for spatial sound control, but its gesture concept primarily addresses performance rather than composition structure.

While these contributions recognize the importance of gesture, they typically treat it as an additional layer on top of existing paradigms rather than as the fundamental organizing principle of the composition environment.

## 2.3. Embedded Domain-Specific Languages

Embedded Domain-Specific Languages (EDSLs) combine the expressivity of specialized languages with the tool support of host languages. In the music domain, ChucK [10] provides a specialized language for audio programming but lacks C++'s compile-time guarantees. HMusic [11] leverages Haskell's functional paradigm<fnref n="2" /> for musical patterns and multi-track compositions. The potential of modern C++ features for EDSL design has been less explored in music, despite the language's capabilities for combining expressive syntax with compile-time guarantees.

## 2.4. Gap Analysis

The review of existing systems reveals several gaps that the Gesture Composition Framework addresses:

1. **Integration of Gesture Concept:** While the importance of gesture is recognized in music research, existing tools do not fully integrate it as the fundamental organizing principle for composition.

2. **Compile-time Safety:** Most music programming environments defer validation to runtime, missing opportunities for early error detection that modern C++ can provide.

3. **Rich Parameter Evolution:** Existing systems support parameter automation but lack integrated tools for defining complex, interdependent parameter trajectories as unified musical gestures.

4. **Balance Between Expression and Rigor:** Current tools tend to favor either intuitive expression (at the cost of technical rigor) or technical precision (at the cost of intuitive expression), with few successfully balancing both aspects.

5. **Temporal Structuring:** Complex temporal structures like *accelerando* or *precipitando* often require verbose implementations, lacking concise abstractions for these common musical patterns.

The Gesture Composition Framework addresses these gaps by treating gesture as the atomic unit of musical expression, leveraging C++ concepts for compile-time validation, providing rich parameter evolution capabilities, and embedding a domain-specific language that balances expressive power with technical rigor.

# 3. CONCEPTUAL FRAMEWORK

## 3.1. Core Components

### 3.1.1. *Gesture as the Atomic Unit*

At the heart of the Gesture Composition Framework (GCF) lies the concept of gesture as the fundamental unit of musical expression. Unlike traditional approaches that focus on individual samples or pure synthesis parameters, a gesture in the GCF represents a complete musical idea with temporal evolution. Gesture defining features include:

1. Duration

2. Start Time

3. **Parameters:** A set of evolving values that define the gesture's characteristics.

4. **Processable:** A reference to the sound-producing constrained class that realizes the gesture.

5. **Pattern:** An optional rhythmic or structural pattern that shapes the gesture.

<footnote n="2">²Haskell is a purely functional programming language known for its strong static typing and elegant handling of abstractions.</footnote>
This approach aligns with how composers naturally think
about musical ideas, allowing for a more intuitive mapping be-
tween musical intention and computational representation.

The current implementation of GCF focuses on mono-channel
audio output, with spatial distribution capabilities planned for
future development (see Section 6.2.1). This initial constraint
allows for a focused exploration of temporal and parameter evo-
lution aspects of gestures before extending to spatial dimensions.

3.1.2. Parameter Evolution

Parameters in the GCF are not static values but evolving trajec-
tories. This evolution can follow various patterns, including:

**Constant** Maintaining a single value throughout the gesture

**Linear** Smoothly transitioning between start and end values

**Quadratic** Following a quadratic curve for more dynamic evolution

**Custom** User-defined functions for complex parameter trajectories

This rich representation of parameter evolution allows com-
posers to create dynamic, expressive gestures that evolve natu-
rally over time.

3.1.3. Processable Concept

The GCF uses C++ concepts<fnref n="3" /> to define the constraints for sound-
producing classes (instruments). Any class that satisfies the Pro-
cessable concept can be used as an instrument in the framework.
This design decision deliberately favors generic programming
based on templates and concepts rather than adopting an object-
oriented approach, even with static polymorphism. This choice
prioritizes compile-time safety and optimization opportunities
while avoiding the runtime overhead and complexity of virtual
method dispatch, resulting in both more expressive code and bet-
ter performance. This approach provides:

1. Type Safety: Compile-time validation of instrument compatibility.

2. Extensibility: Easy integration of new instrument types.

3. Clarity: Clear requirements for instrument implementation.

The *Processable* concept requires:

* A static `getParameters()` method that returns parameter constraints.

* A `process()` method that produces audio samples based on current parameters.

This approach allows for a flexible, yet type-safe, integration
of various sound synthesis techniques.

3.1.4. Pattern Structure

The *Pattern* component provides a way to structure gestures ac-
cording to rhythmic or structural patterns. Patterns consist of:

**Sound Elements** Periods where the instrument is active.

**Silence Elements** Periods where the instrument is inactive.

**Time Unit** A scaling factor for the pattern elements.

**Transformation Rules** Functions that can modify the pattern over time.

This pattern structure allows for complex rhythmic gestures like *accelerando*, *ritardando*, or *precipitando*, which are challenging to implement in traditional computer music frameworks.

3.1.5. Score and Director

The *Score* component serves as a container for multiple gestures, allowing for:

* Sequential or overlapping arrangement of gestures.

* Validation of gesture relationships.

* Retrieval of gestures in time order.

The *Director* component handles the rendering of scores to
audio files by:

* Orchestrating the execution of gestures.

* Mixing overlapping gestures.

* Managing the overall rendering process.

3.2. Component Interactions

Figure 1 illustrates the relationships between GCF's core com-
ponents. The interactions form a cohesive system for expressing
musical ideas through:

* **Gesture Creation:** Composers define gestures using the EDSL, specifying duration, parameters, and patterns.

* **Parameter Evolution:** Parameter components manage value trajectories over time.

* **Instrument Processing:** Processable-compliant instruments respond to parameter changes, producing audio.

* **Pattern Application:** Pattern components structure gestures' temporal aspects.

* **Score Assembly & Rendering:** Score organizes gestures into compositions, while Director renders to audio.

This architecture enables direct expression of musical concepts as computational entities while maintaining type safety and compositional flexibility.

3.3. Design Principles

The design of GCF's conceptual framework follows several key
principles that distinguish it as a computer-based composition
framework:

**Gesture-Centric Organization** The framework treats gestures as the fundamental unit of musical expression, encapsulating both temporal evolution and parameter relationships within a single conceptual entity. This approach matches how composers can conceptualize musical ideas while providing computational precision.

**Temporal Abstraction** GCF abstracts the representation of time-based musical relationships through pattern structures and parameter evolution trajectories. This allows composers to express complex temporal concepts (*accelerando*, *precipitando*, gradual parameter changes) as unified compositional elements rather than sequences of discrete events.

**Type-Safe Composition** Through C++ concepts and template metaprogramming, the framework ensures compositional validity at compile time. This provides immediate feedback about structural inconsistencies and parameter mismatches, reducing the debugging overhead common in computer music programming.

**Modular Flexibility** The separation between gestures (compositional units) and instruments (sound-producing entities) allows composers to focus on musical structure independently of specific synthesis techniques, while maintaining the flexibility to integrate various sound generation approaches.

These principles collectively address the challenge of maintain-
ing conceptual clarity while working within computational con-
straints, enabling composers to focus on musical decisions rather
than implementation details.

<footnote n="3">³A Concept is a named set of requirements for template parameters introduced in C++20. Concepts allow for more expressive template constraints, clearer error messages, and better IDE support compared to traditional SFINAE techniques. They provide compile-time validation of template parameters, ensuring that only types meeting specific requirements can be used with a given template.</footnote>
<figure><img src="image_42.png" /><figcaption>Figure 1: GCF Component Diagram illustrating relationships between core framework components</figcaption></figure>
# 4. EDSL DESIGN

The Gesture Composition Framework (GCF) features a carefully designed Embedded Domain-Specific Language (EDSL) that bridges the gap between musical thinking and computational implementation. This section explores the design and implementation of the EDSL, highlighting how it enables intuitive and expressive musical composition while maintaining the type safety and performance benefits of C++.

## 4.1. Design Goals and Language Specifications

The development of the GCF’s EDSL was guided by several key design goals:

1. Readability: The syntax should be clear and intuitive, reflecting how some composers think about musical gestures.

2. Type Safety: The EDSL should leverage C++'s type system to catch errors at compile time.

3. Expressivity: The language should allow for complex musical expressions without becoming verbose or convoluted.

4. Extensibility: The design should be open to extension without requiring changes to the core framework.

5. Error Handling: Error messages should be clear and point directly to the origin of the problem.

## 4.2. Syntax Evolution

The EDSL syntax evolved through multiple iterations, with various approaches considered and evaluated. The final design adopts a method chaining approach with a builder pattern<fnref n="4" />, which offers an intuitive and flexible syntax while maintaining type safety.

```cpp
const auto gesture = Gesture<SinOsc>()
    .starting(0) // Start time in seconds
    .lasting(2) // Duration in seconds
    .having("amplitude", 0.7) // Constant amplitude
    .having("frequency", 440, 880, EvolutionType::Linear) // Frequency sweep
    .create();
```

This proposed syntax can allow composers to express musical ideas in a way that closely mirrors how they think about gestures, with clear specification of start time, duration, and evolving parameters.

## 4.3. Builder Pattern Implementation

The EDSL is implemented using a builder pattern, which provides a fluent interface through method chaining while enabling incremental construction, state tracking, and deferred creation. The core implementation is the GestureBuilder class template:

```cpp
template <Processable InstrumentT>
class GestureBuilder {
public:
    GestureBuilder &starting( double time );
    GestureBuilder &lasting( double duration );
    GestureBuilder &having( std::string_view name
                             , double start, double end = 0.0,
                             EvolutionType type = EvolutionType::
                                                    Constant);
    GestureBuilder &pattern( Pattern p );
    [[ nodiscard ]] std::unique_ptr< GestureImpl<
                                                InstrumentT >> create();
private:
    // State tracking and validation
};
```

This builder approach allows for clear error messages when required properties are missing or constraints are violated.

## 4.4. Parameter Evolution

A key feature of the EDSL is its support for parameter evolution, allowing parameters to change over time according to various trajectories:

```cpp
    .having("amplitude", 0.5, 1.0, EvolutionType::
              Linear) // Linear fade-in
    .having("frequency", 440, 880, EvolutionType::
              Quadratic) // Quadratic sweep
```

The implementation resembles the Strategy pattern to represent different evolution types:

```cpp
class Evolution {
public:
    Evolution( double start, double end,
               EvolutionType type)
        : start_(start), end_(end), type_(type) {}
    double evaluateAt( double normalizedTime)
        const {
            // Evaluate parameter at the given
            normalized time
            // according to evolution type (Constant
            , Linear, etc.)
            // ...
        }
private:
    double start_, end;
    EvolutionType type;
};
```

This approach allows for easy extension with new evolution types while maintaining a consistent interface.

## 4.5. Pattern Specification

The EDSL extends to pattern specification, allowing for complex rhythmic structures:

```cpp
auto pattern = make_pattern({ sound(1),
                                silence(1) })
    .time_unit(0.25)
    .transform(
        []( const PatternElement& elem, int iteration) {
            // Transform sound and silence durations
            // based on iteration
            // Create accelerating/precipitando effect
            // ...
        },
        until_duration_below(0.1));
```

This pattern syntax allows for the specification of complex patterns like precipitando, where silence durations progressively decrease.

## 4.6. Compile-Time Validation

The EDSL leverages C++ concepts to provide compile-time validation of instruments and parameters:

```cpp
template <typename T>
concept Processable = requires(T t) {
    T::getParameters() -> std::
        convertible_to < std::map < std::string,
                                ParameterInfo >>;
    t.process( std::vector < Parameter >{}, double
                {} ) -> std::convertible_to < double >;
};
```

This approach catches errors at compile time, providing clear error messages and preventing runtime failures.

## 4.7. Advanced Features

The EDSL includes several advanced features that enhance its expressivity:

* Custom Parameter Functions:

<footnote n="4">⁴The Builder Pattern is a creational design pattern that separates the construction of complex objects from their representation, enabling the same construction process to produce different representations.</footnote>
`.having("frequency", [](double t) { return 440 + 440 * std::sin(2 * M_PI * t); })`

* Pattern Duration Inference:

`.lasting pattern_duration)`

* Pattern Transformations:

`.transform (transformFunction, terminationCondition)`

These capabilities allow complex gesture expressions while maintaining a consistent interface.

## 4.8. Language Features Summary

The EDSL design incorporates several language features specifically chosen to support gesture-based composition:

**Template-Based Flexibility** The use of C++ templates enables type-safe composition while maintaining flexibility in parameter types and value ranges. This allows the same gestural concepts to work with different numerical precisions and parameter scales without sacrificing compile-time validation.

**Concept-Driven Validation** C++ concepts ensure that only valid combinations of gestures, patterns, and instruments can be compiled, moving error detection from runtime to compile time. This approach reduces debugging complexity and provides immediate feedback about compositional structure.

**Functional Composition Operators** The framework provides operators that enable functional composition of gestures and patterns, allowing composers to build complex temporal structures from simpler components. This compositional approach mirrors musical thinking about structural relationships.

**Memory Management Integration** Automatic resource management through RAII and smart pointers ensures that composers can focus on musical structure without manual memory management concerns, reducing technical overhead in the composition process.

These language features collectively create an environment where musical concepts can be expressed directly in code while maintaining the precision and performance characteristics required for computer-based composition.

## 4.9. Embedded DSL for Gesture-Centric Composition

By combining a fluent, intuitive syntax with strong type safety and compile-time validation, the EDSL enables composers to express complex musical ideas clearly and concisely. The gesture-centric approach, combined with rich parameter evolution and pattern specification, provides expressive framework for computer-based music composition.

# 5. QUANTITATIVE ANALYSIS OF GESTURAL EXPRESSIVENESS

This section provides a quantitative evaluation of GCF's gestural expressiveness through the Conceptual Mapping Ratio (CMR) methodology and analysis of type safety benefits.

## 5.1. Conceptual Mapping Ratio (CMR) Methodology

To objectively assess GCF's expressiveness, we introduce the Conceptual Mapping Ratio (CMR), defined as:

**CMR = Musical Concepts / Implementation Elements**

Where higher CMR values indicate more efficient expression of musical ideas relative to implementation complexity.

### 5.1.1. Musical Concepts Definition

**Musical concepts** represent distinct musical ideas that a composer conceptualizes when creating a gesture, independent of implementation details. These include:

1. Temporal Organization: Start time and duration specifications

2. Parameter Definitions: Distinct musical parameters (frequency, amplitude, etc.)

3. Evolution Behaviors: Temporal trajectories for dynamic parameters

4. Instrumental Choice: Selection of sound-producing entities

### 5.1.2. Implementation Elements Definition

**Implementation elements** represent distinct method calls or code constructs required to express musical concepts in GCF:

1. Gesture Construction: Temporal methods (`.starting()`, `.lasting()` and boilerplate method (`.create()`)

2. Parameter Specification: Each `.having()` method call

3. Instrument Selection: Template parameter specification

4. Framework-Specific Constructs: Pattern definitions

### 5.1.3. Methodological Considerations

Our analysis treats temporal evolution as a native framework capability. Since GCF is designed with time as a first-class concept, parameter evolution is considered an intrinsic feature rather than additional implementation burden. This reflects the framework's design philosophy where `.having("amplitude", 0.5, 1.0, EvolutionType::Linear)` represents a single unified operation for temporal parameter control.

## 5.2. Quantitative Analysis Results

We analyze four representative examples that demonstrate different aspects of GCF's gestural capabilities. All examples presented in this analysis are implemented and available as unit tests within the framework repository, ensuring reproducibility and providing practical reference implementations for researchers and composers.

### 5.2.1. Example 1: Simple Sustained Note

```
Gesture<SinOsc>()
  .starting(0)
  .lasting(2)
  .having("amplitude", mf)
  .having("frequency", 440)
  .create();
```

The gesture requires five distinct musical decisions, while the code implementation consists of six distinct elements.

### 5.2.2. Example 2: Complex Parameter Evolution

```
Gesture<SinOsc>()
  .starting(0)
  .lasting(2)
  .having("amplitude", 0.5, 1.0, EvolutionType::Linear)
  .having("frequency", 440, 880, EvolutionType::Quadratic)
  .create();
```

The gesture involves five musical decisions, and the implementation requires six code elements.
### 5.2.3. Example 3: Ring Modulation Instrument

```cpp
Gesture<RingModulator>(
  .starting(0)
  .lasting(2)
  .having("carrier.amplitude", 1.0)
  .having("carrier-frequency", 1.0)
  .having("modulator.amplitude", 0.5, 1.0,
    EvolutionType::Linear)
  .having("modulatorfrequency", 440, 880,
    EvolutionType::Quadratic)
  .create();
```

The gesture encompasses seven musical decisions, while the implementation consists of eight code elements.

### 5.2.4. Example 4: Pattern-based Gesture

```cpp
Gesture<ConstantNode>(
  .starting(0)
  .lasting pattern_duration)
  .pattern (make_pattern({sound(1), silence(1)}))
  .time_unit(0.25)
  .transform(emer const PatternElement& elem, int
    iteration) {
    // precipitando logic
  }, until_durationBelow(0.1)))
  .having("amplitude", 0.7)
  .create();
```

The gesture requires eight distinct musical decisions, and the implementation involves eight code elements.

## 5.3. Analysis and Interpretation

The four examples yield CMR values ranging from 0.83 to 1.01.

### 5.3.1. Key Design Achievement: Complexity Invariance

**Critical Observation:** Examples 1 and 2 achieve identical CMR (0.83), demonstrating that GCF maintains constant expressiveness regardless of parameter evolution complexity. This validates the framework's core design goal: making temporal evolution a zero-cost abstraction from the composer's perspective. A composer can transform any static parameter into a sophisticated dynamic evolution without increasing the conceptual-to-implementation burden, proving that temporal behavior is truly a first-class framework feature.

### 5.3.2. Type Safety Benefits

GCF's compile-time validation provides comprehensive error detection. The framework's parameter validation system detects range violations such as negative frequency values or amplitude parameters exceeding 1.0, while simultaneously ensuring that all required parameters for specific instruments are properly specified and that parameter names correspond to valid instrument interfaces. The structural validation capabilities extend beyond individual parameters to encompass the entire gesture architecture, detecting duplicate method calls that would create ambiguous temporal specifications, identifying invalid connections within InstrumentDAG topologies that could lead to signal flow inconsistencies, and performing cycle detection in signal flow graphs to prevent infinite feedback loops that would render the composition inoperable. Temporal validation represents a particularly sophisticated aspect of GCF's compile-time checking, identifying pattern inconsistencies such as empty pattern definitions or invalid time unit specifications, detecting mismatches in evolution parameter configurations that could result in undefined temporal behaviors, and validating duration and timing constraints to ensure temporal coherence across the entire gestural structure.

### 5.3.3. Framework Expressiveness Characteristics

**High Baseline Efficiency** All examples achieve CMR ≥ 0.83, indicating strong expressiveness even with framework overhead.

**Abstraction Benefits** More sophisticated abstractions (patterns, instruments) achieve higher CMR values, demonstrating that complexity encapsulation improves expressiveness.

**Predictable Cognitive Load** Consistent CMR values within example categories suggest predictable learning curves for composers adopting different framework features.

**Scalability** The framework maintains high expressiveness as musical complexity increases, validating its suitability for sophisticated compositional applications.

## 5.4. Discussion

The quantitative analysis reveals several key insights about the gesture-centric approach implemented in GCF compared to traditional synthesis frameworks. These findings have significant implications for both the theoretical understanding of computer music composition and practical tool development.

First, the consistently high Conceptual Mapping Ratio in GCF demonstrates that a gesture-centric approach can effectively bridge the gap between musical intention and computational implementation. This alignment supports the core hypothesis that treating gestures as first-class citizens in the compositional process could lead to more intuitive expression of gestural based musical ideas.

Second, the compile-time safety provided by GCF represents a significant advancement. By leveraging C++ concepts for instrument and parameter validation, GCF shifts error detection from runtime to compile time, potentially eliminating entire categories of compositional errors before they manifest in performance. This safety aspect is particularly valuable in listening sections where runtime errors can be catastrophic.

Third, the consistent interface for parameter evolution in GCF—regardless of trajectory complexity—reduces cognitive load for composers. GCF's uniform methods for specifying linear, quadratic, or custom parameter changes enable composers to focus on the musical outcome rather than implementation details.

The pattern-based example particularly highlights GCF's advantage in expressing complex temporal structures, achieving optimal expressiveness (CMR = 1.0) when dealing with specialized temporal constructs.

It is important to note the context in which GCF was developed. Indeed, GCF provides a middle layer between lower-level C++ and compositional thinking. GCF was initially conceived during the development of *Campi Instabili*, an author's piece exploring granular synthesis and tetrahedral loudspeaker array<fnref n="5" />. This composition required sample-precision control over phase relationships between grains and their internal content—a level of control that motivated direct C++ implementation rather than existing frameworks. GCF emerged from this experience as a solution for composers who accept the overhead of C++ but desire a more intuitive gesture-centric API without rebuilding basic infrastructure for each new piece. In this niche, GCF would excel by allowing composers to focus on the gestural relationships while simplifying instruments and signal processing through clear parameter interfaces and compositional abstractions. Additionally, the author would re-implement *Campi Instabili* using GCF, to understand if the framework acts in a transparent way, as intermediate layer between composer and C++, or as a mediation layer, changing how the composer would behave in the absence of the framework.

<footnote n="5">⁵Spherical Tetrahedral One, as known as S.T.One, is an outer radiating tetrahedral loudspeaker array designed and developed by Giuseppe Silvi.</footnote>
<table><caption>Table 1: Comparative CMR for different Examples</caption><thead><tr><td>Example</td><td>CMR</td><td>Insight</td></tr></thead><tbody><tr><td>Simple Sustained Note</td><td>0.83</td><td>Boilerplate overhead in basic cases</td></tr><tr><td>Complex Parameter Evolution</td><td>0.83</td><td>Complexity invariance achieved</td></tr><tr><td>Ring Modulation</td><td>0.88</td><td>Instrument abstraction benefits</td></tr><tr><td>Pattern-based Gesture</td><td>1.0</td><td>Optimal temporal abstraction</td></tr></tbody></table>

These findings suggest that future computer-based composition frameworks might benefit from considering CMR as a leading criterion for framework design and development, prioritizing conceptual alignment.

# 6. CONCLUSION AND FUTURE WORK

## 6.1. Conclusion

The Gesture Composition Framework is presented as an ongoing research project. The current implementation serves as a proof of concept that demonstrates the potential of a gesture-centric approach to computer-based composition framework. Indeed, it represents a significant shift in computer-based composition framework by elevating gestures to first-class citizens. Our quantitative analysis demonstrates that GCF could offer substantial benefits through higher conceptual alignment between musical intention and computational implementation.

The consistently high Conceptual Mapping Ratio across examples confirms our core hypothesis—that a gesture-centric approach effectively bridges the gap between musical thinking and technical implementation. This alignment is further strengthened by GCF’s compile-time safety through C++ concepts, eliminating categories of errors before they manifest in performance.

Additionally, GCF’s consistent CMR values suggest predictable cognitive load, which could result in predictable learning curves for composers adopting the framework.

## 6.2. Future Work

Several promising directions for future development include:

### 6.2.1. Spatial Distribution and Pattern Extensions

Expanding beyond mono-channel audio with gesture-level spatial relationships; enhancing pattern transformation capabilities for complex temporal structures including Markov-based generation and polyrhythmic layering.

### 6.2.2. Gesture Transformation and State Management

Introducing transformational rules to derive new gestures from existing ones, enabling compositional strategies like theme and variations; implementing instrument state management for gestural continuity across compositional sections.

### 6.2.3. Performance Optimization and Integration with Existing Unit Generator Libraries

Adding real-time capabilities through audio API integration and multi-threading optimizations; creating adaptors for established unit generator libraries such as CSound, SuperCollider’s UGens, and the Synthesis ToolKit (STK), and Faust to leverage signal processing capabilities.

## 6.3. Code Availability

The complete source code for the Gesture Composition Framework<fnref n="6" /> is available as an open-source project under the GPL3 license. The repository is hosted on the author’s personal GitLab

instance; readers interested in obtaining an account may contact the author directly.

As an independent researcher who has learned extensively from open-source software, the author considers maintaining this project as open source an essential contribution to the computer music research community. The open-source nature of GCF facilitates reproducibility of research results, enables collaborative development, and supports educational applications in computer music composition.

# 7. ACKNOWLEDGMENTS

I would like to thank Davide Tedesco for the insightful conversations and thorough review, Francesco Roberto Dani for his unwavering support throughout this project, Alessandro Benedetti for his review, and Agostino Di Scipio for his valuable guidance and mentorship. Additionally, I extend my gratitude to the anonymous reviewers whose feedback significantly improved this paper.

# 8. REFERENCES

[1] Miller Puckette et al., “Pure data: another integrated computer music environment,” *Proceedings of the second international computer music concerts*, pp. 37–41, 1996.

[2] Miller Puckette, “Max at seventeen,” *Computer Music Journal*, vol. 26, no. 4, pp. 31–43, 2002.

[3] Barry Vercoe et al., “Csound,” *The CSound Manual Version*, vol. 3, 1986.

[4] James McCartney, “Rethinking the computer music language: Super collider,” *Computer Music Journal*, vol. 26, no. 4, pp. 61–68, 2002.

[5] Paul Lansky, “A view from the bus: when machines make music,” *Perspectives of new music*, pp. 102–110, 1990.

[6] Claude Cadoz and Marcelo Mortensen Wanderley, “Gesture-music,” *Trends in gestural control of music*, 2000.

[7] Carlos Agon, Jean Bresson, and Marco Stroppa, “Omnchroma: Compositional control of sound synthesis,” *Computer Music Journal*, vol. 35, no. 2, pp. 67–83, 2011.

[8] Hans G Kaper, Sever Tipei, and Jeff M Wright, “Dissco: A unified approach to sound synthesis and composition,” in *ICMC*, 2005.

[9] Marije AJ Baalman, “Spatial composition techniques and sound spatialisation technologies,” *Organised Sound*, vol. 15, no. 3, pp. 209–218, 2010.

[10] Ge Wang, Perry R Cook, and Spencer Salazar, “Chuck: A strongly timed computer music language,” *Computer Music Journal*, vol. 39, no. 4, pp. 10–29, 2015.

[11] Andre Rauber Du Bois and Rodrigo Geraldo Ribeiro, “Hmusic: A domain specific language for music programming and live coding.,” in *NIME*, 2019, pp. 381–386.

<footnote n="6">⁶markidis.space/gesture-composition-framework/gcf</footnote>
LAC-25 Paper Session 3
# BUGUI:
A WIDGET SYSTEM FOR BUTTON GRID CONTROLLERS

Thibaud Keller

Ossia team
Music Dept Maynooth, Ireland

thibaud.keller.2025@mumail.ie

Gordon Delap

Music Dept Maynooth, Ireland

gordon.delap@mu.ie

Victor Lazzarini

Music Dept Maynooth, Ireland

victor.lazzarini@mu.ie

Jean-Michel Celerier

Ossia team
SAT Montreal, Canada

jmcelerier@sat.qc.ca

## ABSTRACT

BuGUI is a library intended to facilitate user interface design on creative hardware, especially button grid controllers. Our approach is to consider these devices as very low resolution touch screens, with some additional controls potentially (buttons, encoders, actual screens). To that effect, the library is structured as a simplistic yet general-purpose UI system.

This effort to keep our implementation generic makes it easily extensible and eventually capable of covering a wider range of both hardware interfaces and software use cases.

This paper details our current, early-stage development, on which we base a plugin for the *ossia score* software sequencer. As we aim to provide a full-featured hardware interface for score, this plugin also offers BuGUI an extensive case study.

## 1. INTRODUCTION

BuGUI<fnref n="1" /> acts as an abstraction layer between software and hardware. Our motivation stems from observing the wide variety of uses for grid controllers. Unfortunately, these creative uses, in most cases, remain specific to a particular piece of software. Whether it is in the form of standalone, fully integrated systems, or desktop applications compatible with external controllers, most projects reimplement their own hardware interactions. This approach not only limits hardware compatibility, as each project must maintain their own catalogue of available controllers, it also duplicates software implementations of similar use cases.

We propose a generic approach to hardware interfaces, inspired by traditional step-sequencer affordances, where these devices are essentially considered in the same way that screen, mouse and keyboard are in regular GUI development.

### 1.1. Ossia score

*ossia score* [1] is an interactive sequencer, based on a branching temporal model. Durations are defined in independent intervals that can play sequentially or concurrently, and otherwise overlap freely. These intervals can also be set to a fixed duration, or a relative one, depending on user-defined conditions.

Scenarios in *ossia score* are defined with a set of generic temporal elements, summarised by the following list:

* *States*: a single point in time

* *Events*: a group of simultaneous *States*

* *Syncs*: a group of simultaneous *Events*

* *Transitions*: instantaneous jump between two distinct *Syncs*

* *Intervals*: flexible duration connecting two distinct *Syncs*

* *Scenarios*: spanning the duration of an *Interval*, it implements recursion and ordering

Furthermore, the user defined conditions that make these elements interactive take one of two forms:

* *Triggers*: extends or interrupts the predefined duration of *Intervals*

* *Conditions*: allows for selecting between parallel *Events* sharing the same *Sync*

For comparison, the set of temporal elements used in hardware sequencers could be defined as follows:

* *Steps*: a single point in time

* *Sequences*: a circular chain of *Steps*, spaced in multiples of a predefined musical subdivision

* *Patterns*: a group of concurrent *Sequences*

* *Songs*: a linear arrangement of *Patterns*

Temporal control requires additional elements, defining for example the direction of the sequence: forward, backward, mirrored (forwards and backwards alternatively), random, etc.

This list features a one-to-one relationship with hardware interfaces elements: either *Steps* or *Patterns* are represented with a single backlit button, and *Sequences* or *Songs* with one or several rows of backlit buttons.

Similarly, *ossia score* provides graphical representation of its temporal elements with dots (*States*), horizontal (*Intervals*) and vertical lines (*Events*, *Syncs*), along with simple icons (*Triggers*, *Conditions*).

The *ossia score* plugin for button grids we describe here proposes first a translation of the UI, simplified down to an arrangement of coloured cells, rows and columns. From our initial assessment, this hardware representation is enough to convey the structure of a scenario. Current developments focus on providing basic control of this structure with tactile inputs on the grid.

The representation and manipulation of interactive scenarios as defined in *ossia score* can then lead to the design of a unique grid interface, demonstrating further the versatility of these devices.

*ossia score*'s GUI also features many common place widgets like sliders, spin boxes, and a nodal patcher, that have been introduced on grid based products. A full-featured hardware interface for it, as we aim to provide with this plugin, would also offer a cohesive, open-source implementation of these widgets on hardware.

<footnote n="1">¹https://codeberg.org/bugui/bugui</footnote>
<figure><img src="image_43.png" /><figcaption>Figure 1: *Representation of a scenario on the Launchpad Pro. Syncs and Events displayed only when a child State is pressed*</figcaption></figure>

## 1.2. Similar projects

In 2017, Van Haaren presented OpenAV's Ctlra[2], a great project we initially considered for the purposes of *ossia score's* hardware integration. BuGUI shares Ctlra's objective of centralizing support for many devices in a single library, as well as abstracting away the communication protocols.

It is worth noting however that Ctlra and BuGUI mainly target different hardware. We aim to provide integration with common grid controllers that are, in most cases, simple MIDI devices. Although Ctlra can work with generic MIDI controllers, this aspect is still marked as "in progress, but not complete" in the code repository<fnref n="2" />. Its primary targets are "Modern Controllers", featuring large numbers of input controls, LED feedback, often high-resolution screens, and communicating over the HID protocol.

The "scripts" used by Ctlra to interface with a particular device are meant to be customizable, and potentially unique to each user, even on the same software. As we'll elaborate further, we took a more fixed approach and limited hardware specifications to an arguably simpler format.

The work of Hunt[3] is also closely related to our endeavour. His Universal Grid Sequencer (UGS) allows creating independent sequences executed simultaneously on one or more button grid controllers. It leverages hardware interactions for more tangible feedback of potentially complex temporal and rhythmical relationships.

Instrumental patterns created with the UGS framework can fall in and out of phase or synchronisation according to predefined numbers of steps and tempos. Such patterns are designed in two distinct stages. Before starting the system, multiple instances are set up in software, with hard-coded voices, steps and tempo. The content of each pattern can then be edited in real-time once the system is running.

Although very different from the more general purpose library we describe here, this work echos our main use case: hardware integration for the *ossia score* sequencer. Button grid controllers are the UGS's central focus, and we also note that the sequences rendered on the devices are defined and behave similarly to common widgets.

## 1.3. Why button grids ?

Button grids have become ubiquitous in digital live performance. More and more of these devices are released every year (see Figure 2 in [5]). Button grids are defined as backlit button matrices, that offer both individual tactile input and light feedback for each cell.

The interaction they afford is closely tied to the usage of hardware step sequencers and drum machines. However, various innovative implementation enable a greater variety of interfaces in different products. Here is a non-exhaustive list of available button grid interfaces:

* Keyboards / drum pads

* Step sequencers

* Discrete sliders

* "Clip" launchers (Ardour, Ableton Live)<fnref n="3" />

* Text displays (Novation Launchpad)

* Basic spin boxes (Novation Circuit)

* Arrangement views (Synthstrom Deluge)

* Routing matrixes (Synthstrom Deluge)

* Rudimentary waveform croppers (Synthstrom Deluge)

* Nodal patchers (Empress Zoia)

These interactions are in most cases specific to a particular hardware, or software.

A notable exception in this field is the company Monome<fnref n="4" /> that pioneered it. They offer an open-ended environment, technically compatible with a wider range of devices, but intended mainly to run on their Norns system, connected to their Grids controller. Over the years, Monome has garnered a dedicated community of artists and developers, due in part to their commitment to open-source.

Norns enable programming Lua scripts that encompass audio processing, graphical user interface, and hardware integration. As such, this system enables very involved controller interactions and graphical renderings, but does not seem to be suited for hardware integration in other software. For example, the port of the Orca sequencer to Norns has apparently required a full Lua rewrite<fnref n="5" />.

## 1.4. Existing literature

Button grid practices have been thoroughly documented in recent years at LMU Munich by Beat Rossmy[4], with Alexander Wiethoff[5], Maximilian Rauh[6] and Sebastian Unger[7]. A taxonomy of grid interaction was also published in 2021[8], consolidating further what we perceive as a strive for a certain standardisation of the practice. The research conducted at LMU also shows, among many insights, a clear upward trend in the release of devices featuring a button grid interface since the late nineties.

The Avendish project[9] is a major influence, as we attempt to mirror its declarative style for our hardware specification classes. Avendish leverages C++ reflection to create generic, framework-agnostic media processors and UI. Users list control parameters in C++ aggregates, and write their processes in basic functions, without any dependency or reference to the environment they will eventually be compiled for. The same generic C++ code can then make use of various wrappers to produce media processors for *ossia score*, Python, PureData and other environments. The naming scheme of these aggregates and functions suffices to retrieve all necessary information at compile-time.

<footnote n="2">²https://github.com/openAVproductions/openAV-Ctlra</footnote>

<footnote n="3">³Desktop applications integrated with external controllers</footnote>

<footnote n="4">⁴see Monome's website: https://monome.org</footnote>

<footnote n="5">⁵see the forum entry for the projecthttps://llllllll.co/t/orca</footnote>
This approach of abstracting away the particulars of the im-
plementation is also applied to the definition of custom graphical
user interfaces. Our use of this technique is currently limited, but
very relevant to our needs, as we will demonstrate.

## 2. IMPLEMENTATION

This section details BuGUI's chosen architecture, in its current, early stage development. We will present this bottom-up design, starting from hardware specifications, then moving to core functionality and finally describing the API.

### 2.1. General device parameters

We are focussing first on supporting the Novation Launchpad
Pro<fnref n="6" />, as it is a fairly well-known, common and full-featured con-
troller. We defined its specification in a simple C++ aggregate,
containing the following static functions:

```cpp
static consteval auto sysex_prefix()
{
    return std::array<uint8_t, 6>
        {240, 0, 32, 41, 2, 16};
}

static consteval uint8_t sysex_sufix()
{
    return 247; }

static consteval auto clear()
{
    return std::array<uint8_t, 2>{14, 0}; }

static consteval uint8_t light_prefix()
{
    return 10; }
```

These functions are marked consteval to ensure that the value they return is available at compile-time. For communication to the device, we elected to use system exclusive messages (Sys-Sex). These types of flexible MIDI messages can carry an unspecified number of data bytes. They allow concatenating several instructions together, reducing the number of messages sent to achieve the same result. Several features of the device are also only available through SysEx, allowing for example to turn off all LED feedback with a single message. We take advantage of this feature when defining the clear function, which returns the specific part of the SysEx message to call this functionality on the device. This pair of values can then be concatenated between the SysEx prefix and suffix, returned by their respective functions, to create the required message and clear the device once sent to the appropriate port.

We use SysEx messages in the same way to set the colour of
individual LEDs on the device. The same procedure of concaten-
ating values between SysEx prefix and suffix is employed. The
specific content of this message starts with the returned value of
the color_prefix function, followed by the identifier of the cho-
sen button, and its desired colour. The full message takes this
form<fnref n="7" />:

240,0,32,41,2,16,10,<LED>,<Colour>,247

Additional information about the device are defined in nested
aggregates as follows.

### 2.2. Identifiers

At present, we only defined a few buttons of interest for the
Launchpad Pro. Here we borrowed a technique from Avendish
that uses empty ints as compile-time flags. Several C++ con-
cepts can be defined to determine the existence of such ints<fnref n="8" />,
and differentiate them by name.

```cpp
struct buttons {
    enum cc{};
```

```cpp
static consteval uint8_t shift_id()
{
    return 80; }

static consteval uint8_t stop_id()
{
    return 8; }

static consteval uint8_t up_id()
{
    return 91; }

...
```

In this context, we use this technique to determine that the type of midi messages the Launchpad's buttons are sending is MIDI Control Change (CC). The static functions in this aggregate simply return the unique identifier of a particular button.

The grid aggregate is structured in the same way. However
in this case, instead of individual functions per button, a sin-
gle function returns a two-dimensional array of all identifiers
for the grid's buttons, so that they can be accessed with a two-
dimensional index.

```cpp
static consteval auto ids() {
    return std::array<std::array<uint8_t, width()
        >, height()>
        {
            std::array<uint8_t, width()>
                {81, 82, 83, 84, 85, 86, 87, 88},
            std::array<uint8_t, width()>
                {71, 72, 73, 74, 75, 76, 77, 78},
            ...
        }
}
```

### 2.3. Colours

The last nested aggregate necessary for the device specification
is reserved for a colour lookup table. The palette function re-
turns a map of hex formatted colours predefined in the device, as-
sociated to an integer value. In this map, the colour is the key and
the value is the associated integer, from 0 to 127. Technically,
the Launchpad Pro's LED can be set with messages containing
RGB value, but this option significantly increases message size,
and is not supported for every functionality that takes colour ar-
guments. For the sake of simplicity and coherence, we decided
to only support colour specification in the form of a lookup table,
at least for now.

### 2.4. Handling the painter

bugui::midi_hardware is a class that offers a generic interface to
a specific device. It enables drawing on a grid and retrieving
input events, similarly to a more traditional GUI library draw-
ing on screen and retrieving mouse clicks and key presses. The
handle_painter function is passed a reference to a painter object
that contains the desired state of the grid. This state is expressed
as a generic two-dimensional vector of colours.

```cpp
if constexpr(sends_sysex<Device>)
{
    const auto p{Device::sysex_prefix()};
    libremidi::midi_bytes
        b{p.cbegin(), p.cend()};
    b.push_back(Device::light_prefix());
    ...
    b.push_back(Device::sysex_sufix());
    m_output.send_message(b);
}
```

Device refers here to midi_hardware's first template argument.
At the moment, it can only be substituted by either of two
Launchpad_Pro types, corresponding to two different modes of
the device (we provide both options for testing purposes).

The function starts by determining if the device specification
allows for sending SysEx messages. The control structure here
could be extended to support devices that only expects plain midi
messages, without any runtime overhead, since the sends_sysex
concept is determined at compile-time.

This code snippet shows the previously stated method of
concatenating together parts of a SysEx message. In this case,

<footnote n="6">⁶https://novationmusic.com/products/launchpad-pro</footnote>

<footnote n="7">⁷as seen in the Launchpad Pro's user guide:
https://media.discopiu.com/files/2020/1/29/648553-original.pdf</footnote>

<footnote n="8">⁸see include / concepts / message.hpp in the source code</footnote>
the colour values are determined from the content of the painter object (not shown here), and the rest of the message is retrieved from the Device type.

## 2.5. Handling inputs

The challenge in handling inputs from the device is that we need to match the incoming control identifier with its associated function call in the most efficient way. To that effect, we first differentiate between the type of MIDI message we receive. In the case of the Launchpad Pro, we previously saw that buttons outside the grid all send control changes, whereas the grid only sends on/off notes. Other devices may not offer such a clear-cut distinction however, so this part of the code in particular needs to be easily extendable.

To avoid repetition, the following control structure is defined in the case\_msg\_type(C) macro, where C is used for text substitution, to call the appropriate concept. When a control change message is received, for example, calling case\_msg\_type(cc) guarantees that only off-grid button identifiers will be evaluated for a match. This use of concepts enables the compiler to completely ignore the possibility of testing identifiers of the grid buttons against an incoming control change message.

Member functions testing for these matches return false if no association is found between the given input and a definition in the controller class (see 2.6). In this case, the control structure would continue to the next condition. To handle more fine-grained distinctions between input message types, additional conditions can then easily be defined in future development.

```c
ifnoon
    (is_##C<ottypename Device:: buttons>)
{
    if (on_button_received
        (ctrlr, message bytes))
        return;
}

ifconstexpr
    (is_##C<ottypename Device:: grid>)
{
    if (on_grid_received
        (ctrlr, message bytes))
        return;
}
```

## 2.6. The controller class

bugui:: controller is the main class users are supposed to inherit. It can be thought of in the same way a main\_window class operates in a regular GUI library. It is the root of the widget hierarchy, and provides generic hardware access through a unique instance of midi\_hardware.

In the derived class, we propose a system to facilitate hardware integration of common application functionalities like *shift*, *play*, *stop*, etc. Most devices also feature dedicated buttons labelled with these functionalities. As shown previously in the device specification, we named the static functions returning button identifiers accordingly.

Several functions can be defined in the controller's derived class, and directly mapped to the appropriate button press and release. These functions are not overridden. Their name must prefix the button label with on_. They must also be void and take a single boolean argument (i.e. `void on_shift(bool pressed)`).

To enable a certain button press/release to call their associated user defined function, we first establish the existence of such button and function at compile-time. If the user has defined no such function, or if the device does not feature this particular button, this possibility won't be compiled into the project, reducing the number of conditions to evaluate at run-time. Here

again, we place the following constraint in a macro to avoid code repetition.

At present, we consider that the static function returning the button identifier is valid so long as it is not void. We aim to correct this shortcoming soon and establish a set of sensible identifier types.

```c
#define feature_exists(F)
if (constexpr
    (requires (Controller c, bool b)
        {
            std::is
            !std::is
            (Device:: buttons::F##_id())
        }));
```

## 2.7. C++ as a DSL

In our approach, the C++ classes required to support a specific device act as configuration files. We do consider supplying device specification through JSON files to be more practical, expressive and less verbose. However, relying on this format would limit us to loading hardware specification at run-time. We would have to rely only on dynamic types, and perform more operations when interacting with the device. We would have to account for all possibilities at run-time, having no prior knowledge of existent, or missing features of a particular device.

Thus, we are focussing on C++ aggregates, and compile-time programming. The first advantage is performance, and the ability to rule out certain interactions not supported by the device or the application. Keeping these C++ classes declarative also protects users against run-time errors such as memory issues, etc. As we reserve all logic and memory allocation to our hardware class, a user contribution to support a particular device can be limited to defining simple aggregates and static functions. Ill-formed classes will be caught by static type checking and cause compile errors. They may then be addressed early.

## 2.8. Widget tree

After deriving a custom class from bugui:: controller, creating widgets is the second major part in BuGUI's API. Users are only meant to interact directly with the controller, the base\_widget and container\_widget classes, as well as the painter and the presser objects.

The controller already inherits from container\_widget, enabling it to handle button presses, to draw on the device, as well as adding, owning and deleting its children widget. In turn, these user defined widgets can be parent of other widgets, also able to draw, handle presses and so on.

Each widget, controller included, can override the paint function, where the draw\_cell and draw\_line method of the painter can be called. The controller owns the painter object. Upon update, a reference to this object is first passed to the controller's paint function, then to all the controller's children widgets. Every widget behaves this way recursively, as to make the painted cells from their paint function the background for their children's.

Custom widgets can also override the on\_press, on\_double\_press and on\_drag functions. When one of these events is received from the device, its x and y position on the grid is determined and stored in the presser object. A reference to this object is then passed to the controller, and recursively to the first widget whose geometry encompasses the `{x, y}` coordinates in question. Depending on the event, either of these three overridden function will be called on the last widget that meets this criteria, so long as none of its children does too.