# 3. SCORE'S HARDWARE PLUGIN

In this section, we discuss the first project utilising BuGUI: *ossia score*'s hardware integration<fnref n="9" />. This plugin is a work in progress, and statements made here may not all apply in the near future.

## 3.1. Proof of concept

Our first attempt at displaying *ossia score*'s interface on a button grid was based on previous versions of BuGUI that did not feature a widget system, but only a painter. *Intervals* created and manipulated in *ossia score*'s main *Scenario* were directly painted on the grid, but the grid did not offer any access to these *Intervals*.

As expected, it was quickly confirmed that readability would be greatly compromised if every element of the *Scenario* were to be displayed on the grid. Indeed, small elements such as *Intervals* and *Events*, represented in the GUI by horizontal blue lines and vertical white lines, take very little space on screen. They can be drawn very close to one another without there being much confusion where one element ends and another starts.

On the grid however, we are working with a significantly smaller area. Additionally, even small elements, like a single dot on screen, cannot appear any smaller than one cell thick on the grid. This early version made it clear that we ought to only show on the grid elements that are strictly necessary to the current interaction (see Figure 1). Before our efforts to display a *Scenario* on the device went any further, we decided to first implement some level of access to the *Scenario* from the grid, if only to identify possible interactions, and draw accordingly.

## 3.2. Navigation

As well as aiming to de-clutter the display, we also needed to easily navigate inside the *Scenario* and zoom in on specific parts. Navigation also helps us avoid rendering the full span of the UI on our limited grid-estate (as in [5]).

Our currently supported hardware, the Launchpad Pro, features navigation buttons above the grid. We defined the on_left, on_right, on_up and on_down functions in the controller as previously described (see 2.6).

These functions apply an offset and a scale amount to the coordinates of the painter. Navigation and zoom features were then trivial to implement before we introduced the widget system. The added complexity in our current state of the project stems from a widget's coordinates being relative to their parent. Offset and scale now also have to apply to the presser object.

Double clicks with the mouse are also fundamental when navigating a *Scenario* in *ossia score*. They are used to put the content of an interval in full-view, and easily move in and out of nested *Scenarios*. Although we don't yet support this type of navigation in the hardware plugin, detecting double presses is functional in BuGUI. When identified, it is set as a flag in the presser and passed to the on_double Press function of the relevant widget.

## 3.3. Intervals first

*Intervals* are central to *ossia score*'s structures. They are at root of its object hierarchy, and can hold any *Process*, whether it be a piece of media, a signal processing graph, or a *Scenario* (remember that *Intervals* can hold more *Processes*).

The first visible widget made with BuGUI is a representation of an *Interval*. The first user interaction we enabled for these widgets was the ability to be moved vertically, as one would on the GUI by dragging up or down with the mouse.

To that effect we implemented drag detection in BuGUI. The expected behaviour is similar to a touch screen, where users can

press down on a specific graphical element that will follow the position of their finger until they lift it up. On a button grid, we can provide a discrete version of this interaction. Since we could already detect presses on the same button in quick succession for double press, drag detection simply employed the same principle with button presses one cell apart. The coordinates of both the first and the second button are passed together to the presser object. We found this interaction quite intuitive, and are able to move *Intervals* up and down smoothly. We are also able to produce these movement in parallel with both hands, moving two *Intervals* independently at the time, something that is not possible with the mouse on the GUI.

## 3.4. Free from the GUI

Going further with *ossia score*'s hardware integration, it became clear that this plugin could not simply be a companion piece to the graphical interface. Initially, we made liberal use of Qt signals and slots from the GUI, propagating and notifying of value changes. This approach required the hardware and the GUI to always display the same *Scenario*. Navigating to a sub *Scenario*, or attempting to open a *Scenario* without the GUI (in "headless" mode), rendered the device wholly unresponsive.

We also currently retrieve *ossia score*'s colour palette from the GUI. The palette is loaded at startup, unless it is running in "headless" mode.

To circumvent our reliance on the GUI's signals and slots, we implemented recursive functions to propagate changes to relevant elements of the *Scenario*. Currently, these functions allow basic edits on an existing *Scenarios*, but are not yet integrated into its undo/redo system.

To address the colour palette issue, we'll have to detect if it was previously loaded by the GUI, and load it ourselves in the plugin if not. More and more, proper hardware integration is starting to look like designing a new, complete interface for *ossia score*.

Our ultimate goal is to provide users with a full experience of creating, editing, and performing *Scenarios* without requiring the GUI. It would be a testament to the viability of grid interaction, not just for *ossia score*, but, we believe, for almost any intermedia authoring software. Although this perspective remains further ahead into the future, it is guiding our design choices at every step.

# 4. FUTURE WORK

In this section, we will detail the project's current objectives, and lay out its future milestones. Keeping with our current workflow for now, additional features in BuGUI will continue to be added as they are needed in *ossia score*'s hardware plugin.

Along with future development, we hope to gain access to a variety of controllers. The hardware specification file described in this paper (see 2.1–2.3) is optimized for being written easily. This would enable us to quickly provide support for additional devices, and open the door for contribution from a wider audience, maybe even by not necessarily technically minded users.

## 4.1. Ideal interaction

For our interactions to be intuitive, we aim to mirror established gestures common in regular touch screens (as in [5][7]). We also need to remain consistent with the use of cursor actions in *ossia score*, such as double clicks, right clicks, drag and drops or mouse hovering.

We are currently in the process of implementing the double click behaviour from the GUI, with double presses on the grid. This action performed on an *Interval* would display the processes contained in that interval in full view. Double pressing the background of a *Scenario* would create a floating *State*.

<footnote n="9">⁹https://codeberg.org/bugui/scoreaddon-hardware</footnote>
Creation via double presses could also be extended with the introduction of held widgets, equivalent to a mouse selection. Pressing down and holding one or more *State* widgets, followed by a double press elsewhere on the grid, could create new *Intervals*. These *Intervals* would connect the held *States* to the *State* that received the double press. If a double press is performed in the background of the *Scenario*, a new *State* would first be created before creating the *Intervals* connecting it to the *States* currently held.

Holding widgets on the grid could also offer an equivalent to mouse hovering in the GUI. For example, hovering over an *Interval* in *ossia score* reveals a *Mute* icon and a slider to set its speed. Similarly, holding down an *Interval* widget on the grid could show a discrete slider as a row of cells. If a *Mute* button is available on the device, it could be mapped to the *Mute* function of that particular *Interval* when held.

## 4.2. Additional Features

Currently, only a subset of common buttons outside of grids are supported. We hope to extend this subset and support additional controls like sliders, knobs and encoders.

We also note that not all controls are labelled on every controller, and many are simply arranged in one dimensional arrays. Our widget system should also target these arrays.

Some devices feature an actual screen of varying resolution, from basic dot matrixes to modern OLED displays. Supporting screen rendering on these devices would mitigate certain issues such as displaying menus or parameter names. Ideally, we would like to support all types of interfaces, even text based, directly on the grid, but have the option to offload particular interfaces that are better served by a screen, if one is available.

Screens could provide some of the functionalities defined in the application, but without a dedicated control on the device. The Launchpad Pro for example, does not feature a *Play* button. If a screen was available to us, possibly the screen of another connected device, and this other device was also missing this button, *Play* ought to be accessible through the screen. This method, if generalized, would provide a “decentralized” UI, where features on screen are always replaced with hardware control if available.

## 4.3. Optional dependencies

As previously mentioned, similar open-source projects like libmonome and Ctrla already support several devices, and provide hardware integration APIs. Both also offer embedded screen rendering. We will consider the feasibility of adding these libraries as optional dependencies.

Although the scope of these projects may differ from ours, one could conceive of a wrapper making use of certain functionalities of interest. It would provide a unified, minimal, modern C++ interface, using the libremidi<fnref n="10" /> backend as we currently do for generic MIDI devices, a Ctrla backend for HID devices, and a libmonome backend for SerialOSC devices like Monome's Grids.

## 4.4. Beyond *ossia score*

The generic widget system we are building could serve many other applications than *ossia score*. Our current focus on Scenarios is very specific to the software's interface, but nodal patchers, automation curves and step sequencers will also be part of our plugin. After integrating these processes, we intend to upstream as much code as possible into BuGUI. In turn, this library may provide a generic implementation of nodal patchers for example, that could be used directly in environments like PureData or Cardinal. In effect, we would provide the user interface basis for an open-source equivalent to the Empress Zoia.

## 4.5. Embedded systems

One of the possible outcomes of this project would be to enable the full use of *ossia score*, or any software integrated with BuGui, on embedded devices. With a grid and a few additional controls, in lieu of a mouse, a keyboard and a regular monitor, this library would ensure no loss in usability. The user base for these embedded grid devices has already shown its dedication through the success of Monome's Norns (with Grids), Empress's Zoia or Sythstrom's Deluge. Not only were these products adopted in significant numbers, they also formed communities of contributors, creating new scripts, patches and revisions to the firmware, expanding their potential further.

We hope a sufficiently extensive version of BuGUI could soon benefit from the feedback of this community. We also hope that embedding *ossia score* in one of the aforementioned grid devices, or a comparable device of our making, opens the software to the same sort of “hands-on”, performance-driven and immediate creative experience for users.

# 5. CONCLUSION

This paper presents BuGUI, a library enabling efficient interaction with hardware grid interfaces. It enables the creation and manipulation of a basic widget tree on such interfaces. The library can also determine automatically the correspondence of a specific button and its appropriate function call.

By leveraging C++ concepts, we reduce run-time overhead and aim to simplify the process of supporting new devices.

The design of the widget system is meant to remain generic and function similarly to general purpose graphical user interface frameworks.

We put forward a plugin for the *ossia score* sequencer based on this library, and are guiding its evolution. Although still in early development, the plugin can already render elements of its *Scenario* tangibly. Basic interactions currently available reveals the potential of translating a common drag motion on a touch screen to a hardware grid.

# 6. ACKNOWLEDGMENTS

We would like to thank the Linux Audio community for their great work and remarkable contribution to open-source software. We are indebted to all of you for making Linux a more than simply viable option for professional audio.

The research conducted in this publication was funded by the Irish Research Council under grant number EP-SPG/2024/1176. It is supervised by Victor Lazzarini, Gordon Delap and Jean-Michel Celerier. Many thanks to them, as well as Raphael Marczak and Simon Archipoff for their continued support!

<footnote n="10">¹⁰see libremidi’s documentation: https://celtera.github.io/libremidi</footnote>
7. REFERENCES

[1] Jean-Michael Celerier, *Authoring interactive media : a log-ical & temporal approach*, phdthesis, Université de Bor-deaux, Mar. 2018.

[2] Harry Van Haaren, “OpenAV Ctrla: A Library for Tight Integration of Controllers,” in *Linux Audio Conference*, Jean MONNET University, Saint-Etienne – France, 2017, pp. 5–11.

[3] Samuel J. Hunt, “Exploring polyrhythms, polymeters, and polytempi with the universal grid sequencer framework,” in *Proceedings of the 15th International Audio Mostly Confer-ence*, New York, NY, USA, Sept. 2020, AM ’20, pp. 101–106, Association for Computing Machinery.

[4] Beat Rossmy, “Buttons, Sliders, and Keys – A Survey on Musical Grid Interface Standards,” in *International Confer-ence on New Interfaces for Musical Expression*, June 2022.

[5] Beat Rossmy and Alexander Wiethoff, “Musical Grid Inter-faces: Past, Present, and Future Directions,” in *International Conference on New Interfaces for Musical Expression*, June 2021.

[6] Beat Rossmy, Maximilian Rauh, and Alexander Wiethoff, “Towards User Interface Guidelines for Musical Grid Inter-faces,” in *International Conference on New Interfaces for Musical Expression*, June 2022.

[7] Beat Rossmy, Sebastian Unger, and Alexander Wiethoff, “TouchGrid – Combining Touch Interaction with Musical Grid Interfaces,” in *International Conference on New Inter-faces for Musical Expression*, June 2021.

[8] Stefan Püst, Lena Gieseke, and Angela Brennecke, “Interac-tion Taxonomy for Sequencer-Based Music Performances,” in *International Conference on New Interfaces for Musical Expression*, June 2021.

[9] Jean-Michaël Celerier, “Rage Against The Glue: Beyond Run-Time Media Frameworks with Modern C++,” in *International Computer Music Conference*, Limerick, Ireland, July 2022.
# COMPOSING RESILIENT SOFTWARE: ARCHITECTURAL EVOLUTION AND COLLABORATIVE GOVERNANCE IN JACKTRIP'S OPEN-SOURCE JOURNEY

Sven Thielen*

Niederrhein University of Applied Sciences
Mönchengladbach, Germany

sven.thielen@hs-niederrhein.de

Chris Chafe†

Center for Computer Research in Music and Acoustics (CCRMA)
Stanford University, USA

lac@ccrma.stanford.edu

## ABSTRACT

Software architecture erosion poses critical risks for real-time audio systems, where strict latency requirements and legacy dependencies contribute to technical debt. This paper investigates erosion symptoms in JackTrip, an open-source tool for networked music collaboration, during its pivotal evolution from academia to commercial use through a mixed-methods case study. Analyzing 19 historical versions (2015–2022) using architectural recovery, static code analysis, and developer interviews, we uncover a paradox: metrics like cluster coverage stabilized, while link overload architectural smells surged. The COVID-19 pandemic accelerated commercialization and feature development (e.g., Virtual Studios) inducing governance fragmentation between stakeholders. Protocol constraints and framework dependencies, including IPv4 workarounds and tight coupling to JACK/Qt, impaired modernization efforts.

Key contributions include (1) evidence that architectural stabilization metrics can mask smell proliferation in rapidly evolving systems, (2) identification of hybrid governance models as catalysts for architectural drift, and (3) practical mitigation strategies tailored for audio software. We demonstrate how tools like ARCADE Core enable proactive erosion detection while Architecture Decision Records (ADR s) facilitate in documenting trade-offs (e.g., IPv6 transition costs vs. backward compatibility). For the open-source audio community, we advocate latency-aware modular decomposition and collaborative governance frameworks to balance real-time performance with sustainable design. This study provides actionable insights for maintaining architectural integrity in systems straddling research innovation and production-grade reliability.

## 1. INTRODUCTION

Software architecture serves as the structural backbone of systems, balancing technical decisions against evolving organizational and environmental contexts [1]. While principles like modularity, loose coupling, and cohesion aim to balance adaptability with stability [2], domain-specific systems face unique challenges. Real-time audio software, such as networked music collaboration tools, exemplifies this tension: strict low-latency requirements and reliance on specialized frameworks (e.g., JACK Audio Connection Kit<fnref n="1" />) often prioritize immediacy over structural hygiene. These systems accumulate technical debt through protocol optimizations and legacy dependencies, rendering them particularly vulnerable to architecture erosion.

JackTrip, a widely adopted tool for low-latency audio streaming via the User Datagram Protocol (UDP), embodies these challenges. Originating as both a research instrument and artistic platform [3], its architecture evolved through conflicting priorities: academic experimentation, pandemic-driven commercialization, and community-driven open-source development. The 2020 governance split into the nonprofit JackTrip Foundation and commercial JackTrip Labs intensified architectural drift, as rapid feature additions (e.g., cloud-based Virtual

Studios) clashed with legacy protocol constraints. This duality positions JackTrip as an ideal case study to investigate erosion dynamics in systems balancing real-time rigor with sustainability.

In this paper, we analyze JackTrip's architectural evolution across 19 historical versions (2015–2022) through a mixed-methods approach. This temporal scope captures a pivotal phase in JackTrip's evolution, encompassing its transition from academia to commercial use during the COVID-19 pandemic. Limiting the study to this bounded period ensures methodological transparency, enabling readers to interpret findings within the technological and organizational context of the referenced versions. We quantify structural trends (e.g., surging architectural smells) alongside paradoxically stabilizing modularity metrics. Semi-structured interviews with key developers reveal potential erosion drivers: rushed pandemic-era evolution, fragmented governance, and protocol technical debt from IPv4 workarounds. Our findings highlight the importance to raise awareness of the erosion phenomenon and underscore the need for domain-specific mitigation strategies.

We contribute three actionable insights: (1) Architectural recovery and analysis tools enable early erosion detection in real-time systems, (2) Governance frameworks must reconcile stakeholder priorities to prevent fragmentation, and (3) Architecture Decision Records (ADR s) institutionalize knowledge to communicate trade-offs for balancing latency requirements with modernization. By contextualizing erosion within audio software's unique constraints, we advocate for practices that harmonize real-time rigor with long-term resilience, offering lessons for open-source projects navigating dual identities as research tools and production systems.

## 2. SOFTWARE ARCHITECTURE FUNDAMENTALS

### 2.1. Foundations and Evolution of Software Architecture

The definition of software architecture by Perry and Wolf as a "set of architectural elements" [4] has evolved with the contextual impact of organization, environment, and lifecycle [2]. Effective architecture requires orchestration of modules, dependencies, and structural integrity. At its core, modularity governs architectural sustainability through coupling and cohesion [2]. Loose coupling synchronizes interdependent modules minimizing maintenance overheads, while cohesion ensures integrity through self-contained functional units. Martin [5] refines this through stability, with dependencies flowing toward architecturally stable foundations to minimize frequent adaption.

While structural hygiene constitutes a critical dimension of software architecture, Richards and Ford stress that "why is more important than how" [1]. They systematize software architecture through four pillars: architectural styles, characteristics, decisions, and design principles. Architectural styles [6, 5] provide reusable frameworks of the overall structure, contrasting with antipatterns like the chaotic "big ball of mud" [1]. Clean architecture's dependency inversion isolates business logic as a central leitmotif, while ISO 9126's quality model prioritizes maintainability and portability, diverging from practitioner-focused characteristics like scalability and legal compliance [7]. Every

<footnote>* This work extends research from the author's master's thesis.</footnote>

<footnote>† Many thanks to JackTrip Labs, Inc. and the JackTrip Foundation.</footnote>

<footnote n="1">¹https://jackaudio.org/</footnote>
architectural decision embodies trade-offs [1]. Design principles like SOLID [5] guide implementation, while environmental factors—infrastructure, budgets, and methodologies like Test-Driven Design—shape evolutionary cadence. A sustainable architecture thus targets equilibrium—structured yet adaptable, precise yet contextually resonant.

## 2.2. Understanding Architecture Erosion

Technical debt arises when developers prioritize speed over quality, implementing expedient solutions (e.g., temporary code fixes, architectural compromises) to meet deadlines. Unresolved technical debt incurs compound interest over time, creating escalating maintenance overhead that inhibits developmental velocity and contributes to the increase of bugs. Ignoring it results in systems that progressively become harder to modify (developers struggle with bug fixes, new features). Sustaining quality requires constant refactoring to keep the system harmonized within a "corridor of low debt." [8]

Architecture erosion describes the gradual degradation of a system's intended design as requirements shift and complexity escalates [9]. Deviations from architectural boundaries (e.g., code violating layered abstractions) or intentional compromises (e.g., prioritizing releases over quality) distort the system's integrity. This process is independent of system scale: smaller systems may degrade through localized misalignments, while larger systems might retain coherence despite incremental compromises. Architecture erosion often begins insidiously [10], evading detection until maintenance efforts intensify, as the system's structure deviates from its foundational design goals.

## 2.3. Common Symptoms: Architectural Smells

Architectural smells represent suboptimal design decisions that degrade structural integrity within software systems. These smells correlate with architecture erosion, as they can originate from flawed high-level design decisions and manifest as symptoms of degradation [10]. Architectural smells disrupt modularity and coupling, propagating systemic entropy. Though smells must not necessarily become an actual issue, their resolution most probably supports in mitigating architecture erosion.

For instance, dependency cycles violate modularity principles by creating circular dependencies between components [11]. Modifying one component in such a cycle triggers cascading changes across interconnected modules. Similarly, link overload occurs when a component accumulates excessive dependencies [12], creating bottlenecks and increasing the risk of systemic fragmentation.

## 2.4. Possible Reasons For Architecture Erosion

According to Baabad et al. [13], architectural smells rank as the second most prevalent symptom of architecture erosion. These smells often originate from rushed evolution—the primary cause of architecture erosion—where expedited timelines force developers to prioritize speed over structural soundness. Successive system versions accumulate unresolved architectural smells, degrading design coherence over time.

Recurring changes, the second-leading cause, introduce instability when new technologies or features are implemented without alignment to the system's core design [13]. Similarly, a lack of developer awareness leads to architectural violations, as inexperienced contributors introduce code that conflicts with the system's foundational structure.

In academic environments, Laser et al. [14] describe the evolution of research tools as a fragmented ecosystem of single solutions (islands) constituting a "software archipelago." Small prototypes, initially developed as proof-of-concept tools, expand into large, long-lived systems (3L-ARTS) driven by short-term goals (e.g., "publish or perish" mindset and associated con-

straints), often neglecting architectural cohesion. Over time, these systems become tightly coupled, burdened by inconsistent design choices and systemic fragmentation.

# 3. CASE STUDY: EROSION SYMPTOMS AND CAUSES IN JACKTRIP VERSIONS 1.1–1.6.8

We conducted a longitudinal study [15] to diagnose erosion patterns in JackTrip, employing methods to measure decay, identify symptoms, and analyze contributing factors.

## 3.1. Methodology

We employed a mixed-methods approach [16], integrating static code analysis of 19 historical JackTrip versions (2015–2022; selected to capture JackTrip’s pivotal transition from academic prototype to production-grade software) with semi-structured interviews of key contributors in 2023.

Architectures were recovered via the Algorithm for Comprehension-Driven Clustering (ACDC) [17] and analyzed via ARCADE Core<fnref n="2" />, an open-source tool under development. In order to identify the overall architectural change, we assessed the similarity of architectures and clusters from one version to another using the architecture-to-architecture (a2a) and cluster coverage (cvg) metrics, respectively. The measurement of architectural instability and quantification of dependency smells (link overload, dependency cycles) [11] revealed further structural trends, while dependency graphs visualized evolving cluster relationships.

We conducted semi-structured interviews with four developers representing distinct roles and tenure: an academic founder (contributing since 2000), a hobbyist (2019–present), a professional stakeholder (joined during the 2020 pandemic), and an academic/hobbyist contributor (joined before commercialization in 2020). Sessions ranged from 37 to 55 minutes, guided by open questions exploring technical debt, governance tensions, and latency trade-offs. A structured interview protocol ensured thematic focus. Transcripts, generated via OpenAI Whisper<fnref n="3" />, were manually corrected against recordings and validated by participants to ensure accuracy.

We used thematic analysis [18] to process the transcripts and triangulated the findings with the metrics and smells to validate erosion patterns. Thematic analysis followed a six-phase approach: (1) familiarization with data through iterative review and note-taking; (2) systematic coding of text segments to identify patterns; (3) grouping codes into candidate themes, (4) refined through iterative comparison and quality checks; (5) defining/naming themes; (6) and synthesizing findings with exemplary quotes. This structured process ensured rigorous interpretation of interview transcripts while maintaining alignment with the study's focus on architecture erosion symptoms and causal factors.

## 3.2. Case Overview: JackTrip's Purpose and Evolution

JackTrip, developed by Stanford's SoundWIRE group at Stanford University's Center for Computer Research in Music and Acoustics (CCRMA), enables networked music performances and investigates network acoustics via uncompressed, low-latency UDP audio streaming. It addresses IP network challenges like jitter and packet loss, which disrupt time-sensitive audio through audible artifacts. By employing buffering and silence insertion, JackTrip mitigates disruptions while repurposing latency fluctuations as creative parameters, such as virtual resonators using synthesis algorithms (e.g., Karplus-Strong, later Freeverb). This dual-use framework positions it as both a technical research tool and artistic medium [3].

<footnote n="2">²https://github.com/usc-softarch/arcade_core</footnote>

<footnote n="3">³https://github.com/openai/whisper</footnote>
Architecturally, JackTrip evolved from a modular CommonC++ design to integrate Qt<fnref n="4" /> and the Synthesis Tool-Kit (STK). The adoption of JACK enhanced deterministic low-latency processing, enabling multi-core CPU use and Mac OS portability. By 2010, demand for multi-location concerts drove the development of a hub mode—a "multi-client concurrent server" using TCP for session coordination and dynamic UDP ports for scalable audio routing. This design centralized mixing and channel management enabling dynamic gain adjustments [19].

Rooted in Stanford's CCRMA—a hub for music-technology innovation—JackTrip reflects interdisciplinary traditions like FM synthesis and contributions to open-source audio software via Planet CCRMA [20]. The COVID-19 pandemic accelerated its transition to production-grade infrastructure, driven by community contributions (e.g., QJackTrip's GUI) and institutional-commercial partnerships. The nonprofit JackTrip Foundation now maintains core open-source code, while JackTrip Labs<fnref n="5" /> offers cloud-based "Virtual Studio" subscription services to ease session setup for the average musician, illustrating a stakeholder ecosystem blending art, technology, and open-source collaboration [21].

### 3.3. Architectural Decay Patterns

The Algorithm for Comprehension-Driven Clustering (ACDC) recovery method organizes source files into modular clusters, representing bodyheader file pairs as circular nodes annotated with filenames and paths. Directed edges denote dependency relationships between these clusters. Our ACDC-derived visualizations reveal incremental architectural changes across Jack-Trip versions, particularly highlighting feature-driven dependency growth.

Figure 1 demonstrates this characteristic for architectural changes from version 1.5.3 to 1.6.0. Red clusters mark components introduced in version 1.6.0 indicating the Virtual Studio feature, while blue edges indicate new inter-module dependencies. Subsequent analysis of version 1.6.8 (Fig. 2) shows how these initial additions evolved, particularly noting the dependency dispersion within the codebase by additional Virtual Studio related clusters (e.g., src/gui/vsDevice.ch) [22].

<figure><img src="image_44.png" /><figcaption>Figure 1: Version 1.6.0 architecture with introduced clusters (red) and new dependencies (blue) since version 1.5.3. A larger version of this figure is available in the appendix as Fig. 4.</figcaption></figure>

<figure><img src="image_45.png" /><figcaption>Figure 2: Cluster evolution from version 1.5.3 to version 1.6.8. Sustained growth of new dependencies (blue) and peripheral cluster additions (red). A larger version of this figure is available in the appendix as Fig. 4.</figcaption></figure>

The accumulation of code smells identified through static code analysis points to progressive architecture erosion. Link overload smells surge from 1 to 11 between versions 1.1 and 1.6.8 (Figure 3). Specific releases (v1.3.0, v1.4.0, v1.6.0, and v1.6.2) mark their introduction and persistence up to the last version we investigated (v1.6.8). Dependency cycles decrease from 2 to 1. However, the persistent dependency cycle smell includes clusters related to the Virtual studio feature after its inception in version 1.6.0 which have been previously identified to disperse dependencies using the visualizations [22].

<figure><img src="image_46.png" /><figcaption>Figure 3: Architectural smells in JackTrip's evolution.</figcaption></figure>

In contrast to the smells trend, the metrics (Table 1) show a path towards stabilization. Architectural instability declines from 0.56 to 0.42, while architecture-to-architecture (a2a) and cluster coverage (cvg) metrics consistently reach 100% from version 1.4.1 onward, indicating architectural stabilization and cluster consistency in later JackTrip versions. Notable deviations occur in version 1.3.0, where sharply lower a2a (57.64%) and cvg (57.90%) align with a temporary architectural instability spike (0.58). Lower a2a correlates more strongly with instability reduction and new link overload smells in subsequent releases, reflecting an erosion paradox during evolution [22].

### 3.4. Environmental and Organizational Factors

The interviews identify seven interrelated causes for architecture erosion and potential drivers. We outline the most relevant here. JackTrip's academic origins and real-time audio focus increase risks of architecture erosion. Its evolution accelerated during the COVID-19 pandemic in 2020 due to urgent demand for remote music collaboration, driving commercialization and usability improvements, as described by the interviewees.

<footnote n="4">⁴https://wwwovi.io/</footnote>

<footnote n="5">⁵https://www.jacktrip.com/</footnote>
<table><caption>Table 1: Architectural metrics for JackTrip versions 1.1–1.6.8 (2015–2022) using ACDC recovery. Metrics include Architecture-to-architecture (a2a), Cluster coverage (cvg), and Architectural Instability (Arch Inst).</caption><thead><tr><th>Release Date</th><th>Version</th><th>a2a</th><th>cvg</th><th>Arch Inst</th></tr></thead><tbody><tr><td>2015-05-27</td><td>1.1</td><td>-</td><td>-</td><td>0.56</td></tr><tr><td>2020-06-07</td><td>1.2</td><td>92.76</td><td>89.47</td><td>0.56</td></tr><tr><td>2020-08-05</td><td>1.3.0</td><td>57.64</td><td>57.90</td><td>0.58</td></tr><tr><td>2021-11-02</td><td>1.4.0</td><td>83.22</td><td>95.24</td><td>0.53</td></tr><tr><td>2021-11-11</td><td>1.4.1</td><td>100</td><td>100</td><td>0.53</td></tr><tr><td>2021-12-05</td><td>1.4.2</td><td>98.43</td><td>100</td><td>0.52</td></tr><tr><td>2021-12-19</td><td>1.4.3</td><td>100</td><td>100</td><td>0.52</td></tr><tr><td>2022-01-03</td><td>1.5.0</td><td>100</td><td>100</td><td>0.52</td></tr><tr><td>2022-01-07</td><td>1.5.1</td><td>100</td><td>100</td><td>0.52</td></tr><tr><td>2022-03-02</td><td>1.5.2</td><td>97.53</td><td>96.43</td><td>0.52</td></tr><tr><td>2022-03-28</td><td>1.5.3</td><td>100</td><td>100</td><td>0.52</td></tr><tr><td>2022-06-01</td><td>1.6.0</td><td>90.25</td><td>100</td><td>0.47</td></tr><tr><td>2022-06-21</td><td>1.6.1</td><td>100</td><td>100</td><td>0.47</td></tr><tr><td>2022-08-17</td><td>1.6.2</td><td>93.59</td><td>100</td><td>0.44</td></tr><tr><td>2022-08-23</td><td>1.6.3</td><td>100</td><td>100</td><td>0.44</td></tr><tr><td>2022-09-16</td><td>1.6.4</td><td>98.03</td><td>100</td><td>0.44</td></tr><tr><td>2022-11-02</td><td>1.6.6</td><td>95.69</td><td>100</td><td>0.44</td></tr><tr><td>2022-12-02</td><td>1.6.7</td><td>97.69</td><td>100</td><td>0.42</td></tr><tr><td>2022-12-06</td><td>1.6.8</td><td>100</td><td>100</td><td>0.42</td></tr></tbody></table>

Early governance ambiguity between JackTrip Labs and the OSS project initially caused instability, as uncoordinated commits from volunteers and commercial developers have generated synchronization risks. Disproportionate contributions from full-time commercial teams—employing systematic bug resolution (e.g., automated package checks) and Scrum workflows— contrast with transient OSS engagement patterns, where volunteers address niche features or maintenance tasks. While process refinements (shorter commit cycles, enforced pull requests) mitigated merge conflicts, tensions persisted between JackTrip Labs’ customer-driven prioritization due to Virtual Studio related development and the OSS community’s passion-driven, ad hoc contributions [22].

Legacy architectural decisions, particularly tight coupling via Qt’s signals/slots and CLI/GUI "double management" of settings complicates modernization efforts. JackTrip Labs’ push for microservices and modern C++ standards clashes with the OSS codebase’s entangled cross-platform code (e.g., OS-specific macros), while usability improvements like the GUI from QJackTrip tend to have introduced redundant logic. Interviewees advocate decoupling core protocol logic into a shared library, separating it from application layers (CLI/GUI, client/server). This approach would reduce platform-specific code fragmentation while the HackTrip prototype—a Qt 6/RtAudio rebuild—is considered as a template for incremental refactoring. Despite consensus, framework dependencies (e.g., JACK) and backward compatibility requirements delay full realization of this vision [22].

JackTrip’s protocol evolution efforts reveal tensions between modern usability demands and its foundational low-latency requirements (e.g., minimal headers and small packets optimized for 2000s-era bandwidth constraints). Early design choices, such as non-compliant IPv4 port-binding, prioritized real-time performance but created technical debt that hindered IPv6 compatibility and extensibility (e.g., lacking version negotiation or dedicated MIDI channels). While developers agree on modernizing the protocol, progress stalls primarily due to insufficient documentation. Concurrently, JackTrip Labs has shifted focus toward accessibility leveraging geolocated Virtual Studio rooms to reduce latency, underscoring the challenge of balancing real-time audio fidelity with broader adoption through usability improvements [22].

## 3.5. Implications for JackTrip’s Architecture

The most significant architectural change occurs in version 1.3.0, which corroborates with interviewees’ perception of a rapid evolution during the pandemic. This aligns with the increase of the architectural smells trend, mirroring Baabad et al.’s second most observed symptom of architecture erosion during a rushed evolution [13]. However, version 1.6.0 marks another significant spike of link overload smells, correlating with the introduction of the Virtual Studio feature by JackTrip Labs.

JackTrip Labs’ adoption of modern practices (e.g., microservices, newer C++ standards) clashes with dependencies (e.g., JACK, Qt) from earlier design choices, contributing to the risk of a fragmented architecture, as described by Laser et al.’s archipelago model of independently evolving "islands" [14]. The original protocol’s lack of formal specifications and versioning led to workarounds (e.g., during IPv6 adoption), complicating modernization while maintaining user trust. Such efforts mirror the tension between preserving backward compatibility while advancing infrastructure for future needs and foster innovation. Balancing these priorities remains key to sustaining JackTrip’s evolution while honoring its technical roots.

## 4. LESSONS LEARNED AND MITIGATION STRATEGIES

Our study shows how employing architectural recovery and analysis facilitates diagnosing potential architecture erosion. Metrics like link overload and dependency cycles can provide early warnings of decay, while visualizations expose dispersion trends. Regular architectural assessments can support projects in balancing innovation with sustainability, particularly during transitions (e.g., academic-to-commercial). Software architecture recovery and analysis, as exemplified by our case study, is considered as the first step in a mitigation strategy with technical debt removal and countermeasure implementation described by Link et al. [23].

Contextual background and reasons for architecture erosion can be investigated using semi-structured interviews with developers. We employed a mixed-methods approach for triangulation and to better understand the underlying reasons of potential issues, as well as to investigate the perception of erosion symptoms and causes (e.g., rushed evolution). Our findings highlight the need for governance frameworks that reconcile divergent stakeholder priorities. Proposals like protocol versioning and separation of the core functionality and end application suggest pathways to find technical solutions. However, a lack of documentation and trade-offs regarding the real-time focus of JackTrip remain challenges in its evolution.

We propose the adoption of Architecture Decision Records (ADRs)<fnref n="6" /> to institutionalize knowledge, reduce ambiguity, and ensure architectural integrity as systems scale. Embedding ADRs in version-controlled repositories (e.g., Git) or wikis enhances traceability, while concise pros-and-cons discussions foster alignment on trade-offs, such as usability versus low-latency prioritization. For instance, an ADR might document IPv6 transition trade-offs, mirroring retrospective analysis of IPv4 workarounds to address JackTrip’s protocol documentation gaps.

## 5. CONCLUSION: TOWARD SUSTAINABLE ARCHITECTURES IN AUDIO SOFTWARE

The investigated historical system versions in JackTrip’s evolution underscore the fragile balance between innovation and architectural integrity in open-source audio software. The project’s pandemic-driven acceleration, marked by features like Virtual Studio and governance tensions between stakeholders, reveal

<footnote n="6">⁶https://adr.github.io</footnote>
how rapid growth induces architecture erosion, even as modu-
larity metrics suggest stabilization. Legacy dependencies, pro-
tocol limitations, and fragmented priorities exemplify systemic
challenges faced by real-time systems, where technical debt can
accumulate over time.

Mitigating architecture erosion demands proactive strate-
gies: We propose adopting ADRs to document trade-offs trans-
parently, using tools like ARCADE Core for early decay de-
tection, and harmonize backward compatibility with modern-
ization. However, audio software's architectural sustainability
depends on domain-specific requirements and may not bene-
fit from immediate mitigation. For instance, strict low-latency
guarantees often force tight coupling to available frameworks
(e.g., JACK) and simple network protocols (e.g., UDP) while
creative workarounds may accumulate technical debt to circum-
vent unsolved technical challenges.

The dual identity of tools like JackTrip, which navigates cre-
ative experimentation and commercial reliability, distorts gov-
ernance priorities: academic contributors may emphasize ex-
perimental features, while commercial stakeholders want sta-
bility, resulting into architectural drift. These challenges de-
mand tailored mitigation strategies, such as latency-aware mod-
ular decomposition and explicit documentation of perceptual
trade-offs in ADRs. By foregrounding audio's unique needs,
the open-source community can advance domain-specific archi-
tectural practices that harmonize real-time rigor with long-term
resilience.

6. ACKNOWLEDGMENTS

We thank JackTrip contributors, users and JackTrip Labs for collaboration, and Stanford's CCRMA for fostering interdisciplinary innovation. This work utilized the ARCADE Core toolset, revitalized by Marcelo Schmitt Laser, whose expertise in software architecture and responsiveness to technical requests facilitated robust structural analysis. We acknowledge his indispensable contributions to this research.

7. REFERENCES

[1] Mark Richards and Neal Ford, *Fundamentals of Software Architecture*, O'Reilly, 2020.

[2] Arne Koschel, Andreas Rausch, and Mahbouba Gharbi, *Software Architecture Fundamentals*, 3 edition, 2019.

[3] Chris Chafe, "I am Streaming in a Room," *Frontiers in Digital Humanities*, vol. 5, 2018.

[4] Dewayne E. Perry and Alexander L. Wolf, "Foundations for the study of software architecture," *SIGSOFT Softw. Eng. Notes*, vol. 17, no. 4, pp. 40–52, 1992.

[5] Robert C. Martin, *Clean Architecture: A Craftsman's Guide to Software Structure and Design*, Pearson, 2017.

[6] Eric Evans, *Domain-Driven Design: Tackling Complexity in the Heart of Software*, Addison-Wesley Professional, 2003.

[7] Martin Glinz, "A risk-based, value-oriented approach to quality requirements," vol. 25, no. 2, pp. 34–41.

[8] Carola Lilienthal, *Sustainable Software Architecture*, 2019.

[9] Erik Whiting and Sharon Andrews, “Drift and erosion in software architecture: Summary and prevention strategies,” in *Proceedings of the 2020 the 4th International Conference on Information System and Data Mining*. 2020, ICISDM 2020, pp. 132–138, Association for Computing Machinery.

[10] Ruiyin Li, Peng Liang, Mohamed Soliman, and Paris Avgeriou, “Understanding software architecture erosion: A systematic mapping study,” *Journal of Software: Evolution and Process*, vol. 34, no. 3, pp. e2423, 2022.

[11] Duc Minh Le, Carlos Carrillo, Rafael Capilla, and Nenad Medvidovic, “Relating architectural decay and sustainability of software systems,” in *2016 13th Working IEEE/IFIP Conference on Software Architecture (WICSA)*, 2016, pp. 178–181.

[12] D. Le, Daniel Link, Arman Shahbazian, Yixue Zhao, C. Mattmann, and N. Medvidović, “Toward a Classification Framework for Software Architectural Smells,” 2017.

[13] Ahmed Baabad, Hazura Binti Zulzalil, Sa'adah Hassan, and Salmi Binti Baharom, “Software Architecture Degradation in Open Source Software: A Systematic Literature Review,” *IEEE Access*, vol. 8, pp. 173681–173709, 2020.

[14] Marcelo Schmitt Laser, Duc Minh Le, Joshua Garcia, and Nenad Medvidović, “Architectural archipelagos: Technical debt in long-lived software research platforms,” in *2021 IEEE/ACM International Conference on Technical Debt (TechDebt)*, 2021, pp. 94–98.

[15] Robert K. Yin, *Case Study Research and Applications*, Sage Publishing, 6 edition, 2018.

[16] Per Runeson, Martin Höst, Austin Rainer, and Björn Regnell, *Case Study Research in Software Engineering*, John Wiley & Sons, Ltd, 2012.

[17] V. Tzerpos and R.C. Holt, “ACCD: an algorithm for comprehension-driven clustering,” in *Proceedings Seventh Working Conference on Reverse Engineering*, 2000, pp. 258–267, ISSN: 1095-1350.

[18] Virginia Braun and Victoria Clarke, “Thematic analysis.,,” in *APA handbook of research methods in psychology, Vol 2: Research designs: Quantitative, qualitative, neuropsychological, and biological.*, APA handbooks in psychology®, pp. 57–71. American Psychological Association.

[19] Juan-Pablo Cáceres and Chris Chafe, “JackTrip/SoundWIRE Meets Server Farm,” *Computer Music Journal*, vol. 34, no. 3, pp. 29–34, Sept. 2010.

[20] Andrew J. Nelson, *The Sound of Innovation: Stanford and the Computer Music Revolution*, The MIT Press, Mar. 2015.

[21] Marina Bosi, Antonio Servetti, Chris Chafe, and Cristina Rottondi, “Experiencing remote classical music performance over long distance: A JackTrip concert between two continents during the pandemic,” vol. 69, no. 12, pp. 934–945, 2021.

[22] Sven Thielen, “Investigating jacktrip’s software architecture: A case study on prominent reasons for architectural erosion in open source software originating from academia,” Master’s thesis, Hochschule Düsseldorf, 2023.

[23] Daniel Link, Pooyan Behnamghader, Ramin Moazeni, and Barry Boehm, “The value of software architecture recovery for maintenance,” New York, NY, USA, 2019, ISEC ’19, Association for Computing Machinery.

[24] Juan-Pablo Cáceres and Chris Chafe, “JackTrip: Under the Hood of an Engine for Network Audio,” *Journal of New Music Research*, vol. 39, no. 3, pp. 183–187, 2009.

8. APPENDIX

For improved readability of the cluster topologies and their de-
pendency relationships, expanded versions of Figures 1 and 2 are
provided in this appendix (Figures 4 and 5, respectively). These
high-resolution renderings enable detailed inspection of node la-
bels and edge connections.
<figure><img src="image_47.png" /><figcaption>Figure 4: Version 1.6.0 architecture with introduced clusters (red) and new dependencies (blue) since version 1.5.3.</figcaption></figure>

<figure><img src="image_48.png" /><figcaption>Figure 5: Cluster evolution from version 1.5.3 to version 1.6.8. Sustained growth of new dependencies (blue) and peripheral cluster additions (red).</figcaption></figure>
# CSOUND 7: BARE METAL AND CO-PROCESSING HARDWARE

Victor Lazzarini

Music Department
Maynooth University, Ireland
victor.lazzarini@mu.ie

Aman Jagwani*

Music Department
Maynooth University, Ireland
aman.jagwani.2023@mumail.ie

## ABSTRACT

This paper introduces the support for Bare Metal and co-processing computing devices available in the newest major version upgrade of Csound. Alongside a host of significant changes to the system and language, we have invested significant effort to expand the range of platforms in which Csound can be run. After an introduction setting the context for these developments, we explore first the challenges to porting the system to computing environments without an operating system. This is followed by a discussion of two different implementations on ARM hardware. We then turn our attention to co-processing devices and the possibility of partially running Csound on these systems, through the support of the module API. We discuss two cases: general-purpose graphic processing units and field-programmable gate arrays. The paper then concludes with an overview of potential applications.

## 1. INTRODUCTION

Csound is a Sound and Music Computing system. It is Free Software, licensed by the Lesser GNU Public Licence. Since its inception, the development of Csound has been focused on portability. It came to life as a C-language port (hence its name) of the MUSIC 11 system developed mostly in PDP-11 assembler by Barry Vercoe at MIT [1]. Originally written in Kerningham & Ritchie C, it was ported again in 1992-3 to ANSI C by John ffitch at Bath University. Also early on, versions for what may be classed as embedded or co-processing hardware came to light in terms of experimental parallel implementations on transputers [2] and digital signal processing hardware [3].

The development of an object-oriented application programming interface (API, as well as a fully re-entrant library) for version 5 in 2003 allowed further possibilities for porting to different platforms. This was applied to the One Laptop Per Child project [4], a hardware/software project that pre-empted the advent of single-board Linux-based computers such as the Raspberry PI. Ports to mobile phones (under Android and iOS) soon followed. With the appearance of Csound 6 in 2013, Csound became present on a variety of platforms, running on Web browsers (first through PNaCl, then JavaScript / Web Audio, and finally WASM / Web Audio), embedded under its own Linux distro (Intel Gemini/Edison) [5], single-board computer systems (Raspberry PI [6], Beaglebone black [7], Bela [8]), and co-processing hardware (general purpose graphic programming units, GGPUs) [9], all the way up to supercomputers. From this perspective, the last frontier to conquer in version 7 became the bare-metal microcontroller (MCU) / system-on-chip (SoC) platforms: all environments to which Csound 6.x was ported relied on some form of operating system (OS).

In this paper, we discuss the porting of Csound version 7 to bare-metal hardware as well as some more recent developments in running components on co-processing hardware. We first discuss the support for bare-metal ARM [10] based platforms, in particular for the Electrosmith Daisy and the Xilinx Zynq 7000 boards. We complement the article by looking at the implementation of co-processing on GPGPUs and field-programmable gate (FPGA) hardware.

## 2. EMBEDDED SYSTEMS AND BARE METAL PLATFORMS

Embedded audio programming platforms enable users to create customised tools, instruments, and interfaces for musical performance, interaction, and expression. They extend the creative process into the design of the musical instruments or tools themselves [11]. Moreover, the easy availability and affordability of programmable embedded systems allow both artists and hobbyists, as well as larger commercial and research entities, to explore this field. This underscores the significance of the programming language or environment used in these devices, particularly in the audio domain. The availability of specialised and higher-level systems like Csound within these platforms can significantly enhance audio processing outcomes and accessibility.

These platforms can be classified into two main categories, Linux or operating system-based single-board computers like Raspberry Pi and the BeagleBone Black and bare-metal systems or MCUs without operating systems like the STM32 [12] and ESP32 [13]. Most software that target desktop environments work seamlessly on Linux-based embedded systems, allowing users to leverage existing tools like Csound with ease. The convenience of operating systems also means that several tasks can easily leverage existing drivers. An example of this is peripheral communication, MIDI or OSC. However, generally these boards are not audio specific and require external break-out boards for audio, MIDI etc. They also come with overheads inherent in operating systems containing functionalities that are not required for audio applications and can result in higher latency or less available computational power for real-time audio. Specialised single-board computer systems like the Bela mitigate these issues by using dedicated sister-boards and Xenomai RT Linux to optimize for real-time audio with low latency. However these boards are relatively expensive.

On the other hand, inexpensive bare-metal microcontrollers have recently become very powerful, capable of performing most audio processing tasks with low latency and quick startup times [14]. The advantages of these devices include their cost, portability (due to small form factors), and specificity. Since all functionality is programmed directly onto the microcontroller, it can be optimised and tailored to perform a specific audio processing function at its highest level of capability. Furthermore, the cross-compilation process for microcontrollers can also be portable across different chips with the same architecture. For example, our experience with ARM-based bare-metal MCUs showed that building Csound for one chip, such as the Electrosmith Daisy [15], enabled it to work on other chips like the Xilinx Zynq 7000 [16] with minimal adjustments. This opens up several different MCUs and boards that have ARM CPUs with just one build process, such as the Teensy [17], Raspberry Pi Pico [18], and the STM32 Nucleo Boards [19].

However, bare-metal systems may be limited by the resources available on the MCU. Parallel or multi-core processing is also less common in these systems. Another challenge with bare-metal systems is that every task needs to be implemented manually for these chips, and most programming is done at a lower

<footnote>* This work was funded by the Maynooth University Hume Scholarship Programme</footnote>
level, potentially increasing development time. For example, a user may need to implement MIDI drivers individually on different microcontrollers to establish communication with their specific sets of peripherals. Similarly, functionalities that are typically managed by operating systems, such as file I/O, need to be implemented manually on microcontrollers as well. With the emergence of platforms like Arduino [20] and Daisy, as well as comprehensive vendor-supplied Hardware Abstraction Layers (HALs), a wide range of libraries and drivers are available to mitigate these issues.

Both types of platforms have their sets of advantages and disadvantages and are best suited for specific use cases. In terms of Csound and its supported platforms, it became clear to us that enabling its use on both types of platforms would be beneficial for users. Until now, Csound was only supported on Linux-based embedded systems, as demonstrated by the pioneering work done for Csound on the Bela [21] and the Raspberry Pi [22]. This limitation arose because Csound relied on the services of an operating system and libraries such as libsndfile [23], which were only supported on OS-based platforms. Now, with Csound 7, it can be built without these dependencies, enabling cross-compilation for bare metal.

## 2.1. Porting Csound to Bare Metal Platforms

Csound 6.x requires the services of an OS in order to run. However, since it is mostly made up of portable C/C++ code (mostly C in fact), it is not beyond the realms of possibility that a port of the system could be designed to run directly on a SoC without an OS. This is supported by the fact that many such platforms provide C/C++ compiler and linker toolchains. Unlike the previous attempts to run Csound on dedicated DSP hardware, we would not require any platform-specific programming through micro-coding and/or assembly.

In porting Csound to run on bare metal, the first step is to identify the parts of the system that required the presence of an OS. We can split these into three elements: (a) soundfile input/output (IO); (b) memory (free store); and (c) miscellaneous non-implemented syscalls. All other non-compatible parts of the Csound system can be turned off in the build, simplifying the process.

### 2.1.1. Soundfile Input and Output

Soundfiles have been essential means of IO for Csound since the very beginning. In fact, before any realtime device access was implemented in the early 1990s, the only means of audio programming required the use of soundfiles. Up until Csound 5 was released, Csound relied on its own implementation for accessing the various soundfile formats in common usage. From Csound 5 onwards, we relied on the services of libsndfile, which in these 20+ years has become the Free software standard for soundfile IO. In fact, in Csound 6.x it became the only required external dependency for the system, which reflected on the fact that it had become somewhat untangled internally with Csound. Fortunately, all other forms of file IO had either been moved to use core memory files (such as for instance temporary compilation files etc) or made optional. These meant that running in a platform with no file services boiled down to removing the libsndfile dependency.

By 2018, bare metal started to emerge as a potentially important platform for audio programming. At that time, some initial investigations on disentangling Csound and libsndfile demonstrated that some form of major code surgery was required. At the time, we also started to plan a new major update to replace version 6.x. The opportunity to make significant changes to the code could not be missed. We developed version 7.0 in a very slow and incremental way, and in 2022 the removal of libsndfile as a dependency was carried out. This was followed, in the more

recent redesign of the API with a means of supporting alternative forms of audio IO, which can be designed in such a way to support any existing disk/memory file access in a SoC.

### 2.1.2. Memory

Another issue that we had not originally anticipated was the need to handle free store (heap) access. Under an OS, we in most typical situations only need to use the malloc syscall to access the heap, and then manage the memory for a program. Under the bare metal platform, this might not be enough. The system may not give direct access to all available memory. The solution is to implement our own version of malloc, which can then handle the allocation at a designated memory location, which we can set at compile time and declare in a loader script. With this in place, we are able to use the available memory resources of a SoC in a much more flexible way.

### 2.1.3. Miscellaneous Syscalls

A number of syscalls such as the ones handling directory access, dynamic loading, localisation, temporary files, IP network, threading, etc. had to be excluded from the bare metal build. This also meant for instance that no plugin libraries (for opcodes and other modules) would be available. However, Csound 7 has also introduced means of building the system with or without opcode plugins (allowing on one hand different library footprints, and support for systems with no dynamic loading, on the other). Therefore we are able to provide more or less a complete core system, without of course any components that may be dependent on external libraries.

## 2.2. The Daisy Platform

The Daisy is an open-source audio programming platform based around the STM32H7 microcontroller [12]. It contains an ARM Cortex-M7 CPU along with several digital and analog inputs and outputs as well as other peripherals such as audio ADCs and DACs, USB serial and MIDI communication, SPI, UART interfaces etc. It also contains different volatile and non-volatile memory locations, each of varying size and performing at different speeds, allowing flexibility in the trade offs between program size, RAM requirements and performance [24]. The Daisy also provides higher level access to the STM32 HAL through their libDaisy [25] hardware library, simplifying the programming process.

Apart from the Csound implementation, the Daisy supports programming in C++, Arduino, PureData, Max/MSP gen. Each of these environments have some limitations. The C++ and Arduino programming is generally done with DaisySP [26], Elektrosmith's DSP library containing DSP objects ported from different open source libraries (including Csound). The capabilities of audio programs done in this context would therefore be limited by the availability of objects in the library which may not cover all DSP or audio programming aspects. For example, there is not much support for generative techniques in this library. Surely, in a C++ environment missing features can be implemented from scratch or ported from other libraries as done in [11], but that is not suitable for users or artists with limited experience. Max/MSP comes with the inherent limitation of being closed source and requiring a paid license. In terms of Pure Data support, the Daisy utilises the PlugData [27] environment or the pd2daisy utility [28] which in turn use the Heavy Compiler/HVCC to generate C or C++ audio code from Pure Data patches. In this context, Pure Data is more like a front end rather than an audio engine that is used to generate the audio code. Also, all pure data objects are not supported by this compiler, so patches may be limited [29].

Therefore, Csound support can greatly enhance the Daisy platform. Firstly, Csound is a mature and complete audio pro-
gramming system. It covers virtually every DSP and audio pro-
gramming task that a user would require. Secondly, bare-metal
Csound builds and uses the Csound source code itself as a static
library. This enables portability of Csound programs to this plat-
forms as well as the leveraging and consistency of almost all of
Csound features.

2.2.1. Implementing Csound on Daisy

The development process for Csound on the Daisy had two parts. Firstly, following the basic work to enable the bare metal build, we worked on attempting to run Csound code on the board. We made a simple test program utilising the Csound API within the Daisy C++ environment as a test to enable Csound to compile, start and perform. As we were conducting our experiments we had to tackle some execution issues to have a successful run. One of the first issues was related to the Csound library footprint. By default, Daisy programs are stored on the internal 128kb flash memory of the board. This was not sufficient for our build of Csound, which required around 2MB of memory. The solution was to move the binary to an external flash memory, which supported 8MB of memory. While it provides enough space, there is a small impact on performance as a tradeoff.

Another issue was the location of the heap for our custom al-
locator. The Daisy board has 1MB of internal memory available
to programs. We identified that this was not enough for a typical
Csound usage; while it was enough for our stack needs, we had
to place the heap on the external 64MB SDRAM available on
the board. Following these adjustments, we were able to run a
program on the Daisy, which employed the Csound host API.

Once all of these steps were completed, Csound code was
able to run without any issue. The following is a basic example
of audio callback for the platform:

void AudioCallback (AudioHandle::InputBuffer in,
 AudioHandle::OutputBuffer out, size_t size) {
 const MYFLT *spout = csoundGetSpout (csound);
 int end = csoundGetKsmps (csound) *
         csoundGetChannels (csound, 0);
 for (size_t i = 0; i < size; i++) {
  if (cnt == 0) csoundPerformKsmps (csound);
  out[0][i] = spout[cnt];
  out[1][i] = spout[cnt + 1];
  cnt = cnt == end - 2 ? cnt + 2 : 0;
 }
}

The next step was to create an interface between the board
hardware peripherals and Csound code so that users could write
Csound code to access them. For this, we take advantage of the
Csound module API, and used the existing BelaCsound code as
a reference [21]. For analogue inputs, we simply connected into
software bus channels named Analog0, Analog1 etc. To han-
dle digital inputs, we also added an opcode with the following
signature:

out:k = digiInDaisy (pin:i, mode:i)

We also connected the Csound message callback mechanism
with the serial logging available on the board, so we could have
access to system running information. For MIDI, we imple-
mented a set of callbacks as a backend, connecting the Daisy
USB MIDI interface to Csound. Since both of these features use
the USB port on the board, only one of them is accessible at any
given time, unless a separate UART interface is used for MIDI.

The daisy Csound build is part of the Csound core codebase.
Examples can be found at

https://github.com/csound/csound

As shown in [11], the Daisy is useful for creating custom
audio processing devices for live performance. The small form
factor and relatively low price of this board also means that it
can easily be embedded into artworks and installations and com-
bined with various forms of interactivity through sensors. Csound
support can greatly enhance these benefits of using this platform.

2.3. Csound on the Zynq 7000

Besides the Daisy board, there are several other SoCs that con-
tain ARM Cortex CPUs. One of them is the Xilinx Zynq 7000,
which is an FPGA-based SoC. It contains two parts, the process-
ing system (PS), consisting of a dual-core ARM Cortex-A9 CPU
and the programmable logic (PL), containing the FPGA fabric
[16]. This enables interfacing between programs running on the
CPU with custom accelerated hardware running on the PL.

There are several development boards available with this
SoC. We used the Digilent Zybo Z7020 [30] for our experimen-
tal and development process due to the availability of a large
amount of hardware peripherals as well as logic resources on it.

Following our experience with the ARM CPU in the Daisy
board, we were able to develop a second bare metal build for
Csound to run on this SoC with minimal changes from the Daisy
board build. The main difference was that we had to re-target
our build and the cross-compile script to the Cortex-A9 instead
of Cortex-M7. The same compiler toolchain can be reused to
create the binaries. This allowed us to run Csound on the PS of
the Zybo board. To enable audio input and output, we had to
develop an interface with the PL.

The Zybo board contains an Analog Devices SSM2603 au-
dio codec that facilitates audio input and output to and from the
PL of the Zynq SoC. This codec transmits audio using the I2S
format [31], so to interface with the rest of the system, we used
the I2S transmitter and receiver intellectual property (IP) cores,
which are pre-packaged with the Xilinx development toolchain
in the Vivado software suite [32].

To send and receive audio between Csound on the PS and
these IPs, we used the AXI Direct Memory Access (DMA) con-
troller IP along with the Xilinx AXI DMA driver [33]. The
I2S IPs communicate audio data using the AXI Stream protocol
[34], so audio output from Csound was written to the memory-
mapped-to-stream (mm2s) channel of the DMA, while audio
input to Csound was read from the stream-to-memory-mapped
(s2mm) channel. For the PS application code, we again used the
Csound module API, similar to the Daisy implementation, and
connected the Csound engine main IO buffers (spin, spout) to
the audio transfers via the DMA controller.

3. CO-PROCESSING

In this section, we outline some aspects of the implementation
of Csound components (opcodes) using co-processing hardware.
We will first discuss the use of GPGPUs. This will be followed
by a discussion of FPGAs and their applications.

3.1. GPGPUs

The first published implementations of Csound opcodes employ-
ing GPGPUs as co-processors stretch back over a decade. De-
veloped through the use of the module API, these included im-
plementations of the sliding phase vocoder (PV) [35], a sample-
by-sample highly parallel spectral process, and the implementa-
tions of hopping PV analysis and synthesis, as well as additive
synthesis (of amplitude-frequency PV data) [36]. These were
built using the proprietary Compute Unified Device Architecture
(CUDA) API and as a result targeted only NVIDIA hardware.
Fast partitioned convolution was also explored. Results were
very impressive, particular on consumer-grade GPGPUs. Later
a comprehensive implementation of streaming spectral analysis
opcodes was developed, and tested on two separate GPGPUs
with excellent results [37].

The over-reliance on a specific vendor, both in terms of software library and hardware, led us to explore alternatives. A portable option with open-source implementations is offered by the Open Computing language [38] (OpenCL) framework. For spectral processing, the main difficulty at the outset was that it
did not offer an off-the-shelf fast Fourier transform (FFT) im-
plementation (as CUDA did). Nevertheless, it appeared to be
the best avenue for further exploration. Moreover, OpenCL also
supports heterogenous computing, which may be useful in some
applications.

A highly efficient radix-2 FFT was thus developed as part
of the effort to implement time-varying convolution on the GP-
GPU [39]. With this, we were able to offer opcodes for both
time-varying convolution and linear time-invariant convolution,
as well as complex-to-complex and real-to-complex FFT,

```
out:a = clconv(in:a, table:i, size:i, dev:i)
out:a = cltvconv(in1:a, in2:a, freeze1:k, freeze2:k,
    partsize:i, size:i, dev:i)
out:k[] = clfft(in:k[], fwd:i, dev:i)
out:k[] = clrfft(in:k[], fwd:i, dev:i)
```

The code for the OpenCL FFT/convolution implementation
as well as the Csound opcodes can be found at

https://github.com/vlazzarini/opencl_ftt

A more generic co-processing approach has also been devel-
oped with a pair of OpenCL opcodes that serve as an interface to
the GPGPU. Taking advantage of the fact that OpenCL provides
a just-in-time compiler, we have put forward the idea that users
can embed kernel processing code in their application, and exe-
cute them directly on the device, with a means of sending inputs
and collecting the outputs in their Csound code.

A trivial example is shown below. The user specifies the
kernel as a string (in OpenCL code) and the clopd opcode is
used as the interface.

```
ksmps = 64
kernel@global:S = {
    kernel void process(global float *out,
                        global const float *in) {
        int i = get_global_id(0);
        out[i] = in[i]*in[i];
    }
}

clproc:InstrDef = create({
    dev:i = 0
    sig:a = oscili(p4, p5)
    sig = clopf.kernel, dev, ksmps, sig)
    out(sign)
})

schedule(clproc, 0, 10, 0dbfs, 100)
```

During execution, the kernel code is sent to each process-
ing cell defined by OpenCL on a given device (ID 0 in this case,
multiple devices may exist in a system). The number of threads
(one per cell) is in this simple example set to the number of sam-
ples in the audio vector (ksmps), but typically will be one or
two orders of magnitude larger, depending on the process being
implemented. The other opcode of the pair can be used if double-
precision floats are employed (and supported by the hardware).
Any number of inputs and outputs may be used, control, audio,
or numeric array types.

https://github.com/vlazzarini/clop

3.2. FPGAs

In the context of FPGAs, co-processing can be achieved by in-
terfacing Csound running on the PS with custom hardware mod-
ules running on the PL. On the PS side, we developed plug-in
opcodes fpgaOut, which sends audio data into the FPGA, and
fpgaIn, which receives audio data from the FPGA back into
Csound. These opcodes have the following signatures:

```
left:a, right:a = fpgain()
fpgout(left:a, right:a)
```

These opcodes can be used within Csound code with stan-
dard audio-rate variables, allowing, similarly to the OpenCL case,
FPGA interfacing directly with the Csound language. As with

the engine main IO buffers spin and spout, these opcodes are in-
ternally connected to an AXI DMA IP, and data is transferred to
and from the PL using the AXI DMA driver.

On the PL side, we created hardware ports of existing Csound
opcodes to leverage FPGA processing. This approach is ad-
vantageous because FPGAs allow a high degree of parallelism
and can perform ultra-low-latency, sample-by-sample process-
ing. They support a large number of GPIOs and are capable
of handling computationally intensive tasks with high through-
put [40]. These features make FPGAs especially suitable for
complex audio processing tasks such as spectral transformations,
large reverbs, high-channel-count processing, or physical mod-
elling, as demonstrated in [41] and [42].

However, programming FPGAs is generally difficult, requir-
ing specialised low-level hardware design knowledge. One solu-
tion to this challenge is the use of High-Level Synthesis (HLS)
techniques, which allow hardware designs to be generated from
C or C++ code. While HLS still requires certain hardware-
specific programming practices, it is considerably more accessi-
ble than traditional hardware description languages like VHDL
or Verilog [43, 40]. We use HLS to generate custom audio pro-
cessing IP cores which are like standalone modules that run on
the PL. Each of these modules can be considered a port of a
Csound opcode.

An example of this is the `reverbsc` opcode, which im-
plements an eight delay-line feedback network `reverb` based on
Sean Costello's original Csound orchestra version of the algo-
rithm [44]. To port this opcode into an IP core, the entire au-
dio processing functionality was wrapped in a single top-level
function, which was then synthesised into a hardware IP. The
arguments of this function define the communication interfaces,
while the function body contains all the signal processing logic.

In our system, all audio communication within the PL is con-
ducted with the AXI Stream protocol, as with the I2S IPs. Using
platform-specific data types and directives provided by the HLS
tool—Vitis HLS [45]—the input and output ports are defined as
AXI Stream interfaces. Parametric control is also supported, and
similar to the original `reverbsc` opcode, the FPGA hardware
version requires parameters such as feedback amount and damp-
ing frequency. These are set from within Csound on the PS and
transferred to the IP using the AXI Lite protocol. This interface
is also defined using HLS pragmas. The code excerpt below
shows the signature of the `reverbsc` top-level function:

```
void reverbsc(hls::stream<stream> &in,
              hls::stream<stream> &out, volatile float &fb,
              volatile float &cf)
{
    #pragma HLS INTERFACE axis port = in
    #pragma HLS INTERFACE axis port = out
    #pragma HLS INTERFACE s_axilite port = fb
    #pragma HLS INTERFACE s_axilite port = cf
    #pragma HLS INTERFACE ap_ctrl_none port = return
    ...
}
```

The complete HLS code is available at:

https://github.com/amanjagwani/ModFPGA/tree/
main/hardware/hls/ReverbSC

4. CONCLUSIONS

The effort in porting Csound to these different computing plat-
forms supports our aim towards *Csound Everywhere*. The sys-
tem is now available to run on small to large platforms, with or
without operating systems, on the web, as plugins in digital au-
dio workstations, on mobile phones, and use co-processing facil-
ities such as GPGPUs and FPGAS. Alongside these, other devel-
opments in the language, API, and engine appearing in Csound
7 have also enhanced its capabilities as a system.

With the support for bare-metal deployment, new opportu-
nities emerge for artistic, commercial, and scientific explo-
ration. Running Csound on small, portable, and low-cost mi-
crocontroller boards such as the Daisy provides flexibility and
mobility. This opens up possibilities for deploying Csound pro-
grams in installations, interactive sculptures, and embedded en-
vironments. New devices, synthesizers, pedals, and modular in-
struments can also leverage the extensive audio processing ca-
pabilities of Csound as part of their audio engine design. For
experienced Csound users, the transition is seamless—since the
bare-metal build uses essentially the same Csound source code,
existing applications can be directly adapted to new embedded
targets.

In terms of co-processing, combining Csound running on
a CPU with custom modules or processing kernels on GPGPU
or FPGA hardware enables accelerated, ultra-low-latency, and
highly parallel audio processing, within the compact footprint of
an embedded system. This hybrid approach allows users to re-
tain all of Csound's rich capabilities while leveraging specialised
hardware for performance-critical tasks. Another powerful use
case is to utilise Csound's generative capabilities, such as ran-
dom number generators, flexible probabilistic scheduling, and
algorithmic sequencing, to control synthesis or processing run-
ning on the FPGA that leverages the high-sampling rates, par-
allelism or low-latency. An example of a generative synthesis
design running on the Zynq-7000 SoC (Zybo Z7020 board) can
be found here:

https://youtu.be/sEWzhKcb3wc?si=
3xstMXMTkqE8QEzx

5. ACKNOWLEDGMENTS

Aman Jagwani would like to acknowledge the support of the
Hume Scholarship Programme, Maynooth University

6. REFERENCES

[1] B. Vercoe, “Computer system and languages for audio research,” *The New World of Digital Audio* (Audio Engineering Society Special Edition), pp. 245–250, 1983.

[2] Ian Bowler, Peter D. Manning, Alan Purvis, and Nick Bailey, “A transputer-based additive synthesis implementation,” in *Proceedings of the 1989 International Computer Music Conference, ICMC 1989, Columbus, Ohio, USA, November 2-5, 1989*. 1989, Michigan Publishing.

[3] Barry Vercoe, “Extended csound,” in *Proceedings of the 1996 International Computer Music Conference, ICMC 1996, Hong Kong, August 19-24, 1996*. 1996, Michigan Publishing.

[4] Victor Lazzarini, “A toolkit for music and audio activities on the XO computer,” in *Proceedings of the 2008 International Computer Music Conference, ICMC 2008, Belfast, Ireland, August 24-29, 2008*. 2008, Michigan Publishing.

[5] Victor Lazzarini, Joseph Timoney, and Shane Byrne, “Embedded sound synthesis,” in *Proceedings of the Linux Audio Conference 2015*, Johannes Gutenberg University, Mainz, Germany, 2015, pp. 1–6.

[6] “Raspberrypi,” https://www.raspberrypi.org/, Accessed: April 24, 2024.

[7] “Beagleboneblack,” http://beagleboard.org/black, Accessed: April 17, 2024.

[8] Andrew McPherson, “Bela: An embedded platform for low-latency feedback control of sound,” *The Journal of the Acoustical Society of America*, no. 3618, 2017.

[9] Victor Lazzarini, John ffitch, Joseph Timoney, and Russell Bradford, “Streaming spectral processing with consumer-level graphics processing units,” in *17th International Conference on Digital Audio Effects (DAFx-14)*, 2014, pp. 1–8.

[10] “Arm architecture,” https://www/arm.com/architecture/cpu, Accessed: April 28, 2024.

[11] Aman Jagwani, “Creative possibilities and customizability of live performance systems with open source programming platforms.,” in *Proceedings of the Ubimus 2023*, 2023.

[12] “Stm32 32-bit arm cortex mcus,” https://www.st.com/en/microcontrollers-microprocessors/stm32-32-bit-arm-cortex-mcus.html, Accessed: April 24, 2024.

[13] “ESP32,” https://www.espressif.com/en/products/socs/esp32, Accessed: April 24, 2024.

[14] M. Mulshine and J. Snyder, “Oops: An audio synthesis library in c for embedded (and other) applications,” in *Proceedings of the International Conference on New Interfaces for Musical Expression (NIME 2017)*, Copenhagen, Denmark, 2017.

[15] Electro-Smith, “Daisy,” https://www.electro-smith.com/daisy, 2023, Accessed: 2023-06-08.

[16] Xilinx, “Zynq-7000 soc,” https://www.xilinx.com/products/silicon-devices/soc/zynq-7000.html, 2023, Accessed: [2023-05-07].

[17] “Teensy USB development board,” https://www.pjrc.com/teensy/, Accessed: April 28, 2024.

[18] “Raspberry pi pico,” https://www.raspberrypi.com/products/raspberry-pi-pico/, Accessed: April 28, 2024.

[19] “STM32 nucleo boards,” https://www.st.com/en/evaluation-tools/stm32-nucleo-boards.html, Accessed: April 28, 2024.

[20] “Arduino,” https://www.Arduino.cc/, Accessed: 2024-02-16.

[21] “Csound on bela,” https://csound.com/site/news/2019/05/06/csound-on-bela, Accessed: April 28, 2024.

[22] “Csound 30 years maynooth - workshop csound and raspberry pi,” https://cosmoproject.github.io/Workshop_Maynooth/, Accessed: April 28, 2024.

[23] “libsndfile,” http://www.mega-nerd.com/libsndfile/, Accessed: April 28, 2024.

[24] “Electrosmith - memory: What is the difference?,” https://electro-smith.com/pages/memory-what-is-the-difference, Accessed: April 28, 2024.

[25] “libdaisy - hardware library for daisy audio platform,” https://github.com/electro-smith/libDaisy, Accessed: April 28, 2024.

[26] “Daisysp - digital signal processor library for daisy audio platform,” https://github.com/electro-smith/DaisySP, Accessed: April 28, 2024.

[27] Timothy Schoen, “Plugdata: A visual programming environment for audio experimentation, prototyping, and education,” https://plugdata.org/, 2025, Accessed: January 28, 2024.

[28] “pd2dsy - a tool for converting pure data patches to daisysp code,” https://github.com/electro-smith/pd2dsy, Accessed: April 28, 2024.

[29] “hvcc - heavy compiler collection,” https://wasted-audio.github.io/hvcc/, Accessed: April 28, 2024.
[30] Digilent, “Zybo z7 reference manual,” https://digilent.com/reference/programmable-logic/zybo-z7/start, 2023, Accessed: 2023-08-24.

[31] Advanced Micro Devices, Inc., I2S Transmitter and I2S Receiver LogiCORE IP Product Guide (PG308), 2024, Accessed: April 28, 2024.

[32] Xilinx, “Synthesis,” https://docs.xilinx.com/r/en-US/ug892-vivado-design-flows-overview/Synthesis, 2023, Accessed: 2023-08-18.

[33] “AXI DMA standalone driver,” https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842100/AXI+DMA+Standalone+Driver, Accessed: January 28, 2025.

[34] Xilinx, “Axi4 stream user guide,” https://docs.xilinx.com/r/en-US/ug1399-vitis-hls/Introduction, 2023, Accessed: 2023-02-18.

[35] Russell Bradford, John ffitch, and Richard Dobson, “Real-time Sliding Phase Vocoder using a Commodity GPU,” in Proceedings of ICMC2011. University of Huddersfield and ICMA, August 2011, ICMC, pp. 587–590, ISBN 978-0-9845274-0-3.

[36] Victor Lazzarini, John ffitch, Joseph Timoney, and Russell Bradford, “Streaming spectral processing with consumer-level graphics processing units,” in Proc. of the Int. Conf. on Digital Audio Effects (DAFx-14), Erlangen, Germany, 2014, pp. 1–8.

[37] Andrea Gianfranco Crespi, “Spectral manipulation of audio using general-purpose graphics processing units,” M.S. thesis, Politecnico di Milano, Milan, Italy, 2016.

[38] Khronos OpenCL Working Group, The OpenCL Specification, Khronos Group, Beaverton, USA, 2019.

[39] Victor Lazzarini, “Parallel computation of time-varying convolution,” Journal of New Music Research, vol. 0, no. 0, pp. 1–13, 2020.

[40] A.P. Jagwani, “Developing a modular sound synthesis platform for fpgas with high level synthesis techniques,” M.S. thesis, Maynooth University, Department of Music, 2023, Supervisor: Prof. Victor Lazzarini.

[41] Michon R. Risset, T. Cochard, P. Letz, S. Orlarey, Y. de Dinechin, F. Popoff, M., “Audio dsp to fpga compilation: The syfala toolchain approach,” Tech. Rep. RR-9507, Univ Lyon, INSA Lyon, Inria, CITI, Grame, Emeraude, May 2023.

[42] C. Wegener, S. Stang, and M. Neupert, “Fpga-accelerated real-time audio in pure data,” in Proceedings of the 19th Sound and Music Computing Conference, June 5-12 2022.

[43] Matai J. Neuendorffer, S. Kastner, R., “Parallel programming for fpgas,” https://arxiv.org/abs/1805.03648, 2018, ArXiv e-prints.

[44] Istvan Varga, reverbsc — 8 delay line stereo FDN verb, 2005, Accessed: January 28, 2024.

[45] Xilinx, “Vitis hls user guide,” https://docs.xilinx.com/r/en-US/ug1399-vitis-hls/Introduction, 2023, Accessed: 2023-02-18.
LAC-25 Paper Session 4
# ENHANCING INTERACTIVE COMMUNICATION WITH THE OVBOX: NEW FEATURES AND APPLICATIONS IN HEARING RESEARCH

Giso Grimm *

Auditory Signal Processing
and Cluster of Excellence “Hearing4all”
Department of Medical Physics and Acoustics
Carl von Ossietzky Universität Oldenburg, Germany
g.grimm@uol.de

Volker Hohmann

Auditory Signal Processing
and Cluster of Excellence “Hearing4all”
Department of Medical Physics and Acoustics
Carl von Ossietzky Universität Oldenburg, Germany
volker.hohmann@uol.de

## ABSTRACT

The OVBOX system provides low-latency, real-time transmission of audio and multi-modal data, facilitating interactive communication in a variety of applications. This paper presents new features, including data encryption, and demonstrates the capabilities of the OVBOX system through three exemplary applications. The first application involves streaming a live concert from four different locations, demonstrating the system's ability to handle complex, geographically distributed audio transmissions. The second application presents a hearing research study investigating the impact of head movement on communication success, highlighting the system's utility in scientific investigations. The third application explores the feasibility of conducting EEG hyperscanning studies using the OVBOX system, illustrating its potential for advancing neuroscience research. Our findings provide insights into the design of effective interactive communication systems and highlight the potential of the OVBOX system to advance our understanding of human communication and perception.

## 1. INTRODUCTION

With energy costs rising and borders closing, the importance of low-latency acoustic telepresence tools for music and voice communication cannot be underestimated. Telepresence technology allows people to communicate over the internet. This is usually achieved using established videoconferencing tools such as Zoom, Facetime, Jitsi or many of the other tools that received a lot of attention during the COVID19 pandemic. When it comes to music and research applications, a limiting factor of video-conferencing tools is the delay between the sender and receiver (end-to-end delay), as well as the use of noise reduction, echo cancellation and sound gating, which prevent natural turn taking in speech conversation and rhythmic musical interaction.

In face-to-face communication, the delay between sender and receiver is limited only by the speed of sound and the distance between them. It typically reaches values between 3 ms, corresponding to a distance of 1 m in a typical speech conversation, and 30 ms, corresponding to a distance of 10 m, which can occur between musicians on stage. For acoustic music, this is also the upper limit for unhindered rhythmic interaction [1]. The transition time of turns in speech communication is typically in the order of 300 ms [2], which is much less than the latency of speech production. This means that in interactive communication the prediction of turn takes is crucial. This requires low end-to-end delays as well as the transmission of the cues necessary for prediction [2, 3].

Telepresence systems typically rely on an Internet connection that is shared with many other users and applications. Given the uncontrollable nature of Internet delays, minimising latency at both the sender and receiver ends is critical. This has led to a number of telepresence tools that specialise in delay minimisation. An overview of existing solutions for music applications

can be found in [4]. Solutions with the shortest latencies often run on dedicated hardware running the Linux operating system [5, 6, 7]. Low-latency audio transmission for acoustic interaction is conceptionally different from distributed recording systems (e.g. WAM-studio, [8]) or telematic music systems based on event triggers or pre-scored (e.g., L2ORK tweeter, [9]), just to mention a few examples from the Internet of sounds literature [10]. Such systems are therefore not further discussed in this paper.

The aim of this paper is to report on extensions to the open source tool OVBOX [11, 7], which was first presented at LAC2020. The OVBOX was developed to enable rhythmic interaction of music with acoustic sources using the Internet. In this paper we describe the extension to support low-latency transmission of user-provided UDP data and timestamps for post-hoc transmission time correction, and the extension to support end-to-end encryption of audio and user data. Both of these extensions are essential for hearing research applications. Therefore, in addition to these extensions, we highlight three example applications: a concert streamed live from different locations, and two applications in the fields of hearing research and neuroscience.

## 2. OVERVIEW OF THE OVBOX SYSTEM

In response to the increased need for remote music collaboration tools during the COVID-19 pandemic, we developed the OVBOX system [7, 12], a system designed for low-latency network audio communication. The OVBOX system overcomes the limitations of existing tools such as Zoom and Jamulus by achieving end-to-end delays below 30 ms for distances up to approximately 1500 km (see [7] for a detailed delay and jitter analysis of different network technologies), making it suitable for low-latency communication applications in music and research.

The system architecture comprises client devices, a configuration server and session servers, using a client-server model for remote configuration and traversal of network address translation barriers, see Figure 1 for an overview. The client software, implemented in C++, integrates a Representational State Transfer Application Programming Interface (REST API) client, a User Datagram Protocol (UDP) forwarding tool, a network audio client [13], and the TASCAR acoustic simulation engine for 3D spatial audio rendering [14, 15]. This setup allows for individualised audio mixing and spatial rendering, which is particularly beneficial for musicians playing instruments with similar timbres.

Optimised for headless operation on a Raspberry Pi 4B, pre-configured images for the OVBOX client are offered to ensure low-latency audio processing, a feature that is difficult to achieve on typical desktop systems. However, desktop client packages are available for Ubuntu Linux, Archlinux, MacOS and, on an experimental basis, Microsoft Windows. The system supports both Selective Forwarding Unit (SFU) and Peer-to-peer (P2P) network topologies. Jitter compensation is user adjustable to balance low-latency and packet loss.

<footnote>* Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - Project-ID 352015383 - SFB 1330 B1</footnote>
<figure><img src="image_49.png" /><figcaption>Figure 1: *Server structure of the OVBOX system. The OVBOX devices request their configuration from the configuration server, and the session servers advertise their services to the configuration server (green lines). When a device joins a session, it establishes a session management connection with the session server and exchanges its IP address, port number, operating mode and public encryption key (blue lines). Finally, all devices within a session can exchange audio and other user data via encrypted UDP messages (red line).*</figcaption></figure>

Performance evaluations show that the OVBOX system achieves high computational efficiency. In a comparison including a Mac Mini (M1), Linux Desktop (i5), Raspberry Pi 3B+ and 4B, the Mac Mini was leading in CPU usage, followed closely by a Linux desktop and Raspberry Pi 4B. The Raspberry Pi 3B+ shows limitations with smaller block sizes, highlighting the importance of hardware selection.

<table><caption>Table 1: *CPU usage and self-monitoring round trip time in a session with four peers. Data from [7].*</caption><thead><tr><th rowspan="2">Hardware</th><th colspan="2">2 ms block size</th><th colspan="2">1 ms block size</th></tr><tr><th>CPU %</th><th>RTT ms</th><th>CPU %</th><th>RTT ms</th></tr></thead><tbody><tr><td>Raspberry Pi 3B+</td><td>62</td><td>9.25</td><td>n.a.</td><td>5.4</td></tr><tr><td>Raspberry Pi 4B</td><td>24</td><td>9.13</td><td>34</td><td>5.13</td></tr><tr><td>Linux i5-8500 (5.15.0-97-lowlatency)</td><td>8.2</td><td>6.04</td><td>15</td><td>3.9</td></tr><tr><td>mac mini M1 2020 (monterey 12.7.3)</td><td>4.6</td><td>9.04</td><td>5.7</td><td>7.04</td></tr></tbody></table>

The end-to-end delay between two peers, which can typically be achieved over distances of up to 1500 km using fibre optics or DSL technology with default settings, is in the range of 30 to 50 ms. For example, 30 ms end-to-end delay has been reported for connections between Pristina (Kosovo) and Oldenburg (Germany) and between Rome (Italy) and Kiel (Germany), and 40 ms for connections between Paris (France) and Oldenburg (Germany) as well as Budapest (Hungary) and Oldenburg (Germany). A detailed analysis of delays as well as an open data set can be found in [7].

## 2.1. Data Routing, Message Handling and Session Management

The core of the OVBOX is a data router that packs and forwards UDP packets from the local network to the other peers, either directly from peer to peer or via the session server, depending on the operating mode. A packet consists of a header containing a session-wide secret code, the sender identification, the destination port number and a packet counter. The rest of the packet is the original data packet. When a message is received on the external port, the header is decoded. If the session code matches, the packet is sorted to reduce the number of out-of-order messages and finally forwarded to the appropriate local port.

Data packets with reserved special port numbers are used for special purposes, such as measuring round trip time and communicating with the session server. Each client signals its presence to the session server by sending periodic registration messages. The session server shares the IP address, external port number and operating mode with all devices participating in the same session. Devices are considered to be offline if no registration or data message has been received for several seconds.

## 2.2. Audio Transmission

The open source tool 'zita-njbridge' [13] is used for network encoding and decoding of audio as well as jitter buffer handling. On the sender side the OVBOX client software launches an instance of 'zita-j2n' with as many channels as the device should send. The destination port number depends on the session ID of the device, which is unique within a session. For each remote client participating in the same session, an instance of 'zita-n2j' is started, listening on the port number assigned to the remote session ID.

## 2.3. User Data Transmission

User data can be forwarded to all remote devices of the session in the same way as audio data is transmitted, since the data router is completely agnostic of the packet content. The only difference is that the forwarded port needs to be configured explicitly (using the configuration web interface). This way any unicast UDP-based data, e.g., OSC messages, can be shared within a session with little effort.

In neuroscience research, the Lab Streaming Layer (LSL) is commonly used to exchange EEG signals, trigger events or other biophysical sensor data [16]. This protocol uses a mixture of multicast and unicast UDP messages, which cannot be routed directly through the OVBOX system. To enable the transmission of LSL data streams and events, a bidirectional converter between LSL and OSC was developed and added to the system. Although not all the metadata embedded in LSL data streams can be transferred in this way, it is possible to exchange LSL data within an OVBOX session.

Two special ports are allocated by the OVBOX process: UDP port 9871 is used to receive OSC messages, which control features of the local acoustic simulation tool, including any extensions (see section 2.5 for details). Data sent to UDP port 9870 will be forwarded to port 9871 of all other peers, but will not be processed locally.

## 2.4. Encryption

The OVBOX system uses libsodium [17], a modern and efficient cryptographic library, to implement data encryption using key pair encryption. This method uses asymmetric cryptography, where a public key is used for encryption and a corresponding private key for decryption, to ensure secure data transmission. The session server plays a key role in managing the exchange of public keys between clients, facilitating the establishment of secure communication channels without prior contact between the parties. Encrypted data is sent to all devices for which a public key is available. The encryption status is displayed in the user interface on the configuration server.

By using encryption, each data package is increased in size by 32 Bytes. This results in an increase of overall data usage of approximately 22%; for a measurement see Table 2. Despite the added security, the system maintains minimal data and computational overhead, ensuring efficient operation even on resource-constrained devices such as the Raspberry Pi single-board computer. This balance of security and performance is critical for real-time applications where both data integrity and low latency are essential. By using libsodium's optimised algorithms, the
OVBOX system achieves seamless and secure communication without breaking compatibility with previous versions.

<table><caption>Table 2: Data usage of an OVBOX device in a session with three participants. All parameters except encryption and operating mode were left at their default settings.</caption><thead><tr><td>Encryption</td><td>Operating mode</td><td>Upstream MBit/s</td><td>Downstream MBit/s</td></tr></thead><tbody><tr><td>off</td><td>peer-to-peer</td><td>1.78</td><td>1.78</td></tr><tr><td>off</td><td>server</td><td>0.9</td><td>1.78</td></tr><tr><td>on</td><td>peer-to-peer</td><td>2.17</td><td>2.17</td></tr><tr><td>on</td><td>server</td><td>1.09</td><td>2.17</td></tr></tbody></table>

## 2.5. Extending the OVBOX Functionality

The underlying acoustic simulation tool, TASCAR, provides laboratory management tools that go far beyond acoustic simulation, e.g. data logging, sensor integration and OSC interfaces that allow almost any feature to be controlled in real time. This allows most of the features of TASCAR to be used in the OVBOX. In addition to the generic acoustic scene controlled by the OVBOX system, it is possible to add any TASCAR configuration written in XML code.

### 2.5.1. Example: Controlling DMX Lamps remotely

For example, it is possible to add a USB based DMX controller and control the connected 3 channel DMX controllable RGB fixtures with the following additional TASCAR code, entered via the configuration web interface:

```xml
<?xml version="1.0"?>
<session>
  <modules>
    <lightctl driver="opendmxusb"
              device="/dev/ttyUSB0">
      <lightscene name="scene"
                  channels="3">
        <layout>
          < fixture addr="15"
                    label="lamp1"/>
          < fixture addr="30"
                    label="lamp2"/>
        </layout>
      </lightscene>
    </lightctl>
  </modules>
</session>
```

To control 'lamp1' to fade to green from any other client, the RGB or HSV values and a fade time can be sent via OSC:

```
send_osc osc.udp://192.168.0.50:9870/
    /light/scene/lamp1/fade 0 255 0 1
send_osc osc.udp://192.168.0.50:9870/
    /light/scene/lamp1/hsv 120 1 1 1
```

Here we assume that the remote OVBOX client device has the IP address 192.168.0.50; if the desktop client is used, the OSC messages can be sent locally. Similarly, most other TASCAR modules can be integrated by adding the appropriate configuration. All plugins are documented in the TASCAR documentation [18].

### 2.5.2. Example: Sending Time Stamps and LSL Streams

To send time stamps from two devices to a central data logging, we assume a setup with three devices. One of the device is a desktop client running on the same computer as the data logging. On the first device, enter this additional TASCAR code:

```xml
<?xml version="1.0"?>
<session>
  <modules>
    <lsl2osc streams="EEG"
              prefix="/lab1"/>
    <system path="/cal"
              seepath="/lab1/time"
              sendsesstiontime="true"/>
    <oscrelay startwith="/lab1"
              url="osc.udp://localhost:9870/"/>
  </modules>
</session>
```

On the second device, the same code can be used but with 'lab1' replaced by 'lab2'. On the data logging client (desktop GUI client), this configuration is required to record data in Matlab format:

```xml
<?xml version="1.0"?>
<session>
  <modules>
    <datalogging outputdir="${HOME}">
      <osc path="/lab1/EEG" size="33"/>
      <osc path="/lab2/EEG" size="33"/>
      <osc path="/lab1/time" size="2"/>
      <osc path="/lab2/time" size="2"/>
    </datalogging>
  </modules>
</session>
```

We assume that a 32 channel EEG LSL stream is available on the subnet of OVBOX 1 and 2; the LSL time stamp is additionally sent in the first channel, thus resulting in a total of 33 channels. The time line of the sending devices can be controlled via OSC on the data logging computer:

```
send_osc 9870 /transport/locate 0
send_osc 9870 /transport/start
```

The timestamps now include the remote session time and the system clock in seconds since midnight. A screenshot of the data logging window is shown in Figure 2.

<figure><img src="image_50.png" /><figcaption>Figure 2: Screenshot of the data logging window during EEG stream and timestamp recording, with one device active and one inactive. The time line of the sending device was started at approximately second 25.</figcaption></figure>
# 3. EXAMPLE APPLICATIONS

## 3.1. Streaming Concert played live from different Locations

The OVBOX system is actively used by many ensembles and individuals for online rehearsals, teaching, streaming concerts or international master classes [19]. In this section we describe the setup used for a concert for five viola da gamba, played live from four different locations [20], see Figure 3 for a screenshot.

During the pandemic, concerts were put on hold due to lockdowns, leaving both musicians and audiences unable to gather. To bridge this gap, streaming concerts became a popular alternative. In one such concert, five musicians from different locations in northern Germany (Hamburg, Bremen, Oldenburg and Ottersberg) connected via the internet to perform together. The musicians’ locations were connected using a variety of technologies, ranging from direct Ethernet connections to fibre optics and DSL. For two of the locations, the audio was routed through a central server in Frankfurt/Main to optimise the use of the bandwidth required for the low quality internet connection.

At one of the sites, the audio was not only used for live performance, but also remixed with a physical model of a concert hall using TASCAR [14]. This mixed audio was then streamed live on YouTube, allowing a wider audience to experience the concert in real time. To provide a visual experience as well as the audio, videos of each musician were streamed to the streaming location using jitsi [21]. A background video with a dome projection was rendered using the Blender game engine [22] and mixed with the musician videos and the stereo mix of the audio using OBS Studio [23], streamed as a video stream in 1440p resolution. In the youtube studio the stream was configured as a 360 degree interactive video. Unfortunately, ambisonics audio is not supported in Youtube live streams, so only a stereo mix was used, regardless of the current viewing angle. The musicians did not see any video.

In the OVBOX system, a second instance of zita-n2j was started for each musician, with a larger jitter compensation buffer size, set to 50 ms instead of 10 ms, for the broadcast audio stream, as a slight additional delay in the streamed audio is not critical (compared to the audio for direct musical interaction). An additional audio delay of 400 ms was added in the mix to achieve a rough temporal alignment of the jitsi videos with the audio signal.

<figure><img src="image_51.png" /><figcaption>Figure 3: Screenshot of a streaming concert for five viola da gamba, played live from four different locations [20].</figcaption></figure>

## 3.2. Evaluation of Hearing Aids in Interactive Conversation

We have used the OVBOX system for several applications in hearing aid and communication research to achieve telepresence in virtual reality [24, 25, 26]. Participants were placed in a virtual reality lab with a 300-degree cylindrical projection screen and a 45-channel spherical loudspeaker array. The confederates

were in separate rooms in the same building and wore headphones with a headset microphone. The focus was on the transmission of multiple modalities along with the audio, such as head movements and gaze behaviour. A schematic overview of this setup is shown in Figure 4.

<figure><img src="image_52.png" /><figcaption>Figure 4: Overview of the setup to measure the effect of head movements on conversation success. The test participant is seated in a virtual reality lab with a 300 degree field-of-view video projection within a 45-channel spherical loudspeaker array (top left part of the figure). The two confederates were located in different rooms (bottom right part of the figure). Their audio signal and head movement data was transmitted to the laboratory using the OVBOX system. Depending on the measurement condition, different methods of avatar control were used.</figcaption></figure>

One of the studies based on the OVBOX system investigated how avatar head movement animation affects immersion and conversational success in virtual environments [25]. The aim is to improve evaluation methods for assistive listening technology, in particular to develop evaluation paradigms in which test subjects display conversation-specific gaze and head movement behaviour. Key to such paradigms is the evaluation of hearing aids in interactive conversations. The use of telepresence and avatar representation of interlocutors can disentangle the underlying factors of conversational behaviour.

Participants engaged in triadic conversations with two confederates in a simulated virtual pub environment, with their interlocutors represented by avatars. Four levels of avatar animation realism were tested: static heads, automatic head movements controlled by onsets in the audio signal, transferred head movements, and transferred video. In this study, the OVBOX system was used to transmit audio and head movement data, see Figures 4 and 5 for an overview. The head motion data was captured using an open hardware head tracker based on an MPU6050 motion processor connected to an ESP8266 WiFi microcontroller. The head movement data was used locally to control the orientation of the virtual receiver, resulting in an interactive binaural rendering of the virtual acoustic scene. Simultaneously, the head movement data was transmitted to the avatar representations presented to the participant and to a central data logging system. The experiment was controlled by OSC messages sent from a node.js web server. In this way, the entire experiment was controlled by one of the confederates via a web interface.

The results show that when representing the interlocutors with avatars, the animation level of the head movements has a
<figure><img src="image_53.png" /><figcaption>Figure 5: Example image of a test participant in the lab (top panel), and the two confederates connected via OVBOX systems (bottom panel).</figcaption></figure>

significant effect on the participant's speech behaviour as well as head movement behaviour, but also influences the success of the conversation (for details see [27]).

### 3.3. EEG Hyperscanning

This paper presents an innovative application of the OVBOX system, extending its capabilities to include EEG hyperscanning for real-time, low-latency transmission over the Internet [28]. EEG hyperscanning involves the simultaneous recording of brain activity from multiple people in order to study interpersonal neural synchronisation during social interactions. The OVBOX, originally designed for low-latency audio transmission, has been adapted to transmit EEG data, motion sensors and audio signals, enabling telepresence scenarios for research in social neuroscience and beyond.

The study demonstrates the real-time transmission of EEG data alongside audio and motion data. It also proves that low-latency transmission is possible: The system achieves minimal delay and jitter, which is critical for synchronising data from geographically separated participants, ensuring accurate analysis of interpersonal neural interactions. In the study, feasibility of using the OVBOX in a variety of tasks is shown: The paper presents results from four experimental conditions – response to tone, hand clapping, forced turn-taking, and free conversation-demonstrating the system's versatility and effectiveness in capturing synchronised neural activity and communication behaviour.

The system enables researchers to study brain activity during real-time interactions, providing insights into interpersonal synchrony and communication dynamics. The integration of EEG hyperscanning into a low-latency audio system opens new avenues for the study of musical interactions and ensemble performance. The capabilities of the system suggest applications in virtual environments, enhancing immersive experiences and enabling new forms of remote collaboration and interaction.

<figure><img src="image_54.png" /><figcaption>Figure 6: Overview of the setup for EEG hyperscanning. The two participants were located in two different labs, the data were recorded in a third lab. Next to the audio signals, EEG data and clock signals were transmitted, to allow an exact measurement of the end-to-end delays.</figcaption></figure>

## 4. CONCLUSIONS

The OVBOX system is designed for low-latency interactive communication over the Internet. The two main applications are in music, to enable distributed rehearsals and hybrid concert formats, and in hearing and communication research, to transmit audio and behavioural data in real time. Key features are both flexibility and robust data security. The effectiveness of the OVBOX system is demonstrated by three example studies presented here. The incorporation of end-to-end encryption for both audio and user data resulted in a marginal increase in data usage with no significant impact on computational performance.

## 5. REFERENCES

[1] Cristina Rottondi, Michele Buccoli, and Massimilliano Zanoni, “Feature-based analysis of the effects of packet delay on networked musical interactions,” *Journal of the Audio Engineering Society*, vol. 63, no. 11, pp. 864–875, dec 2015.

[2] Stephen C. Levinson and Francisco Torreira, “Timing in turn-taking and its implications for processing models of language,” *Frontiers in Psychology*, vol. 6, June 2015.

[3] Lauren V. Hadley and John F. Culling, “Timing of head turns to upcoming talkers in triadic conversation: Evidence for prediction of turn ends and interruptions,” *Frontiers in Psychology*, vol. 13, dec 2022.

[4] Patrick Müller, Roman Haefeli, Johannes Schütt, and Matthias Ziegler, “Telemersive audio systems for online jamming,” in *2024 IEEE 5th International Symposium on the Internet of Sounds (IS2)*. Sept. 2024, vol. 1, pp. 1–15, IEEE.

[5] Chris Chafe and Scott Oshiro, “Jacktrip on raspberry pi,” in *LAC-Linux Audio Conference*, 2019.

[6] Christian Hoene, Ian Howell, and Alexander Carôt, “Networked music performance: Developing soundjack and the fastmusic box during the coronavirus pandemic,” in *Audio Engineering Society Conference: 2021 AES International Audio Education Conference*. Audio Engineering Society, 2021.
[7] Giso Grimm, “Interactive low delay music and speech communication via network connections (OVBOX),” *Acta Acoustica*, vol. 8, pp. 1–7, Apr. 2024.

[8] Michel Buffa and Samuel Demont, “Can you daw it on-line?: The challenges of a web-based open source work-station,” in *2024 IEEE 5th International Symposium on the Internet of Sounds (IS2)*, 2024, pp. 1–8.

[9] Ivica Bukvic, “Latency-, sync-, and bandwidth-agnostic tightly-timed telematic and crowdsourced musicking made possible using 12ork tweeter,” in *NIME 2022. PubPub*, 2022.

[10] Luca Turchet, Mathieu Lagrange, Cristina Rottondi, György Fazekas, Nils Peters, Jan Østergaard, Frederic Font, Tom Bäckström, and Carlo Fischione, “The internet of sounds: Convergent trends, insights, and future directions,” *IEEE Internet of Things Journal*, vol. 10, no. 13, pp. 11264–11292, July 2023.

[11] Giso Grimm and Volker Hohmann, “ovbox: Remote music and speech communication with interactive low-delay virtual acoustics using the open source toolbox TASCAR,” in *Proceedings of the Linux Audio Conference*, 2020.

[12] Giso Grimm, “ovbox - ORLANDOviols consort box,” https://ovbox.de/, 2024.

[13] Fons Adriaensen, “Controlling adaptive resampling,” in *Linux audio conference, Stanford, USA*, 2012.

[14] Giso Grimm, J. Luberadzka, and V. Hohmann, “A toolbox for rendering virtual acoustic environments in the context of audiology,” *Acta Acustica united with Acustica*, vol. 105, pp. 566–578, 5 2019.

[15] Giso Grimm, “Tascar toolbox for acoustic scene creation and rendering,” https://tascar.org/, 2025.

[16] Christian Kothe, Seyed Yahya Shirazi, Tristan Stenner, David Medine, Chadwick Boulay, Matthew I. Grivich, Tim Mullen, Arnaud Delorme, and Scott Makeig, “The Lab Streaming Layer for synchronized multimodal recording,” Feb. 2024.

[17] Frank Denis, “libsodium,” https://github.com/jedisct1/libsodium.

[18] Giso Grimm, Joanna Luberadzka, Fenja Schwark, Tobias Herzke, and Volker Hohmann, *TASCAR user manual*, Carl von Ossietzky Universität Oldenburg, 2025, https://github.com/gisogrimm/tascar/wiki/master/manual.pdf.

[19] Netzwerk 4.0 der Musikhochschulen, “barock:connect - hochschulübergreifender digitaler Meisterkurs Barockvioline,” https://www.youtube.com/watch?v=U2Bdn_KpTps, 2024.

[20] Frauke Hess, Júlia Vető, Hille Perl, Marthe Perl, Giso Grimm, Claas Harders, and David Grimm, “The gentle true spirit - streaming concert,” https://www.youtube.com/watch?v=p_GSfDVmFfg, 2020.

[21] Jelena Caiko, Antons Patlins, Arapov Nurlan, and Vladimír Protsenko, “Video-conference communication platform based on webrtc online meetings,” in *2020 IEEE 61st International Scientific Conference on Power and Electrical Engineering of Riga Technical University (RTUCON)*. Nov. 2020, pp. 1–6, IEEE.

[22] Ton Roosendaal and Carsten Wartmann, *The Official Blender Game Kit: Interactive 3d for Artists*, No Starch Press, 2003.

[23] Lain Bailey, “OBS studio,” https://obsproject.com/, 2012.

[24] Marie Hartwig, Volker Hohmann, and Giso Grimm, “Speaking with avatars - influence of social interaction on movement behavior in interactive hearing experiments,” 2021, pp. 94–98.

[25] Giso Grimm, Angelika Kothe, and Volker Hohmann, “Effect of head motion animation on immersion and conversational benefit in turn-taking conversations via telepresence in audiovisual virtual environments,” in *Proceedings of the 10th Convention of the European Acoustics Association Forum Acusticum 2023*. Jan. 2023, FA2023, European Acoustics Association.

[26] Giso Grimm, Hendrik Kayser, Angelika Kothe, and Volker Hohmann, “Evaluation of behavior-controlled hearing devices in the lab using interactive turn-taking conversations,” in *Proceedings of the 10th Convention of the European Acoustics Association, Forum Acusticum 2023*. Jan. 2023, European Acoustics Association.

[27] Angelika Kothe, Volker Hohmann, and Giso Grimm, “Effect of avatar head movement on communication behaviour, experience of presence and conversation success in triadic conversations,” 2025.

[28] Giso Grimm, Mareike Daeglau, Volker Hohmann, and Stefan Debener, “EEG hyperscanning in the internet of sounds:low-delay real-time multi-modal transmissionusing the OVBOX,” in *2024 IEEE 5th International Symposium on the Internet of Sounds (IS2)*. Sept. 2024, pp. 1–8, IEEE.
LAC-25 Paper Session 5
INSIDE ANDREA MOLINO'S
LA VÉRITÉ, PAS TOUTE:
THE CREATION OF SWARM SOUNDS

Nicola Bernardini

Conservatorio "O.Respighi"
Latina, Italy
bernardini@conslatina.it

Luca Richelli

Conservatorio "A.Pedrollo"
Vicenza, Italia
luca.richelli@consvi.it

Alvise Vidolin
Alessandro Fiordelmondo

Centro di Sonologia Computazionale
Università di Padova
Padova, Italy
{vidolin|fiordelmondo}@dei.unipd.it

Mattia Pizzato

Sound and Music Processing Lab (SaMPL)
Conservatorio "C.Pollini"
Padova, Italy
pizmat.7@gmail.com

ABSTRACT

The paper is a technical description of a specific component of *La Vérité, pas toute*, a large scale dense and theatrically engaging work by Italian composer Andrea Molino.

Specifically, it outlines the tools and techniques developed to digitally create the “sonic swarms” that are called for in the score. Such swarms are generated, controlled and spatialized in real time and, as such, they present an interesting use case for live electronics environments.

1. INTRODUCTION

*La Vérité, pas toute* is a work commissioned by the Accademia Chigiana in Siena to the italian composer Andrea Molino in 2022. Written in 2022-2023, it is scored for 32 voices, 8 percussions, 16 mobile videocameras, fixed media recordings and live electronics. It was premiered during the Accademia Chigiana International Festival on the 14th of July of 2023.

It is a very dense and theatrically engaging work which centers around the use of words in spoken languages, traversing in total continuity the whole gamut of perceptual attributes ranging from sense and meaning down to their rhythmic and sonic properties. The concert setting of the piece is also unusual, with the public seating in the middle of the hall broadly facing each other, the percussion players distributed around the public, the conductor in the middle, the choir moving around and assuming different positions along the piece and a host of mimes who run around the hall holding small videocameras and videorecording specific details of the piece.

Among the many musical devices used to explore the dual valence of words there is a quite peculiar one, created digitally by the crew who built the performance environment for the piece. The request of the composer was to create some subliminal continuous sonic textures that would fill up the musical interstices of sections 1, 6 and 7 of the work (cf. for example Fig.1). Described by the composer as “swarms of phonetic materials flying around in the room” and hence called “swarms” during the development of the tools, these textures were created from some sonic vocal material defined and designed by the composer: short phrase fragments read by elements of the choir within different paralinguistic modes (murmurs, mutterings, blows, pants, etc.). These fragments were all recorded with extreme close-miking techniques in a soundproof booth (cf. Sec.2.2). They constituted a large database of some 800 sonic grains subdivided in various types and emission modalities.

An algorithmic tool was then created in *Pure Data* (cf.[1, 2, 3, 4, 5]); this tool is able to generate constantly different sonic swarms playing back with controlled casuality the fragments from different partitions of the database. This tool is entirely controlled in real time and as such it may be played live; it is constituted by a dynamic crop of up to 256 simultaneous generators which may read and reproduce the sonic grains thus creating swarms for which it is possible to control the density through progressive rescaling of the relative amplitudes of each grain.

The spatial disposition of the swarms may be controlled too, ranging in continuity from a totally “immersive” listening (all around the public) to a localised one virtually travelling through space.

Although this paper will focus only on these sonic swarms, the electronic part of *La Vérité, pas toute* also includes:

* spoken fragments of famous people voices

* virtual spatialization of sound sources

* dynamic miking

2. SWARMING SOUNDS

Swarms of sounds have long been the focus of many artists and researchers as a way to produce rich, moving and musical sound textures (cf.[6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]). The development of electronic “swarming” sounds focuses on creating complex, dynamic, and organic audio textures that mimic the behavior of natural swarms (e.g., insects, birds, or fish).

Key techniques have included to date:

1. **Algorithmic Composition:** Using generative algorithms (e.g., particle systems, boids, or cellular automata) to simulate swarm behavior, which is then mapped to sound parameters like pitch, amplitude, and spatialization.

2. **Time Granulation:** Breaking sound into tiny "grains" and manipulating them to create dense, evolving textures that emulate the chaotic yet cohesive nature of swarms.

3. **Ambisonics or binaural rendering:** to create immersive, moving soundscapes that replicate the spatial dynamics of swarms.

4. **AI and Machine Learning:** Leveraging neural networks to generate or enhance swarm-like sounds, either by training on natural swarm recordings or by learning patterns from synthesized data.

5. **Interactive Systems:** Integrating real-time input (e.g., motion sensors or MIDI controllers) to allow performers or audiences to influence swarm behavior and sound dynamically.

<figure><img src="image_55.png" /></figure>
<figure><img src="image_56.png" /><figcaption>Figure 1: La Vérité, pas toute, pag.1 (upper half)</figcaption></figure>

6. **Hybrid Approaches:** Combining synthesized sounds with processed field recordings of actual swarms to achieve a balance between realism and artistic expression.

These techniques have been widely used in film scoring, video game sound design and experimental music, pushing the boundaries of how swarming sounds can be perceived and experienced.

In *La Vérité, pas toute* only a small subset of these techniques have been used, aiming to maximize the expressivity and reliability of sound production in a mission-critical environment (large, long-range multi-dimensional music work). In particular, techniques 1, 2 and 6 have been used extensively, while sound spatialization (3) has been applied in *a posteriori* with a separate spatialization engine (cf. Sec.3, and [17, 18]).

<figure><img src="image_57.png" /><figcaption>Figure 2: abstract scheme of the swarm generator</figcaption></figure>

## 2.1. Technical description

Fig.2 summarizes the swarm generator algorithm of *La Vérité, pas toute*.

Everything centers around a configuration manager which reads a configuration text file which looks like this:

<table><thead><tr><th></th><th></th><th>sciami.conf</th></tr></thead><tbody><tr><td>1</td><td>sciami_1_1</td><td>3 1 2 3</td></tr><tr><td>2</td><td>sciami_1_2</td><td>1</td></tr><tr><td>3</td><td>sciami_6_1</td><td>1</td></tr><tr><td>4</td><td>sciami_6_2</td><td>3 1 2 3</td></tr><tr><td>5</td><td>sciami_7_1</td><td>4 1 2 3 4</td></tr><tr><td>6</td><td>sciami_7_2</td><td>4 1 2 3 4</td></tr></tbody></table>

The configuration file is a simple space-separated list of configurations (one per line) which get activated by the performer according to the location in the score.

The parameters of each line are as follow:

1. a string corresponding to the name of the sound folder whose content should be loaded into the sound file table; for example: `sciami_1_1` is the name of the folder used for section 1.1 of the score, etc.

2. the number of parallel swarms to be launched for the section

3. the list of specific swarm types to be launched (the list length should correspond to the number of parallel swarms)

We provide a “reading” of the first line as a matter of example:

1. (first field): the sound file path points to the `sciami_1_1` folder, which contains the `sciami_1_1.wav` file and three cue lists for three different sets of sounds (cf.2.2);

2. (second field): this is the number of sets played in parallel by the system

3. (following fields): these are the indexes of the cue list files to be used to read the different sets of files (in this case: `sciami_1_1_cue_list_1.txt`, `sciami_1_1_cue_list_2.txt` and `sciami_1_1_cue_list_3.txt`)

Each line gets interpreted by a simple configuration parser and the various tokens are sent to `creator` subpatches which create dynamically the number of `fragment_reader` abstract-
tions needed for each specific section. These are responsible for loading the cue lists for each voice in memory. Simultaneously, the interpreted configuration line is also sent to a `generator_manager` which dynamically allocates the specified number of `generator` abstractions. These will in turn spawn a variable number of parallel clones of table reader abstractions which are actually responsible of reading small sample fragments.

These table reader abstractions are actually the most intricated patches; their schematic workflow is outlined in Fig.3.

<figure><img src="image_58.png" /><figcaption>Figure 3: Schematic workflow of the table reader abstractions</figcaption></figure>

A `random picker` abstraction selects with appropriate tendency masks a specific fragments whose parameters (file, offset skip, duration) are passed to a `tabread4` object which will actually read the sound file. This selection is also time-correlated with the other clone readers to provide the “swarming” effect.

The amplitude of each sound fragment is rescaled with the following equations

$$amp = 4^{(apot_v + b)/127} \quad (1)$$

$$r_v = 1 - 2^{n_v/127} \quad (2)$$

where $pot_v$ is the (MIDI) value coming from the potentiometer ($0 <= pot_v <= 127$) and {$a = -0.4881, b = 116$} are linear function factors derived by simulated annealing. The resulting amplitude $amp$ is further rescaled according to the specific voice clone number that is being used ($n_v$) with the rescaling function (2), leading to the individual voice rescale factor $r_v$. The plots in Fig.4 illustrate the actual rescaling mechanism.

<figure><img src="image_59.png" /><figcaption>Figure 4: Density compensation functions</figcaption></figure>

The combination of functions (1) and (2) allow to simulate effectively a sort of “solo vs. tutti” effect compensating amplitude with density (fewer voices get an amplitude boost and vice versa).

The generated voices can then be routed to 5 different outputs via an `output_pipeline` abstraction as follows:

1. one of 8 channels routing (see below)

2. 4 independent mono channels

The 8-channel routing allocates the output channel according to a round-robin mechanism (the abstraction `channelize`) which will distribute each different voice on a separate channel with a $c = (n_v + k) \mod 8$ algorithm, where $k = \{1, 4, 5, 8\}$

is a loudspeaker offset, $n_v$ is the voice number, and $mod$ is the usual modulo integer division remainder operator. This 8 channel modality allows for a fully immersive experience of the swarms.

4 independent mono channels can instead be driven through the sound spatialization system (cf.Sec.3) in order to provide directed moving swarms across the concert hall.

Finally, a mono-sum of all channels is sent to an external reverboration system which can be manually adjusted according to the requirements of the hall. All in all, 13 separated and decorrelated output channels of audio are provided by the system.

## 2.2. Recording the base material

In order to prepare the voice fragment base materials, the composer has written a number of separate parts to be performed individually by several members of the choir. Supposedly, such pages are to be considered “working elements” of the score. Fig.5 provides an example of such parts (specifically, the swarm 7.2).

<figure><img src="image_60.png" /><figcaption>Figure 5: Example page of swarm fragment scoring</figcaption></figure>

All these fragments have been recorded by members of the choir of the Siena Cathedral “Guido Chigi Saracini” in the soundproof booth of the Center for Computational Sonology (CSC) of the University of Padua on June 14, 2023. Recordings were then followed by a post production phase in which all fragments were labeled and time-tagged to be read by the `sfreader` abstraction (cf.2.1). A truncated example of the result of the post production cue list files is shown below.

<table><thead><tr><th colspan="3">cue list example</th></tr></thead><tbody><tr><td>2.128784</td><td>2.394422</td><td>1_01</td></tr><tr><td>4.835803</td><td>5.981221</td><td>1_02</td></tr><tr><td>8.588798</td><td>8.728753</td><td>1_03</td></tr><tr><td>10.894367</td><td>12.227620</td><td>1_04</td></tr><tr><td>...</td><td></td><td></td></tr><tr><td>227.153815</td><td>229.267867</td><td>6_05</td></tr><tr><td>230.976787</td><td>237.215820</td><td>6_06</td></tr><tr><td>238.431216</td><td>253.354804</td><td>6_07</td></tr><tr><td>256.198094</td><td>262.842258</td><td>6_08</td></tr></tbody></table>

The cue list files are pure ascii text files which have three fields:
1. the start time of the fragment (in seconds)

2. its end time (in seconds)

3. the label of the fragment

Such files can be easily created by other audio Free Software (such as Audacity cf.[19]) and can be directly read by Pure Data.

## 2.3. Man-machine and machine-machine interfaces

Fig.6 shows a screenshot of the user interface of the swarming software presented herein.

<figure><img src="image_61.png" /><figcaption>Figure 6: Example page of swarm fragment scoring</figcaption></figure>

The five potentiometers (1 8-channel + 4 1-channel) and the swarm selector (the yellow vertical collection of buttons on the right) are relayed via OSC to a physical control surface which is operated in concert by the live electronics performers.

## 3. SWARM SPATIALIZATION

The spatialization of the swarms was designed to address multiple musical needs that could arise during both the composition phase and the rehearsals and performance of the piece.

<figure><img src="image_62.png" /><figcaption>Figure 7: Swarm spatialization routing</figcaption></figure>

The generation of sound fragments is structured to produce swarms with a broad sonic perspective – featuring foreground fragments alongside others that gradually appear more distant, expanding in all directions. The diffusion of these swarms occurs in eight-channel audio, controlled by a parameter that globally defines the swarm’s density and depth of field (the master_8ch potentiometer in Fig.6).

In addition to this immersive-type swarm, other variations can be introduced, where a single swarm has a defined origin and trajectory, with specified speeds and durations. Specifically, we prepared four simultaneous monophonic swarms (master_1-4) with independent paths, which can be activated separately from the immersive swarm. The latter could also function as a kind of “dark matter” into which the moving swarms dissolve once they complete their trajectory.

Finally, the sum of all swarms is sent to a reverberator, calibrated to simulate the maximum desired spatial dimension, emulating the far-distant zones of “dark matter”.

<figure><img src="image_63.png" /><figcaption>Figure 8: A visual simulation of swarm movement</figcaption></figure>

In the final outcome, only two to three swarms were used in *La Vérité, pas toute*, with spatialization movements favoring asynchronous, randomized motion. As an example, the visual simulation of Fig.8 shows the spatialization of *Swarm 1* split into two randomized movements with very smooth transitions – one with a step time of 1500 msec and the other with 1900 msec.

## 4. CONCLUSIONS

This paper outlines the inner technicalities of the *ad hoc* software created to generate the “swarms of sound” required by the work *La Vérité, pas toute* by Andrea Molino.

Having to be played in real-time in a live electronics ecosystem, this software took the shape of a *Pure Data* set of patches and abstractions. The choice of the host interpreter (*Pure Data*) was mainly dictated by the following requirements:

* real-time operation

* fast prototyping to allow last-minute changes during production

* use of Free Software to improve sustainability of the composition itself

All materials may be found, downloaded, cloned, forked, studied and modified at https://gitlab.com/smc-chigiana/sciami-molino.git.

## 5. CREDITS AND ACKNOWLEDGEMENTS

A large creative project such as *La Vérité, pas toute* needs the intense cooperation of a fairly large group of people; the technical aspects and challenges are only a small, if not minor, part of it. The musical creation is definitely the most difficult one; because of this, we owe our deepest gratitude to Andrea Molino, without whose huge work nothing would have existed.

The second most complex challenge is constituted by the actual logistics required to get such a project on stage; the Accademia Chigiana is one of the rare institutions which is willing to undertake this endeavor (and many others like this), fund it, get the necessary resources for it and solving big and small issues to reach the target. We want to warmly thank Nicola Sani who is at the head of Accademia Chigiana for having supported this project from the beginning, acknowledging also the great help from all the staff of the Accademia Chigiana and in particular from Barbara Valdambrini, Maria Rosaria Coppola and Anna Passarini.

Our warmest thanks go also to the performers, the *Coro Chigi Saracini* and its director and conductor Lorenzo Donati and the *Chigiana Percussion Ensemble* directed by Antonio Caggiano.

The intricate technical part of *La Vérité, pas toute* could not have been carried out without the fundamental participation of
SaMPL (Sound and Music Processing Lab) and its coordinator Julian Scordato, and that of the Center for Computational Sonology (CSC) of the University of Padua and its director Sergio Canazza.

Last but not least, we would like to thank all our students from the 2023 *Live Electronics – Sound and Music Computing* Summer Seminar at the Accademia Chigiana who took part to the production with many helping hands.

## 6. REFERENCES

[1] Miller Puckette, “The patcher,” in *Proceedings of the International Computer Music Conference (ICMC)*, 1988.

[2] Johannes Kreidler, *Programming Electronic Music in Pd*, Wolke Verlag, 2009.

[3] Miller Puckette, “Combining event and signal processing in the max graphical programming environment,” *Computer Music Journal*, vol. 15, no. 3, pp. 68–77, 1991.

[4] Miller Puckette, “Max at seventeen,” *Computer Music Journal*, vol. 26, no. 4, pp. 31–43, 2002.

[5] Miller Puckette, “Pure data: Another integrated computer music environment,” in *Proceedings of the Second International Computer Music Concerts*, Tachikawa, Japan, 1996.

[6] Philippe Codognet and Olivier Pasquet, “Swarm intelligence for generative music,” in *2009 11th IEEE International Symposium on Multimedia*. IEEE, 2009, pp. 1–8.

[7] Tim Blackwell and Michael Young, “Self-organised music,” *Organised sound*, vol. 9, no. 2, pp. 123–136, 2004.

[8] Philippe Codognet and Olivier Pasquet, “Sound agents,” in *European Conference on the Applications of Evolutionary Computation*. Springer, 2010, pp. 391–400.

[9] Maximos Kaliakatsos-Papakostas, Andreas Floros, Konstantinos Drossos, Konstantinos Koukoudis, Manolis Kyzalas, and Achilleas Kalantzis, “Swarm lake: A game of swarm intelligence, human interaction and collaborative music composition,” in *Joint Conference ICMC/SMC 2014. INTERNATIONAL COMPUTER MUSIC ASSOCIATION*, 2014.

[10] Christian Jacob and Gerald Hushlak, “Evolutionary and swarm design in science, art, and music,” in *The Art of Artificial Evolution: A Handbook on Evolutionary Art and Music*, pp. 145–166. Springer, 2008.

[11] Daniel Bisig, Martin Neukom, and John Flury, “Interactive swarm orchestra an artificial life approach to computer music,” in *ICMC*, 2008.

[12] Daniel Bisig and Martin Neukom, “Swarm based computer music-towards a repertory of strategies,” *computer music*, vol. 12, pp. 13, 2008.

[13] Tom Davis and Orestis Karamanlis, “Gestural control of sonic swarms: composing with grouped sound objects,” in *4th Sound and Music Computing Conference*, 2007.

[14] Arne Eigenfeldt and Philippe Pasquier, “A sonic ecosystem of self-organising musical agents,” in *Applications of Evolutionary Computation: EvoApplications 2011: EvoCOMNET, EvoFIN, EvoHOT, EvoMUSART, EvoSTIM, and EvoTRANSLOG, Torino, Italy, April 27-29, 2011, Proceedings, Part II*. Springer, 2011, pp. 283–292.

[15] Jonathan Impett, “Interaction, simulation and invention: a model for interactive music,” in *Proceedings of ALMMA 2001 Workshop on Artificial Models for Musical Applications*. Citeseer, 2001, pp. 108–119.

[16] Roger K Moore, “Vocal interactivity in crowds, flocks and swarms: implications for voice user interfaces,” *arXiv preprint arXiv:1907.11656*, 2019.

[17] Andrea Belladonna and Alvise Vidolin, “Applicazione MAX per la simulazione di sorgenti sonore in movimento con dispositivi musicali a basso costo,” in *Atti del X Colloquio di Informatica Musicale*, Milano, 1993, pp. 351–358.

[18] Andrea Belladonna and Alvise Vidolin, “spAAce: un programma di spazializzazione per il live electronics,” in *Proceedings of the Second International Conference On Acoustics and Musical Research*, Ferrara, 1995, pp. 113–118.

[19] Roger Dannenberg and Dominic Mazzoni, “A fast data structure for disk-based audio editing,” *Computer Music Journal*, pp. 62–76, 2002.
# LINUX AUDIO ON ANDROID

Victor Zappi

College of Arts, Media and Design

Northeastern University, US

v.zappi@northeastern.edu

## ABSTRACT

Although Android is based on the Linux kernel, conventional development practices force real-time audio code to pass through multiple layers of the Android framework, resulting in high complexity and limited performance. In this paper, I demonstrate a workflow that treats Android devices as if they were standard embedded Linux boards, thereby bypassing much of the Android audio stack and gaining near-direct access to ALSA drivers in the kernel. I explain the key steps, including the need for rooting the phone, identifying hardware routing paths and cross-compiling native C++ code, and show how all these elements are conveniently streamlined by the open-source LDSP environment. This allows developers to work with minimal overhead and fully exploit hardware capabilities, even on older devices. Through benchmarks on several phone models, ranging from legacy phones to recent flagships, I briefly show that this approach yields substantial gains in performance compared to the typical Android app model, focusing on the case of oscillator polyphony and real-time analog effect emulation. Finally, I situate these findings in broader discussions of hacking, appropriation and e-waste reduction, illustrating how reimagining Android phones as embedded Linux audio platforms can unlock new pathways for sustainable, high-performance and readily accessible music technology.

## 1. INTRODUCTION

Android is widely recognized as a mobile operating system, yet it fundamentally builds on a fork of the Linux kernel for each release. The practice began in 2008, when the first Android phone (the HTC Dream) shipped with a modified version of the Linux 2.6.25 kernel<fnref n="1" />. The modifications addressed the particular needs of mobile devices, most notably power management, limited memory, security and specialized inter-process communication. Over time, efforts such as the Android Generic Kernel Image project<fnref n="2" />, introduced in 2020, have sought to bring Android's kernels even closer to mainline Linux, although certain patches and development branches remain to accommodate device-specific requirements.

Despite these lingering differences, Android and Linux share many core components, one of which is the audio subsystem. Since Android 2.3, this subsystem has relied on ALSA drivers—just as numerous embedded Linux boards do—forming the technical foundation for high-performance audio processing [1, 2]. In addition, most Android devices employ ARM-based system-on-chips (SoCs), the same architecture frequently found in embedded Linux boards (e.g., Raspberry Pi, BeagleBone Black, Orange Pi). Consequently, it is reasonable to expect that native C++ audio applications can be developed on Android in a manner analogous to embedded Linux systems, namely by cross-compiling a single binary for direct command-line execution.

In practice, however, Android imposes a different paradigm for application development. The platform is designed around “apps” running under the Android runtime, complemented by libraries and Java/Kotlin application program interfaces (APIs). This workflow has proven successful in the domain of mobile computing<fnref n="3" /> [3], as it delivers a streamlined user experience with robust security, for example through sandboxing each app under a unique user identity. However, the additional layers of abstraction make development quite challenging [4, 3] and introduce a significant performance overhead, especially in areas such as audio processing and user interaction [5, 6]. This impact is further compounded by the complexity of the development process. Because Google designates Android Studio as the official environment for Android app development<fnref n="4" />, most production workflows gravitate toward it, which in practice limits developers’ flexibility to adopt other solutions. Given that modern smartphones are among the fastest-evolving technological platforms—each new model offering more powerful CPUs and specialized accelerators [7]—it becomes compelling to explore development approaches that harness this raw performance more directly, much like the efficient, low-level development practiced in embedded Linux systems.

To address these issues, I developed LDSP, an open-source environment that enables developers to build, deploy and test low-level audio applications directly on Android devices using native C++ code. LDSP allows a workflow resembling that of embedded Linux boards: cross-compile a single C++ binary, run it from the command line and interact directly with the audio drivers. This toolkit leverages TinyALSA<fnref n="5" />, the same audio library commonly used by original equipment manufacturers (OEMs), giving direct access to the underlying audio system and Android sensor data. It also provides support for communication protocols such as MIDI and OSC, alongside a host of relevant libraries and example projects. Since LDSP depends on device modifications (such as rooting and specific configurations) accompanying documentation and scripts are provided to streamline this “hacking” process.

In this paper, I describe the architecture of LDSP and how it interfaces with Android’s audio system, highlighting where its workflow diverges from traditional embedded Linux approaches. I emphasize the notion of “low-level C++” to underscore two key distinctions: (1) the generated native binary operates outside the typical Android app model (in which C++ code is wrapped in higher-level Android components), and (2) communication with the hardware drivers is as direct as possible, bypassing multiple layers of the Android framework. In addition to these technical contributions, I place LDSP in a broader context of design and philosophical inquiry. Specifically, I explore how hacking Android’s internals not only yields performance benefits but also engages with questions of appropriation, openness and critical design, suggesting that reclaiming control over closed, black-boxed systems can serve as a creative and sociocultural act.

<footnote n="1">¹https://android.googlesource.com/platform/vendor/htc/dream-open/%2B/la7ffe0f7552f4d3875685d2c15ccdc526134cca/kernel.version</footnote>

<footnote n="2">²https://source.android.com/docs/core/architecture/kernel/generic-kernel-image</footnote>

<footnote n="3">³At the time of writing, Android is the largest player worldwide in the mobile market (Statcounter Global Stats, Mobile Operating System Market Share Worldwide, April 2025).</footnote>

<footnote n="4">⁴https://developer.android.com/studio/intro</footnote>

<footnote n="5">⁵https://github.com/tinyalsa/tinyalsa</footnote>
## 2. RELATED WORK

Multiple projects have sought to repurpose Android phones as platforms for real-time audio processing and musical interaction. One early endeavor is *Nexus*, which exclusively relies on web technologies to deliver a browser-based framework for designing mobile graphical user interfaces and controlling audio processes [8]. Another notable project, *MoMuPlat*, builds on Pure Data (Pd) and *libpd* to facilitate audio synthesis and signal processing on Android devices, allowing users to run existing Pd patches through a customized mobile interface [9]. In both of these environments, graphical user interfaces (GUIs) are either manually scripted (via HTML or JSON) or generated procedurally, and ultimately interface with Android's built-in Java/Kotlin layer for rendering.

Of these mobile audio efforts, *faust2api* is perhaps the most comprehensive, featuring a toolchain to generate optimized Android apps that integrate both Faust's signal-processing code and device-specific capabilities such as sensor access [6]. Graphical interfaces for real-time user interaction, along with access to built-in sensors, transducers and mechanical resonances of handheld devices are supported by *faust2api*, which compiles Faust programs into a standard mobile application structure.

Despite these innovations, existing frameworks inherit substantial overhead and latency from Android's audio stack. Even "native" code in C++ or Faust must traverse multiple layers before reaching the driver: the application framework, Android's mixer and a HAL [3]. Each layer can introduce buffering, scheduling, or format conversion, elevating CPU load and compromising timing precision, factors that prove especially restrictive on older devices [6]. While Google's *Exclusive Mode* in the AAudio API offers a partial bypass of these layers, it remains limited to newer phones (Android 8.0 or above) and may not address the needs of interactive audio applications that demand minimal latency and maximum flexibility. Hence, although these environments have broadened the scope of mobile music making, the inherent constraints of the Android ecosystem continue to challenge developers seeking high performance on a wide range of devices.

## 3. ANDROID LOW-LEVEL C++ DEVELOPMENT

Android's deep ties to the Linux kernel enable a development workflow reminiscent of embedded Linux systems. In theory, one can cross-compile a standard ELF (Executable and Linkable Format) binary on a host machine, transfer it to an Android device and run it for testing or debugging. The file system structure on Android accommodates shell scripts (though not necessarily with Bash) and it includes many familiar shell commands, even if certain utilities are missing or differ slightly from common desktop distributions.

A key requirement for low-level development is a proper toolchain consisting of a cross-compiler, linker, system libraries (including the C standard library) and, optionally, debugging tools. While one might assume that a generic ARM Linux cross-compiler would suffice, the Android runtime and its sysroot differ from those of typical embedded Linux boards. Thus, the *Android NDK* (Native Development Kit) is commonly used. It bundles toolchains for all Android versions still supported by Google at the time of each NDK release (which means older devices are compatible with older NDKs). It also provides CMake "toolchain files," which seamlessly configure builds by generating build scripts for *Ninja*<fnref n="6" />—requiring that both CMake and *Ninja* be installed on the host system.

Moving binaries to an Android device and running them typically does not involve the usual workflow based on the secure shell protocol found on most embedded boards. While so-

lutions such as Termux<fnref n="7" /> provide partial secure shell functionality, the Android Debug Bridge (ADB) is the official, universally supported mechanism. ADB has been part of the Android Open Source Project (AOSP) since 2007 and is available as a command-line tool on all major desktop operating systems<fnref n="8" />. It communicates with the ADB daemon pre-installed on every Android device, allowing users to push and pull files, open a shell and perform various debugging tasks over USB or Wi-Fi. By default, the shell user on Android is heavily restricted, lacking a conventional home directory. However, the path `/data/local/tmp` is fully writable and binaries stored there can be executed without root privileges. Consequently, combining the Android NDK (for cross-compilation) with ADB (for deployment and shell access) creates a robust pipeline for low-level C++ development on virtually any Android phone, all without rooting or third-party package managers.

## 4. LOW-LEVEL AUDIO AND THE NEED FOR HACKING

While it is somewhat straightforward to run a generic native binary on Android, obtaining truly low-level access to the audio subsystem poses additional challenges. By default, the Android shell user lacks the necessary permissions to open the hardware device nodes that handle audio I/O. In practical terms, any attempt to use TinyALSA—e.g., by calling `pcm_open()`—will fail, since the underlying sound devices are off-limits for security and privacy reasons.

To circumvent this restriction, Android apps rely on the privileged *audioserver* process, a system service that centrally manages all audio streams and has direct access to device drivers via TinyALSA. However, developers aiming to communicate with audio hardware at a lower level, outside the conventional app paradigm, need to run their binaries with elevated privileges. The simplest approach is to *root* the device and run the executable as the superuser. Once root access is obtained, the audio device nodes become directly accessible and all driver-level calls succeed.

Superuser privileges offer additional benefits as well. They enable direct access to input events (e.g., touchscreen and buttons) and to various kernel subsystems such as notification LEDs or vibration motors, which are typically out of reach for unprivileged users. Furthermore, root access permits fine-grained control over performance-related parameters, including real-time priorities and CPU frequency scaling, ensuring that audio applications can run optimally with minimal latency. In essence, this "hacking" step is essential for achieving the same level of low-level control commonly available in embedded Linux workflows, where unrestricted root privileges are routinely employed to ensure high-performance audio (notable examples are Bela [10] and systems leveraging *Jack*<fnref n="9" />).

### 4.1. Hardware Mixer: Linux vs. Android

While the steps described above are sufficient to run low-level audio applications on embedded Linux boards, Android introduces an additional challenge related to hardware routing. Modern SoC designs for smartphones typically include an audio codec equipped with a hardware mixer. This mixer routes various inputs and outputs (e.g., headphones versus speakers, long- versus short-range microphones) and allows the adjustment of corresponding gain levels from user space. However, the specifics of this hardware routing are closely tied to each codec's unique feature set and operational logic<fnref n="10" />.

<footnote n="7">⁷https://termux.dev/en/</footnote>

<footnote n="8">⁸https://developer.android.com/tools/adb</footnote>

<footnote n="9">⁹https://jackaudio.org/faq/linux rt config.</footnote>

<footnote>html</footnote>

<footnote n="10">¹⁰https://www.kernel.org/doc/html/v4.19/sound/soc/overview.html</footnote>

<footnote n="6">⁶https://ninja-build.org/</footnote>
On most embedded Linux boards, including the Raspberry Pi and the BeagleBone Black, the kernel driver abstracts these hardware details<fnref n="11" />. When user code requests operations (like switching from a built-in microphone to a line input or adjusting a channel's gain), the driver consults a *device tree* file that specifies how to translate the request into hardware-dependent controls. In contrast, Android omits such a device-tree-based approach for audio, placing the onus on user-space code to handle the hardware mixer directly. More precisely, the mixer is configured via two key components:

* `mixer_paths.xml`, a loosely standardized XML file that enumerates all controls for the SoC's codec and defines valid routing paths, input/output options and level settings.

* An *audio HAL*, an OEM-compiled static library that links against TinyALSA and handles all operations on the audio codec<fnref n="12" />, including configuring the mixer. In particular, it parses the controls defined in `mixer_paths.xml` and translates user requests into commands tailored for the underlying hardware.

This approach makes low-level audio handling on Android more fragmented than on standard embedded Linux systems. Whereas the latter rely on a kernel-based mixer abstraction and a coherent device tree, Android devices use a user-space model that can vary significantly between manufacturers and hardware revisions.

## 4.2. Mixer Configuration

The first requirement for routing audio in a low-level native binary is access to the `mixer_paths.xml` file, whose location can vary by device. Once located, its contents must be parsed at runtime so that user-level routing requests can be mapped to the corresponding low-level controls, akin to what the audio HAL does. In particular, `mixer_paths.xml` organizes controls into container elements identified by a `name` attribute. Each user request typically corresponds to one such element, which can itself include one or more controls.

<figure><img src="image_64.png" /><figcaption>Figure 1: Generic playback routing configuration. Highlighted in red is the sound device opened in user code via the ALSA driver, whose output is routed to the phone's built-in speaker via mixer controls.</figcaption></figure>

On most Android phones, the full playback route can be established via two requests only, referencing two distinct container elements in the XML:

* One XML element that routes the output stream of the chosen sound device into the mixer; it typically contains a single control, referencing the ALSA string identifier for an internal sound device (e.g., "MultiMedia1"). Modern

smartphones often feature multiple such internal devices for different scenarios, like voice, media or VoIP. We refer to this particular control as the *device activation*.

* Another XML element configures the routing of the mixer to a physical destination (like built-in speaker or headphones), along with any necessary enable switches and gain levels. We refer to this element as the *path*.

Figure 1 illustrates a playback routing configuration, while Figure 2 displays the two XML elements—extracted from an Android phone's `mixer_paths.xml`—that are essential for this configuration. Equivalent container elements exist for capture routing, while the overall number of elements (and their names) depends on the SoC's available sound devices, sources and destinations. In the Android framework, the audio HAL is preconfigured by the OEM with the names of the XML elements that contain the mixer controls. When a user routing request is received, the HAL matches the request with the appropriate XML element name, uses the name to retrieve the corresponding controls from `mixer_paths.xml` and then executes these controls on the hardware. Unfortunately, there is no universal naming standard and, while the `mixer_paths.xml` format loosely follows AOSP guidelines, each vendor's implementation can differ.

<figure><img src="image_65.png" /><figcaption>Figure 2: XML elements used in a Motorola phone to set up the route shown in Figure 1. The single control contained in the top element performs the “device activation;” in blue is highlighted the ALSA string identifier of the sound device activated by this control and opened in user code.</figcaption></figure>

Hence, to configure a full-duplex audio route, a native application must know at least:

* The general format of the device activation controls needed for playback and capture, together with the ALSA string identifiers for the chosen sound devices.

* The names of at least one playback path and one capture path, used to finalize the routing on each side.

Both pieces of information can be obtained by examining `mixer_paths.xml` on a target device. These final steps complete the minimal requirements for building, deploying and running a C++ low-level audio application on Android, as implemented in LDSP.

# 5. LDSP ENVIRONMENT

The LDSP environment comprises two main components: a C++ framework built around a TinyALSA-based audio engine and a supporting infrastructure of tools for building, deploying and debugging, as well as for assisting with the hacking process. The C++ framework is designed to host user code and facilitate the rapid prototyping of interactive audio applications, while the supporting infrastructure streamlines the technical steps needed to get these applications running on target devices.

## 5.1. C++ Framework

At the heart of the C++ framework is a full-duplex audio engine that communicates directly with the ALSA subsystem in

<footnote n="11">¹¹https://www.kernel.org/doc/Documentation/devicetree/bindings/sound/simple-card.txt</footnote>

<footnote n="12">¹²Google provides a reference design for the audio HAL in the AOSP, but each OEM often modifies or extends it to accommodate specific hardware features and optimizations.</footnote>
the kernel. This engine is organized as a CMake project and adopts a code structure similar to that of Bela. User code is integrated by supplying at least one "render" file that implements the three main functions of the LDSP API. These functions are: `setup()`, which is invoked at the beginning of execution; `render()`, which serves as the real-time audio callback function; and `cleanup()`, which is called just before the application terminates. In addition to the engine, the framework incorporates a broad set of libraries for both audio and control tasks, including: sample loading, filtering, FFT, MIDI, OSC and serial/Arduino integration. It also grants access to all sensors exposed by the Android NDK via `libandroid`, the same ones accessible to ordinary Android apps. Moreover, the framework provides functionality to read touchscreen events, monitor volume and power buttons and control hardware features such as the flashlight, vibration motor and screen state. Several example projects accompany the framework to illustrate common usage patterns and demonstrate how its programming interface can be adapted to diverse use cases.

Beyond these higher-level components, the LDSP framework offers full run-time control of audio parameters for both capture and playback, including the selection of the sound device, period size, sample format, sample rate and number of channels. Furthermore, it supports routing each application to any of the physical sources and destinations available in the codec, while also accommodating external audio interfaces connected over USB. Low-level system performance settings are similarly exposed: the audio thread can be elevated to the highest real-time priority, with secondary tasks (e.g., sensor readings) assigned lower, though still real-time, priorities. Additionally, the CPU governor can be switched to *performance mode* (forcing the maximum clock rate) when the application starts and restored to defaults upon exit, and it is possible to adjust the CPU affinity of the audio thread. In other words, the engine encapsulates most of the capabilities of a generic audio HAL (including mixer routing—more on this in the next subsection), along with additional performance-related features that are not available on Android due to its restricted, sandboxed environment.

To enable such deep system-level adjustments, applications built with LDSP must be run with superuser privileges. As noted in Section 4, rooting the device is specifically required in order to gain direct access to kernel submodules such as audio and touchscreen events, configure scheduling priorities and modify CPU performance settings. Moreover, from Android 8 onward, root access is also necessary to retrieve sensor data via `libandroid`.

Finally, to support more advanced audio workflows and interactive control scenarios, LDSP integrates real-time inference frameworks—namely *RTNeural* [11] and *ONNX*<fnref n="13" />—thereby enabling advanced machine-learning pipelines to run directly on the device. Alongside these inference engines, LDSP provides a lightweight C++ webserver library that serves as the backend for a web-based GUI (see next subsection) and can also be repurposed in user code for other network-related features.

## 5.2. Supporting Infrastructure

LDSP provides a main script (available as both a Bash and a Batch file) that manages the full development workflow on the host machine. By specifying the paths to the Android NDK and to the user's LDSP project to build, this script generates build instructions via CMake and Ninja, transfers the resulting binary (along with any required resources) to the target device, and can run or stop it on demand. The latter is done as root, to grant the needed superuser privileges. It also installs on the phone the LLDB server included in the Android NDK and configures it for remote debugging through ADB, allowing developers to attach a local debugger on the host side. Crucially, neither the script nor

the LDSP framework itself is tied to any specific development environment or operating system, freeing developers from the typical reliance on Android Studio. Instead, one can use any preferred editor or integrated development environment.

Beyond user-managed resources (like audio files, pretrained neural models or other files placed in the user project folder on the host machine), the script deploys a JSON hardware configuration file containing essential information for correct run-time behavior. In particular, this file specifies the location of `mixer_paths.xml`, identifies the device activation controls for both playback and capture, and names the paths for playback (speaker, headphones and handset) and capture (built-in microphone, line-input, handset microphone) scenarios. It may also include optional entries for more complex routing scenarios or to define default audio parameters like period size, channel count and sound device identifier. Because these details vary across different phone and tablet models, LDSP ships with a collection of pre-configured hardware files for tested devices and a template for defining new ones. To aid in creating or extending these configurations, additional shell scripts are provided; they help developers with tasks like the discovery of the exact path to `mixer_paths.xml`, the enumeration of ALSA device identifiers and the determination of valid parameter settings for both playback and capture. Collectively, these tools guide the hacking process, allowing LDSP to be adapted to a wide range of previously untested Android devices.

LDSP also provides a minimal set of web assets (HTML, JavaScript and CSS) that are deployed onto the target device. These assets are served by the lightweight C++ webserver included in the framework, which launches automatically when the audio application starts. By default, they form a basic example of a front-end interface that can be easily replaced or extended with a custom GUI built using standard web technologies or libraries such as `p5.js`.<fnref n="14" /> Communication between the client-side GUI and the audio engine occurs through simple HTTP or WebSocket requests, allowing real-time control over parameters, visualization of sensor data, and interactive musical experimentation. This design closely follows the paradigm established by Bela, but is entirely self-contained on the Android device. Users connect to the device's IP address and designated port, gaining immediate access to the hosted interface. As a result, LDSP offers a cohesive, out-of-the-box solution for both low-level audio processing and intuitive front-end interactivity.

## 6. DISCUSSION

LDSP was created with the goal of hacking Android's audio infrastructure to achieve better performance for real-time, interactive applications, such as musical instruments and sound installations. By exposing low-level system capabilities and bypassing standard app-model constraints, LDSP harnesses the raw processing power of mobile devices in creative ways that are not normally envisioned by platform designers. In design and human-computer interaction (HCI), this style of hacking has often been described as a form of appropriation, particularly in artistic contexts [12, 13]. Here, appropriation refers to a deliberate reworking of a technology to deviate from its "proper" or expected uses, thereby democratizing design and injecting personal creativity into otherwise closed systems. This perspective resonates with established HCI theories of designing for appropriation, which encourage systems to be flexible enough for users to adapt them in unanticipated ways [14]. It also connects to broader ideas around intentional versus hidden affordances [15], illustrating how technologies might be co-opted for novel ends that lie well beyond the initial design brief.

By focusing on hacking Android's underlying mechanisms and use cases, LDSP operates at the intersection of design, engineering, and sociology—disciplines that increasingly converge

<footnote n="13">¹³https://github.com/microsoft/onnxruntime.gi</footnote>

<footnote n="14">¹⁴https://p5js.org/</footnote>
in the analysis of how devices and infrastructures shape daily experiences. Through its low-level approach, LDSP challenges conventional frameworks and paves the way for broader discussions on appropriation, sustainability and access. In the remainder of this section, these themes will be explored, situating technical performance within both HCI and cultural contexts.

## 6.1. Hacking a Tool vs. Hacking a System

In much of the literature on hacking and appropriation, “hacking a tool” often entails modifying or repurposing a specific object or technology [16, 17, 12]. Circuit-bending a toy keyboard for creative sound output, for instance, is emblematic of hacking as a hands-on manipulation of a discrete artifact. In contrast, “hacking a system” intervenes in a more expansive socio-technical infrastructure by bending or circumventing the rules governing how that infrastructure is meant to operate [18, 19]. This includes everything from subverting artificial intelligence models to exploiting software platforms in ways never originally intended, sometimes simply for playful or artistic goals [20].

In the case of LDSP, hacking a system is precisely what is happening. Rather than altering a single phone model or a particular piece of hardware, we can work within Android's intricate layers of security and abstraction to reshape the platform's behavior for more direct, performance-critical audio tasks. Notably, Android's heavy sandboxing and standardized frameworks were not designed with hacking or appropriation in mind; rather, they aim to enforce consistent user experiences and strong protections for security and—allegedly—privacy [21]. By circumventing these restrictions to unlock advanced audio capabilities, LDSP does more than provide a technical workaround. It uncovers and recontextualizes inner mechanisms of a globally pervasive operating system, illustrating how design constraints and engineering decisions can be challenged and reimagined.

Such an approach aligns with the spirit of the *Critical Engineering Manifesto* [22], which posits that any sufficiently complex technological system—from consumer software to communication infrastructure—“should be considered a potential target for dissection and modification,” both to question the power structures embedded in these systems and to regain agency over them. This stance resonates with Latour’s notion of the “black box,” wherein technologies become opaque over time, obscuring the political and social forces that shaped them [23]. By systematically “breaking open” Android’s layers and rewriting its internal workflows, LDSP exemplifies hacking as a method of inquiry and reflection. In doing so, it reframes hacking from a mere technical exploit to a critical design practice that surfaces how everyday tools function and empowers users to reshape them for novel creative ends.

## 6.2. Is the Effort Worth It? Performance Gains and Broader Implications

Design theorist Scott Burnham describes “design hacking” as a resourceful, user-driven practice that steps in when commercial products and systems fail to meet specific needs [24]. Rather than dismissing older or underutilized devices as obsolete, hacking refashions them into new tools that fill gaps in the creative landscape. The LDSP environment squarely aligns with this ethos, as its very premise is to repurpose hardware in unconventional ways. Yet one might ask whether this effort is truly worth it. Two recent studies suggest it is.

In [25], we tested LDSP on phones released between 2014 and 2018 by running large banks of oscillators—an archetypal benchmark for audio synthesis. We compared LDSP’s native approach to an equivalent “modern” Android audio app built with Java/Kotlin and C++. Across six phone models, LDSP consistently achieved higher oscillator counts and lower latencies, sometimes doubling or even tripling the maximum achievable polyphony. This substantial performance boost can transform

both recent and outdated phones into potent audio processors for musical instruments, sound installations, or other interactive sonic projects.

A second study focused on *neural* audio processing, an area in which even embedded devices can struggle to meet real-time deadlines [26]. Using LDSP to bypass standard Android buffering and security layers, we tested a set of neural models trained to emulate analog effects (two guitar distortions and a compressor) on phones dating from 2014 to 2022 [27]. The results demonstrate that LDSP can run these models in real time on surprisingly old phones, rivaling or surpassing performance on newer devices when confined to the conventional Android stack. Crucially, the direct, low-level access also grants developers far more control over buffer sizing, scheduling and system priorities, further optimizing the neural inference pipelines.

These findings confirm the profound payoff for those willing to navigate Android's deeper layers. By investing time in rooting devices and configuring the custom LDSP environment, users unlock performance levels otherwise inaccessible through typical Android app workflows. In doing so, they also challenge the notion of “proper” use encouraged by commercial platforms and mass-production cycles, reinforcing hacking as a potent form of creative, resourceful design.

## 6.3. From Rapid Prototyping to Sustainable Futures

A key appeal of hacking Android at the system level is the ability to rapidly prototype complex audio applications without being bound by the standard framework's restrictions. In the context of low-level development, this translates into a more direct pipeline from concept to implementation, allowing designers to iterate faster and fine-tune performance-critical components. Much like the tradition of hands-on, do-it-yourself instrument building in circuit bending and open-hardware communities [28], LDSP applies an exploratory and self-sufficient approach to smartphones, providing developers with an instantly available hardware testbed for sonic experimentation. By removing the overhead of multiple buffering layers, audio-thread scheduling and curated app distribution, it fosters an environment in which iterative design can happen both autonomously and with fine-grained control over performance.

Yet the significance of LDSP and similar approaches extends well beyond convenience, into the domain of technological democratization. Many creatives around the globe lack the resources or stable supply chains to obtain specialized audio boards and embedded platforms. In Latin America, for instance, import tariffs and high shipping costs place widely-touted digital musical instrument hardware out of financial reach [29]. Repurposing the smartphones already on hand is a step towards leveling the playing field, enabling artists, students and researchers in underserved regions to harness modern digital signal processing and sensor capabilities.

Equally pressing is the prospect of preservation and sustainability. As Dorigatti and Masu note [28], circuit bending's very roots trace back to a response against rapid technological obsolescence, transforming “discarded” devices into new creative canvases. In the same spirit, LDSP fosters an *upcycling* paradigm. By enabling native C++ audio applications on older or unwanted Android phones, it diverts them from the e-waste stream and grants them a second life as fully programmable audio engines. This approach not only rescues valuable hardware from landfills but also reveals new paths for environmental stewardship—particularly crucial in regions disproportionately affected by the negative externalities of technological consumption. Seen in this light, hacking Android becomes an active resistance against planned obsolescence; it reframes the phone as an enduring, flexible musical instrument rather than a disposable commodity, thereby challenging the prevailing assumption that cutting-edge musical systems always require newly manufactured hardware.
## 7. CONCLUSIONS

In this work, I have demonstrated that it is indeed possible to develop low-level C++ audio applications on Android devices as if they were standard embedded Linux boards. I analyzed in detail the key similarities and differences between the two platforms, and showed how the LDSP environment conveniently streamlines all the components necessary for a complete workflow. LDSP offers a powerful alternative to the conventional Android app model by transforming Android phones—ranging from legacy devices running Android 2.3 to the latest releases<fnref n="15" />— into flexible, low-level Linux audio platforms. Through its straightforward native C++ workflow, LDSP provides both an efficient audio engine (built atop TinyALSA) and seamless integration with hardware features like sensors, touchscreen and notification LEDs. Beyond C++, it also supports Pd patches via libpd, enabling for the harnessing of a wide variety of established creative coding paradigms. In doing so, it effectively narrows the gap between consumer mobile hardware and specialized Linux audio boards, significantly expanding the scope of what can be accomplished on everyday smartphones.

Naturally, limitations persist. Older phones that deviate from AOSP standards may not fully support LDSP's reverse-engineered audio stack and performance characteristics vary drastically according to device-specific hardware (e.g., CPU clock speed, memory, touchscreen refresh rate). Additionally, camera-based features are not yet available, as incorporating further HALs requires extensive reverse-engineering work. Nonetheless, benchmarks showcased in this paper and in earlier studies demonstrate LDSP's ability to deliver substantial gains in real-time performance, unlocking applications such as polyphonic synth engines, neural audio effects and sensor-rich interactive systems on hardware that would otherwise be constrained by Android's app framework.

Looking ahead, LDSP's feature set continues to grow through community-driven development. Ongoing efforts include the integration of performance-optimized libraries (such as parallelized FFTs), the addition of real-time visualization tools like a software oscilloscope and improvements to documentation for both developers and end users. Planned future work also includes an open-source Android Studio project that embeds the LDSP audio engine as a native .so library, allowing it to serve as a drop-in replacement for AAudio inside ordinary Java/Ktlin apps and thereby realizing a true "Linux audio backend" within conventional Android workflows.

By facilitating immediate, low-level access to ALSA drivers, the framework reimagines mobile hardware as a full-fledged audio workstation, with compelling implications for resource-limited environments and upcycling initiatives. Moreover, it underscores how hacking and appropriation can push beyond the boundaries of typical Android development, transforming mainstream devices into sustainable, high-performance solutions for music technologists worldwide. The environment is freely available for download at https://github.com/victorzappi/LDSP.

## 8. REFERENCES

[1] James William Topliss, Victor Zappi, Andrew McPherson, et al., "Latency performance for real-time audio on beatable black," in Proceedings of the Linux Audio Conference, LAC, 2014.

[2] Luca Vignati, Stefano Zambon, and Luca Turchet, "A comparison of real-time linux-based architectures for embedded musical applications," Journal of the Audio Engineering Society, vol. 70, no. 1/2, pp. 83-93, 2022.

[3] Alessio Balsini, Tommaso Cucinotta, Luca Abeni, Joel Fernandes, Phil Burk, Patrick Bellasi, and Morten Rasmussen, "Energy-efficient low-latency audio on android," Journal of Systems and Software, vol. 152, pp. 182-195, 2019.

[4] Rudi Villing, Victor Lazzarini, Dawid Czesak, Sean O'Leary, and Joseph Timoney, "Approaches for constant audio latency on android," in Proceedings of the International Conference on Digital Audio Effects, DAFx, 2015.

[5] Topi Kaaresoja and Stephen Brewster, "Feedback is... late: measuring multimodal delays in mobile device touchscreen interaction," in International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, 2010, pp. 1-8.

[6] Romain Michon, Yann Orlarey, Stéphane Letz, Dominique Fober, and Catinca Dumitrascu, "Mobile music with the faust programming language," in Perception, Representations, Image, Sound, Music: 14th International Symposium, CMMR 2019, Marseille, France, October 14-18, 2019, Revised Selected Papers, 2021, pp. 307-318.

[7] Andrey Ignatov, Radu Timofte, William Chou, Ke Wang, Max Wu, Tim Hartley, and Luc Van Gool, "Ai benchmark: Running deep neural networks on android smartphones," in Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018, pp. 0-0.

[8] Benjamin Taylor, Jesse T Allison, William Conlin, Yemin Oh, and Daniel Holmes, "Simplified expressive mobile development with nexusui, nexusup, and nexusdrop.," in NIME 2014, 2014, pp. 257-262.

[9] Daniel Iglesia and Iglesia Intermedia, "The mobility is the message: The development and uses of mobmuplat," in Pure Data Conference (PdCon16). New York, 2016.

[10] Andrew McPherson, "Bela: An embedded platform for low-latency feedback control of sound," The Journal of the Acoustical Society of America, vol. 141, no. 5, pp. 3618-3618, 2017.

[11] Jatin Chowdhury, "Rtneural: Fast neural inferencing for real-time systems," arXiv preprint arXiv:2106.03037, 2021.

[12] Victor Zappi and Andrew McPherson, "Hackable instruments: supporting appropriation and modification in digital musical interaction," Frontiers in ICT, vol. 5, pp. 26, 2018.

[13] Isabel Li, Ace S Chen, Eric Rawn, Shm Garanganao Almeda, Bjoern Hartmann, and Jingyi Li, "Reimagining misuse as creative practice: Impressions and implications of usage norms on digital artists," in Proceedings of the ACM CHI conference on Human Factors in Computing System, 2025.

[14] Alan Dix, "Designing for appropriation," in Proceedings of HCI 2007 The 21st British HCI Group Annual Conference University of Lancaster, UK. BCS Learning & Development, 2007.

[15] Victor Zappi and Andrew McPherson, "Dimensionality and appropriation in digital musical instrument design," in Proceedings of the International Conference on New Interfaces for Musical Expression. Citeseer, 2014, pp. 455-460.

[16] Daniela Rosner and Jonathan Bean, "Learning from Limited Hacking: I'm not one to decoupage a tabletop and call it a day.," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2009, pp. 419-422.

[17] Nicolas Collins, *Handmade electronic music: the art of hardware hacking*, Routledge, 2009.

<footnote n="15">¹⁵At the time of writing, the newest Android version is 15</footnote>
[18] Linda Maria Jessica Kronman, “Hacking surveillance cameras, tricking ai and disputing biases: Artistic critiques of machine vision,” *Open Library of Humanities*, vol. 1, 10 2023.

[19] Jennifer Gradecki, “The art index as art,” *Visual Resources*, vol. 34, no. 1-2, pp. 65–92, 2018.

[20] Golan Levin, Scott Gibbons, and Gregory Shakar, “Dial-tones: A telesymphony,” Live concert performance using audience mobile phones as instruments, 2001, Premiered at Ars Electronica Festival, Linz, Austria.

[21] Chuan Yan, Mark Huasong Meng, Fuman Xie, and Guang-dong Bai, “Investigating documented privacy changes in android os,” *Proceedings of the ACM on Software Engineering*, vol. 1, no. FSE, pp. 2701–2724, 2024.

[22] Julian Oliver, Gordan Savičić, and Danja Vasiliev, “The critical engineering manifesto,” https://criticalengineering.org, 2011, Accessed March 28, 2025.

[23] Bruno Latour, “Science in action: How to follow scientists and engineers through society,” *Harvard UP*, 1987.

[24] Scott Burnham, *Design Hacking: Resourceful Innovation and Sustainable Self-Reliance*, VRMNTR, 2019.

[25] Victor Zappi and Carla Sophie Tapparo, “Upcycling android phones into embedded audio platforms,” in *Proceed-

ings of the International Conference on Digital Audio Effects, 2023.

[26] Teresa Pelinski, Rodrigo Diaz, Adan L. Benito Temprano, and Andrew McPherson, “Pipeline for recording datasets and running neural networks on the bela embedded hardware platform,” in *Proceedings of the International Conference on New Interfaces for Musical Expression*, Miguel Ortiz and Adnan Marquez-Borbon, Eds., Mexico City, Mexico, May 2023, pp. 160–166.

[27] Jason Hoopes, Brooke Chalmers, and Victor Zappi, “Neural audio processing on android phones,” in *Proceedings of the International Conference on Digital Audio Effects*, 2024.

[28] Enrico Dorigatti and Raul Masu, “Circuit bending and environmental sustainability: Current situation and steps forward,” in *NIME 2022*, 2022.

[29] Juan Pablo Martinez Avila, João Tragtenberg, Filipe Calegario, Ximena Alarcon, Laddy Patricia Cadavid Hinojosa, Isabela Corintha, Teodoro Dannemann, Javier Jaimovich, Adnan Marquez-Borbon, Martin Matus Lerner, et al., “Being (a) part of nime: Embracing latin american perspectives,” in *NIME 2022. PubPub*, 2022.
# MFP: A GRAPHICAL PATCHER EXTENSIBLE IN PYTHON AND FAUST

Bill Gribble

grib@billgribble.com

## ABSTRACT

MFP is a graphical patching system distinguished by the ready availability of Python data types and operations in patches. Recent work has added support for live coding of message-processing patch elements in Python, and of signal-processing patch elements in Faust. Its extensibility and rich set of message data types give MFP some nice capabilities for interactive patching.

## 1. INTRODUCTION

My first exposure to live coding for audio was Pure Data (Pd) [1]. As a programmer, I have always loved the read-eval-print loop (REPL) as a way of exploring solutions and incrementally building and running code. Pd's interface immediately connected to the part of my brain that likes REPL development. It's such a great way of experimenting, testing, and, once you get something going, improvising and performing on a patch that just keeps running even when you make mistakes.

The original development of MFP was inspired by the realization that a dataflow patcher's "REPL" really is just code, expressed in an unconventional form. I know this isn't exactly a stroke of genius – one of the mottos of Pd is "the diagram is the program", so it's basically printed on the tin. What was interesting to me was looking at what it means for a graphical patcher to be a programming language. How do the concepts of a conventional programming language map onto the expressive vocabulary of a patch? Are there gaps in the correspondence that can lead to insights towards better patching systems? Can the expressive power of the implementation language give a boost to the patcher? Can the need for real-time performance be reconciled with a rich and dynamic data model?

When regarded as programming languages, dataflow patchers have many aspects that connect them to the programming language "family tree". They are somewhat like functional languages, in that values are returned from functions and marshaled into parameter lists of other functions without resorting to variable names or side effects. Message passing, the main mechanism patchers use for data interchange and code invocation, goes back to Smalltalk and beyond. Patchers support modular programming via the ability to start with simple units and compose them to make more complex behavior. They have a variety of data types, input/output mechanisms, and modes of interaction with the wider system. So it's reasonable to expect patching languages to support a pretty wide variety of usual programming paradigms and patterns.

However, when I started trying to make more complex patches in Pd, especially ones that leaned into the manipulation of non-audio data, I quickly ran into frustrations with the data model that made it quite difficult to do things that should (in my opinion) have been simple. I wondered if breaking out of the space -delimited textual syntax and limited type-vocabulary of Pd objects, and using the data types from a dynamic language such as Python instead, could give a graphical patcher more of the power of a full programming language.

The result of my explorations is MFP. The patch editor and message processing engine are implemented in Python and directly expose the Python data model, library, and evaluator to the user. The real-time audio engine is implemented in C and is loosely coupled to the UI and message-processing system via an RPC mechanism. The overall experience is similar to Pd or Max/MSP, but there has been no attempt at compatibility.

When I presented it at the 2013 edition of LAC [2], MFP was functional enough and interesting enough that I thought it was worth continuing to develop. Now, some 12 years later, it has evolved in various foreseen and unforeseen directions and is, I think, worthy of a re-introduction to the Linux audio community.

## 2. WHAT IS IT?

My vision for MFP is as a multitool for REPL-style incremental development that can fit in to music composition, performance, and recording workflows wherever some bespoke functionality is needed.

In operation, MFP will look pretty familiar to users of Pd or Max/MSP. See Figure 1 for a sample session. Its interface allows the user to interactively build *patches*, which are sets of directed graphs of *processors*. A processor has a set of inputs, which handle either *messages* (discrete pieces of data of any type) or *signals* (streams of floating-point audio data). It produces outputs which are likewise either messages or signals. The behavior of a processor is determined from a type-name and arguments provided at creation. The type-name can resolve to either a builtin MFP processor type, of which there are over 150, a user-created patch, or an arbitrary Python callable.

In the patch editing window, processors can appear as plain boxes, or can have a variety of visual representations that allow for patch control or feedback: message boxes, number boxes, buttons, sliders, dials, X/Y plots, text with Markdown formatting. Patches used as building blocks by other patches can export a UI of their choice to appear on their placeholder in the containing patch. Direct connections between processor nodes are represented by lines.

Named message and signal buses, comparable to Pd's `[s]` and `[r]`, appear as dots terminating connection lines, called *vias*. Names and scoping are an important part of MFP, and *vias* represent one of the main user-visible aspects of naming: each *via* is paired with its companion(s) by a name, visible in the UI, that must be resolved to find the intended connection-partner.

The patch as a whole is modeled as a stack of *layers*, which are something like Pd *subpatches*: they are primarily a visual organizational tool to reduce clutter in the patching interface. One layer of a patch is visible at a time. Multiple patches can be displayed simultaneously, using a multi-page tiled workspace manager.

The nomenclature of *layers* and *vias* comes from electronics design. A printed circuit board is made up of a multi-layer sandwich of conductive and nonconductive material. Most of these layers are hidden, with only the components in the outermost layer visible to the naked eye. To me, a dataflow patch looks a bit like a circuit board populated with chips and other components, with most of the layers or "subpatches" being hidden until needed. A *via* is a plated conductive hole, effectively a wire, that connects from one layer through to another. When looking at a layer of a circuit board, a *via* looks like a dot at the end of a conductive trace. The visual representation of a *via* as a dot/circle is intended to evoke that.

## 3. CAPABILITIES

An exhaustive listing of MFP's features would be quite long and probably boring. Instead, I would like to highlight a curated set
<figure><img src="image_66.png" /><figcaption>Figure 1: View of MFP app window, featuring tiled workspace, object info panel on the right, and log window with status/ tooltip line below.</figcaption></figure>

of features that might be of interest when trying to decide if you want to download MFP and give it a try.

## 3.1. Data Model

Message data is plain Python objects of any type. Numbers, strings, arrays, dictionaries, sets, custom classes, anything. Literal data typed into the UI is evaluated as it would be in a Python REPL. This means that message boxes and processor argument lists can contain Python expressions to be evaluated at object load time. See Figure 2 for some examples.

**Partial application syntax.** A special syntax allows for literals that represent partially-applied functions. When sent as a message, `@funcname([args], [kwargs])` will be consumed by its first recipient and applied to the recipient as a method call.

**Deferred evaluation syntax.** Another special syntax extension allows for literals that are not evaluated until they enter the processing network. A initial comma preceding an expression will wrap it in a Python `lambda` form that is not called until the message is sent. This allows for time-varying results from a "literal" message box.

<figure><img src="image_67.png" /><figcaption>Figure 2: Patch fragments demonstrating features of the MFP data model, including `via`s, `partial application`, and `deferred evaluation`</figcaption></figure>

## 3.2. Python Live-Coding

The message-passing system at the heart of MFP is implemented in Python, and there are several pathways enabling the user to access the standard Python library and/or write Python code directly into the patching interface.

**The func processor.** `[func]` is a very thin wrapper around Python's `lambda`. It allows you to write a single Python expression directly in a processor box. The arity of the lambda is

extracted via Python's `inspect`, and inlets are created for each argument.

**Library access.** Any callable Python object can be automatically wrapped as a MFP processor element. If you create a processor element called `re.split`, which is not an MFP builtin, it will look for a callable bound to that name. As with `func`, `inspect` is used to find its argument signature and create an object with the appropriate inputs.