**Custom processor types.** Automatic callable wrapping enables live coding of new message processors in Python. To create a custom processor type called `myfunc`, we can attach the Python code to define a function called `myfunc` directly to a processor via a "Custom code" parameter, and allow the autowrapping process to discover it.

**Offline coding.** A user-provided RC file, loaded at startup, can define new data types and builtin processor types using MFP's internal framework.

See Figure 3 for some examples of these mechanisms in use.

<figure><img src="image_68.png" /><figcaption>Figure 3: Several mechanisms for live-coding Python in MFP: `[func]`, `custom code`, and automatically wrapped Python library functions</figcaption></figure>

<figure><img src="image_69.png" /><figcaption>Figure 4: Live coding in Faust using the `[faust~]` object</figcaption></figure>

**Breakpoints and step execution.** MFP supports step debugging of patches with the `[bp]` breakpoint setter. A message encountering a `[bp]` causes the Python console to enter a simple debug mode, inspired by `pdb`, which allows for direct inspection of internal processor state and tracing of the marshalling and dispatch process.

**Snoop mode.** Enabling "snoop mode" on a connection briefly displays any messages passing through the connection.

**UI state inspector.** A UI state inspector, modeled after the Redox devtools inspector, is provided by the Flopsy state management library [3]. It allows for detailed analysis of the time sequence of changes to the UI.

### 3.3. Faust Live-Coding

The `[faust~]` object supports live coding of DSP algorithms in Faust. Code can be typed directly into the UI or loaded from files, and can be parameterized or customized via Python at compile time.

This is possible via the "Custom code" parameter mentioned above. The `[faust~]` processor looks for a special Python variable called `faust_code` in the custom code block. Each custom code block is specific to a processor, so each instance of `[faust~]` can have its own custom Faust code.

The Faust code can be defined as a constant, literal string (the Python triple-quote syntax works well to support a formatted Faust program embedded in a Python string), it can be the result of Python file operations to load from a named file, or it can be assembled by Python string formatting and manipulation operations.

Once compiled to LLVM within the `mfpdsp` process, the Faust DSP object is inspected to extract parameters and I/O channels so that the processor element's input/output profile matches the signature of the DSP code.

See Figure 4 for how this looks in practice. Note that the sliders controlling Faust parameters are part of the user-defined patch and are not automatically created from the compiled Faust code; only the input ports on the `[faust~]` object, and their metadata, are created this way. This is different from most Faust implementations, which automatically create the control UI from the parameter definitions in the Faust code. Automatic creation of the control UI is under consideration for future work.

### 3.4. Debugging

Since MFP is still very much a work in progress, a lot of the debugging functionality is built as much to debug MFP as it is to debug user patches, but it's useful for both.

**Python console.** There is an always-available Python console that can directly access and manipulate the app's data. For example, the default patch that is created when you open MFP with no arguments is `app.patches['default']`

### 3.5. MIDI and OSC

Every element in a patch can be controlled with MIDI or OSC messages. MIDI and OSC events can be easily handled, created, and sent within a patch.

**MIDI message types.** A set of Python types representing the various flavors of MIDI messages are available to patches: `NoteOn`, `NoteOff`, `MidiCC`, `MidiPitchbend`, etc. These types can be used to filter/route events by type with the `[route]` family processors. Fields are accessible using the `[get]` and `[set!]` processors.

**Automatic controls.** Every element has an OSC address and can accept OSC messages. The "Bindings" tab of the processor info display shows the currently active OSC routes and data types. OSC messages to the displayed address will be sent to the leftmost inlet of the targetted processor.

**Learning controls.** To connect an external MIDI controller to a processor, send the message `@midiLearn` to the processor or choose "Learn MIDI controller" from the context menu. The next MIDI events received will be analyzed and a filter and value accessor will be determined from the properties of the event.

**Explicit message handling.** The builtin processor types `[midi_in]`, `[midi_out]`, `[osc_in]`, and `[osc_out]` enable sending and receiving of MIDI and OSC messages outside the automatic and learned controllers on specific objects. Figure 1 shows some of these processors in action.

### 3.6. Patches as Plugins

An MFP patch can be saved as an LV2 plugin. There are limits on the live editing that can be done (for example, the number of inputs and outputs of the plugin are fixed at save time) but for the most part the patch works as normal.

LV2 plugins exist as "bundles" which are (to simplify) a shared library implementing the plugin, plus metadata, in the form of the `manifest.ttl` file (describing the plugin) and any additional plugin-specific metadata.

For an MFP patch, the shared library is the `libmfpdsp.so` library that implements the MFP DSP engine, and the plugin-
specific metadata includes the MFP patch save file. Since the Python runtime is an essential part of patch loading and operation, we need to make sure that it is loaded on plugin initialization.

Normally, when MFP is launched, the Python "main" process is launched first, and it forks the DSP engine and communicates with it through a socket. The LV2 entry points in `libmfpdsp.so` reverse this process, forking the Python main process (if it is not already running) in a mode that enables it to know what is happening.

Since the Python runtime also manages the MFP UI, the management of UI for the plugin is not currently left for the LV2 host to control via the normal mechanism of the LV2 UI extension. Rather, an "Edit" control added to the plugin causes the MFP graphical editor to open the patch. This may change in the future.

A unique, but probably not extremely useful, feature of the separation of LV2 backend from the Python main program is that messages could be passed between plugin instances hosted in different applications.

## 3.7. Dynamic Object and Scope Creation

When creating a patch as a versatile building block, it may be important to have flexibility to support dynamic numbers of inlets and outlets or other patch resources. For example, there's no right number of inputs for an "audio mixer" patch; it depends on the application how many channels you want to mix.

This could be accomplished by making separate patches for each useful number of channels, like `[mixer4]`, `[mixer8]`, etc. But a nicer way would be to make a single patch called `[mixer]` and create it with the number of channels as an argument, i.e. `[mixer 4]` for a 4-channel mixer. If done properly, the interface could scale up as well, showing a slider or even a full channel strip for each channel.

MFP enables this via a message called `@clonescope` that can be sent to the patch at load time. `@clonescope` makes a specified number of copies of all the objects in a specified *lexical scope*, creating a unique new scope for each set of copies.

Lexical scopes define how the names used to label processors and vias are resolved to actual objects. As in any language, the process of turning a name into a value starts looking in the local scope and becomes more global until the name can be resolved.

Objects in a patch default to their names being in the scope `__patch__`, which is the "global" scope in a patch, but each layer can set a different default scope for objects in itself, and each object can be directly assigned to any other scope via the parameter editor.

With multiple scopes existing in a patch, the same name (of a processor or send/receive via) can refer to different objects depending on the scope in which it is resolved. `@clonescope` uses this to make copies of entire scopes at once. The objects, their interconnections, and send/receive vias can all be duplicated cleanly by just changing the scope name for each object when copying.

A patch wanting to take advantage of this can inspect its own creation arguments, and then send itself a message to duplicate selected parts of its implementation as needed.

See Figure 5 for how this looks in a patch. ` togglegrid ` is a user-defined patch that creates an NxM grid of toggle buttons, each of which has an inlet and an outlet. The implementation is more than a single screenshot can capture, but the file `doc/togglegrid.mfp` is part of the MFP source distribution and can be opened to inspect how lexical scopes, layers, and `@clonescope` work together to make this happen.

<figure><img src="image_70.png" /><figcaption>Figure 5: *Dynamic object creation using the @clonescope message. *ogglegrid* is a user-defined patch that creates an NxM grid of toggle buttons.*</figcaption></figure>

## 3.8. Panel Mode

Panel mode is similar to Max's Presentation Mode. It gives the patch a curated user interface that shows only the controls that are important to patch performance, hiding the implementation details.

Each graphical element in a patch can be added to the panel via a boolean parameter. When panel mode is engaged for the patch (it is toggled with C-ESC), all elements that are "panel enabled" are displayed, regardless of which layer of the patch they are in. Each element can be independently positioned when in panel mode.

When a patch is opened as an element in another patch (by putting the patch name in a processor element) the patch's panel is displayed in the processor box.

Figure 6 shows the same patch in three ways: in patch editing mode, in "panel mode" within the patch window, and as an element in another patch. Note that, unlike Max, only the position of each element can be different in panel mode; size and other attributes are the same. The graphical element in the panel is included as a PNG image in a Markdown text element.

# 4. IMPLEMENTATION

## 4.1. Runtime Components

When running, MFP consists of several components working in concert.

**mfpmain:** A Python program that is the primary execution environment of MFP. It handles the processing of discrete message data, delegating all real-time signal processing to the mfpdsp process.

**mfpgui:** A Python program that implements the UI. It currently supports 2 backends: the original Clutter/Gtk UI and a newer one built with Dear ImGui.

**mfpdsp:** A C program that implements the DSP engine and JACK client interface.

These three are loosely coupled, primarily communicating via an asynchronous RPC mechanism implemented in the Carp [4] package. To enable the special case of displaying signals and other rapidly-changing values in the UI, shared memory buffers are used by the `[buffer~]` object to shortcut directly from the DSP engine to the UI.

Each runtime component implements its own representation of the patch's graph of processing nodes, so there are "parallel" representations of each patch built in the UI process, in the main process, and in the DSP engine, using representations specific to that process.
<figure><img src="image_71.png" /><figcaption>Figure 6: A basic Karplus-Strong synthesis patch displayed three ways: (top) in patch editing mode; (lower left) embedded as an element in another patch; (lower right) in panel mode (as might be seen when used as a plugin in another app)</figcaption></figure>

The DSP and UI processes are launched by default but may be suppressed with command line options. A no-DSP, no-UI "batch mode" is available to run a patch with input from `stdin` and write resulting objects to `stdout`.

Much of the functionality of `mfpdsp` is built in to a shared library called `libmfpdsp`. This library implements the LV2 plugin API, which allows MFP to be used as a plugin in any LV2-enabled host.

## 4.2. User Interface

As of version 0.8, the default user interface for MFP is built on the Dear ImGUI library [5], via the ImGUI Bundle bindings [6]. The move to ImGUI from the previous implementation in `Jakarta`/`Clutter` was somewhat painful, but worth it. ImGUI's flexibility and the rich ecosystem of extensions and addons made things like enabling Markdown in comments and making more sophisticated X/Y plots relatively easy.

## 4.3. Help System

MFP's help system is modeled after Pd's, with builtin processor types providing "help patches" that describe and demonstrate their functionality. Help patches are not yet available for most MFP builtins, but I am working on it.

In addition to the per-processor help (available in the context menu), there are some global help patches available under the app's Help menu. A Tutorial patch walks through the basics of creating a patch, and a Reference patch attempts to organize information about the builtins and how to use them.

# 5. RELATED WORK

MFP's "competitors", if that term is even appropriate when they are so far above MFP's capabilities and popularity, are Pd and Max/MSP. Both of them are amazing pieces of software that

have long track records, expansive feature sets, and large ecosystems of third-party developers building extensions and functionality at a rapid pace. For most users and use cases, either Pd or Max/MSP is likely to be a better option than MFP.

That being said, here are a few points of comparison:

**Data types.** MFP supports any Python data type, built-in or user-defined, for messages within the patch. Pd supports 6 types natively (numbers, symbols, lists of numbers and symbols, the special message `bang`, and pointers to data structures), with extensions providing support for other data types. Max supports a similar set of basic data types, adding Jitter matrices for visual data. MFP's use of Python data is especially powerful in areas like string manipulation, native sets and hash tables, and the ability to route messages based on their data type.

**Dataflow language.** MFP, Pd, and Max have many similarities as dataflow languages. MFP has a more consistent and flexible approach to object name resolution, using explicit scopes and a dotted-path syntax as opposed to Max's global/local and Pd's `$0`-prefix approach. MFP's partial-application syntax is, in my mind, cleaner than the "append" approach to constructing method calls in Max and Pd. The MFP "message" widget, which in Pd has its own domain-specific language for specifying multiple messages with destinations, is simpler and just does one thing – emit a message.

**Patch UI.** MFP provides a basic set of interactive widgets including buttons, sliders, dials, number boxes, message boxes, graphs, and comments with typographical formatting and image loading. Pd and especially Max are much more complete in this area, and some of the gaps are fairly embarrassing for MFP: the table/array widgets, for example, are very commonly used in Pd and Max, and have no equivalent in MFP; many very convenient "plumbing" tools, such as manual connection routing, are not available in MFP. Max's more "eye candy" widgets, such as the virtual keyboard, staff display of musical notes, and VU meter, are also missing.

**UX.** In overall appearance, MFP falls somewhere in the middle between Pd's stark black-on-white UI and Max/MSP's pol-
ished commercial one. The app's UI has a pretty complete menu system, a searchable log window, and an info panel to view and edit object properties. There are keyboard commands for almost every operation. One of the original motivations for MFP was frustration with Pd's UI, particularly the requirement to use the mouse to make connections between elements. In my mind, the keyboard-driven emphasis in MFP is a plus over both Pd and Max. I also prefer MFP's tiled UI to the multiple-window clutter of both Pd and Max. Don't be confused, though – Max's polish is altogether in another league, and the ability to customize with Javascript is powerful.

**Feature set.** MFP has a solid set of patching primitives, DSP processing blocks, and UI elements. It is strongest where the use of Python data allows patches to be more concise and expressive with fewer elements. Both Pd and Max have much larger sets of operations, built over years of user feedback and experience with building complex patches.

**Extensibility.** MFP can be extended via Python code, Faust, and C extension libraries. Pd can likewise be extended in various ways; I am not up to date, but I believe at least C, Faust, Lua, and Tcl/Tk are supported. Max/MSP can be extended with C/C++, Javascript, and Gen.

**Embedding in other apps.** MFP can be embedded as an LV2 plugin in any LV2-capable host, and can communicate via MIDI and OSC with the host. Max/MSP famously has "max4live", a deep integration with the Ableton Live DAW that gives a structured API for patches to interact with Ableton. Pd has Plugdata, a project which enables embedding Pd patches as an LV2, VST, or CLAP plugin, as well as the Heavy compiler that compiles PureData patches directly to C.

**Block size.** MFP is limited to using the JACK block size throughout. This can impact implementation of algorithms requiring feedback, such as Karplus-Strong synthesis (shown in the figure above). Faust code is recommended in such cases. Puredata has a `[block~]` primitive that enables smaller block-sizes, down to 1 sample, within the scope of a patch. Max/MSP has some flexibility for setting block-sizes as well, including use of `[gen~]` for sample-by-sample processing.

**Platform.** MFP has only been tested on Linux, though it might work on other platforms since the switch to ImGUI. Pd is available on MacOS, Windows, and Linux. Max/MSP is available on MacOS, Windows, and iOS.

**Availability.** MFP and Pd are free software with publically-available source code. Max/MSP is commercial software requiring a paid license and no so source-code distribution.

# 6. STATUS

The first public release of MFP, 0.1, was right about the time that the submission window for LAC-2013 closed. The most recent release, 0.8.2, is happening right around the close of the submission window for LAC-2025. Funny how that works!

A significant amount of the recent work on MFP has been in getting the Dear ImGUI UI up to parity with the Clutter UI. A lot of new features have been added along the way, but they were not the primary focus. Going forward, I would like to spend my time on (1) stabilizing existing features and (2) implementing items from the priorities list below.

I use MFP for my own music making and I think it's in good shape for others to use as well. As far as testing its capabilities to do more general-purpose programming, I have tackled the Advent of Code [7] challenge for the last 2 years, creating MFP patches for my solutions [8]. While I have yet to make it all the way through a month of puzzles, I blame myself and my holiday travel plans more than MFP's capability as a programming environment.

I have been fortunate to have the time and energy to work on MFP on a regular basis recently, and I hope to continue to do so.

There's not really a significant user community at this point, but only you can change that!

## 6.1. Development Priorities

To reach what I would consider to be a "1.0" level of functionality for MFP, I would like to implement most or all of the following:

* "mfp4ardour". Ardour has a deep OSC API, and building good wrappers (perhaps modeled after the way max4live does it) could make Ardour + MFP a great combo.

* **Expanded UI functionality.** The table widget, improved graphing options, simpler handling of oscilloscope displays, better connection routing and manual routing support, themeability, loading images and textures...

* **JACK MIDI.** For sample-accurate MIDI event processing and more capable LV2 implementation, it's necessary to build a library of builtin functions in the DSP domain to deal with JACK MIDI data.

* **LV2, CLAP, VST3 plugin hosting.** LV2, CLAP, and VST3 are the current standards for audio plugins, and they should be loadable in MFP. The current LADSPA-only `[plugin~]` is inadequate.

* **Spectral processing with libfft.** Embedded Faust gets a lot of great functionality, but does not enable working in the frequency domain. New DSP builtins will be needed for this.

* **Improved file format.** I would like to have a save format that includes resources such as sample data. I have done some preliminary work on using SQLite as a storage backend.

* **Multirate/variable-blocksize DSP operations.** Antialiasing, timestretch, and many other DSP algorithms require oversampling, decimation, and other primitives that are best implemented using variable sized blocks of data, at least internally. Currently MFP deals only in blocks that are the size of the JACK buffer.

* **Multichannel connections.** Recent versions of Max/MSP support multiple channels on single "wires" for easier multivoice patching. I think this should be a feasible extension to MFP.

## 6.2. Getting MFP

MFP is free software. Source code and issue tracking are available on GitHub:

https://www.github.com/bgribble/mfp

The project is licensed under the GNU General Public License (GPL) version 2. Your interest and participation are invited and welcomed.

# 7. REFERENCES

[1] M. Puckette, "Pure Data," http://www.puredata.info, 1997.

[2] B. Gribble, "Music for programmers (mfp): A dataflow patching language," Proceedings of Linux Audio Conference 2013, 2013.

[3] B. Gribble, "Flopsy," http://github.com/bgribble/flopsy, 2023.

[4] B. Gribble, "Carp," http://github.com/bgribble/carp, 2022.

[5] O. Cornut, "Dear ImGUI," http://www.github.com/ocornut/imgui, 2014.

[6] P. Thomet, "ImGUI Bundle," http://www.github.com/pthom/imguiOFFICE 2023.

[7] E. Wastl, "Advent of code," http://www.adventofcode.com, 2015.

[8] B. Gribble, "Advent of Code solutions," http://github.com/bgribble/advent_of_code, 2023.
# PD-LUA SIGNALS AND GRAPHICS

Albert Gräf

Computer Music Dept.
Johannes Gutenberg University (JGU)
Mainz, Germany
aggraef@gmail.com

Timothy Schoen

@timothyschoen
Utrecht, Netherlands
timschoen123@gmail.com

Benjamin Wesch

Tangible Music Lab
Kunstuniversität Linz, Austria
benjamin.wesch@kunstuni-linz.at

## ABSTRACT

Pd-Lua is a programming extension for Miller Puckette's Pd which lets you develop Pd objects in the Lua scripting language. Pd-Lua was originally designed for control processing only. The paper describes the new facilities for signal and graphics processing which have recently been added, so that you can now program pretty much any kind of Pd object in Lua.

## 1. INTRODUCTION

Pd-Lua lets you develop Pd externals in the Lua scripting language. It was originally written by Claude Heiland-Allen and has since been maintained by a number of people in the Pd community [3]. Pd-Lua also ships with a large collection of instructive examples which you'll find helpful when exploring its possibilities. Lua, from PUC Rio, is open-source (under the MIT license), mature, and supported by a large developer community [11]. It is a small programming language, but very capable and easy to learn. The Lua interpreter is efficient and light-weight, and has been designed to be easily embeddable in other environments. This makes it ideal as a Pd programming extension. For the same reasons, it has also been very popular in game development.

Pd-Lua was originally designed for control processing. The paper reports on some recent work in the 0.12 version to add signal and graphics processing. This effort was spearheaded by Timothy Schoen who designed the API and implemented the (vanilla) Pd and plugdata back-ends, while Benjamin Wesch added the multi-channel support, and Albert Gräf worked on the Purr Data integration. The new release is fully backward-compatible with Pd-Lua 0.11 (and earlier). While the signal and graphics support has been the main focus of the 0.12 release, there are also various other bugfixes and improvements, such as advanced live-coding support [7]. Moreover, Benjamin Wesch reworked our GitHub CI framework so that new releases are now published immediately on Pd's package manager Deken [6].

Pd-Lua works inside any reasonably modern Pd flavor. This encompasses vanilla Pd [14], of course, but also Purr Data [5] which includes an up-to-date version of Pd-Lua for Lua 5.4 and has it enabled by default, so you should be ready to go immediately; no need to install anything else. The same is true for plugdata (version 0.9.1 or later), Timothy Schoen's Pd flavor which can also run as a plug-in inside a DAW [15]. With vanilla Pd, you can install the pdlua package from Deken. You can also compile Pd-Lua from source, using the Github repository [3]. Compilation instructions are in the README, and you'll also find some Mac and Windows binaries there. In either case, after installing Pd-Lua from source or Deken, you also have to add pdlua to Pd's startup libraries.

A general introduction to Pd-Lua is beyond the scope of this paper, instead we refer the reader to the pd-lua tutorial which contains many step-by-step instructions and examples [8]. In the following, we'll first discuss how to write a signal processing (a.k.a. dsp) object in Pd-Lua, and then go on to show the implementation of a simple GUI object using the graphics API. The examples presented in this paper are also available for your perusal in the tutorial/examples folder in the sources.

## 2. SIGNALS

Enabling signal processing in a Pd-Lua object involves three ingredients:

1. **Adding signal inlets and outlets:** As in previous Pd-Lua versions, this is done by setting the `inlets` and `outlets` member variables in the `initialize` method. But instead of setting each variable to just a number, you specify a *signature*, which is a table indicating the signal and control in- and outlets with the special `SIGNAL` and `DATA` values. The number of in- and outlets is then given by the size of these tables. Thus, e.g., you'd use `self.inlets = { SIGNAL, SIGNAL, DATA }` if you need two signal and one control data inlet, in that order. Note that a number as the value of `inlets` or `outlets` corresponds to a signature with just `DATA` values in it.

2. **Adding a dsp method:** This step is optional. The `dsp` method gets invoked whenever signal processing is turned on in Pd, passing two parameters: `samplerate` and `blocksize`. The former tells you about the sample rate (number of audio samples per second) Pd runs at, which will be useful if your object needs to translate time and frequency values from physical units (i.e., seconds, milliseconds, and Hz) to sample-based time and frequency values, so usually you want to store the given value in a member variable of your object. The latter specifies the block size, i.e., the number of samples Pd expects to be processed during each call of the `perform` method (see below). You only need to store that number if your object doesn't have any signal inlets, so that you know how many samples need to be generated. Otherwise the block size can also be inferred from the size of the `in` tables passed to the `perform` method. Adding the `dsp` method is optional. You only have to define it if the signal and control data processing in your object requires the `samplerate` and `blocksize` values, or if you need to be notified when dsp processing gets turned on or the signal processing chain changes for some other reason.

3. **Adding a perform method:** This method is where the actual signal processing happens. It receives blocks of signal data from the inlets through its arguments, where each block is represented as a Lua table containing floating point sample values. The method then needs to return a tuple of similar Lua tables with the blocks of signal data for each outlet. Note that the number of arguments of the method matches the number of signal `inlets`, while the number of return values corresponds to the number of signal `outlets`. The `perform` method is not optional; if your object outputs any signal data, the method needs to be implemented, otherwise you'll get a "perform: function should return a table" or similar error in the Pd console as soon as you turn on dsp processing.

In addition to the `dsp` and `perform` methods, your object may contain any number of methods doing the usual control data processing on the `DATA` inlets. It is also possible to receive
control data on the `SIGNAL` inlets; however, you won't be able to receive `float` messages, because they will be interpreted as constant signals which get passed as blocks of signal data to the `perform` method instead.

## 2.1. Example 1: Mixing signals

Let us take a look at a few simple examples illustrating the kind of processing the `perform` method might do. For starters, let's mix two signals (stereo input) down to a single (mono) output by computing the average of corresponding samples (see Fig. 1).

<figure><img src="image_72.png" /><figcaption>Figure 1: Mixing signals.</figcaption></figure>

We need two signal inlets and one signal outlet, so our `initialize` method looks like this:

```javascript
local foo = pd.class("foo")

function foo:initialize sel, atoms
    self.inlets = {SIGNAL, SIGNAL}
    self.outlets = {SIGNAL}
    return true
end
```

And here's the `perform` method (in this simple example we don't need `foo:dsp()`):

```javascript
function foo:perform(in1, in2)
    for i = 1, #in1 do
        in1[i] = (in1[i]+in2[i])/2
    end
    return in1
end
```

Note that here we replaced the signal data in the `in1` table with the result, so we simply return the modified `in1` signal; no need to create a third `out` table. (This is safe because it won't actually modify any signal data outside the Lua method.)

## 2.2. Example 2: Analyzing a signal

A `dsp` object can also have no signal outlets at all if you just want to process the incoming signal data in some way and output the result through a normal control outlet. E.g., here's one (rather simplistic) way to compute the rms (root mean square) envelope of a signal as control data (see Fig. 2):

```javascript
function foo:initialize sel, atoms
    self.inlets = {SIGNAL}
    self.outlets = {DATA}
    return true
end
```

```javascript
function foo:perform(in1)
    local rms = 0
    for i = 1, #in1 do
        rms = rms + in1[i]*in1[i]
    end
    rms = math sqrt(rms/#in1)
    self:outlet(1, "float", {rms})
end
```

<figure><img src="image_73.png" /><figcaption>Figure 2: Analyzing a signal.</figcaption></figure>

## 2.3. Example 3: Generating a signal

Conversely, we can also have an object which converts control inputs into signal data, such as this little oscillator object which produces a sine wave (see Fig. 3):

```javascript
function foo:initialize sel, atoms
    self.inlets = {DATA}
    self.outlets = {SIGNAL}
    self.phase = 0
    self.freq = 220
    self.amp = 0.5
    return true
end

-- message to set frequency...
function foo:in_1_freq(atoms)
    self.freq = atoms[1]
end

-- ... and amplitude.
function foo:in_1_amp(atoms)
    self.amp = atoms[1]
end

function foo:perform()
    local freq = self.freq
    local amp = self.amp

    -- calculate the angular frequency
    local angular_freq = 2 * math.pi * freq
    / self.samplerate

    local out = {} -- result table
    for i = 1, self.blocksize do
        out[i] = amp * math.sin(self.phase)
        self.phase = self.phase +
            angular_freq
        if self.phase >= 2 * math.pi then
```
```python
self.phase = self.phase - 2 *
    math.pi
end
end

return out
end
```

<figure><img src="image_74.png" /><figcaption>Figure 3: Generating a signal.</figcaption></figure>

## 2.4. Multi-channel support

The examples above all employ Pd's standard single-channel audio signals. But Pd-Lua also has support for Pd's multi-channel signals [4]. This requires Pd 0.54 or later (all signals are single-channel in earlier Pd versions). In Pd-Lua 0.12.20 and later, the `dsp` method receives a *third* `nchannels` argument. This is a Lua table which tells you about the number of channels in the signal data for each inlet. This will be 1 for normal Pd signals, but if `nchannels[i] > 1`, then your `perform` method should be prepared to process that many channels worth of sample data (i.e., the ith table argument of `perform` will actually contain `nchannels[i] * blocksize` samples – a block of `blocksize` samples for the first channel, followed by another block of `blocksize` samples for the second channel, etc.).

For the *output* signals, you can set up the desired number of channels per outlet in the `dsp` method, using the `signal_setmultiout` method which is also new in Pd-Lua 0.12.20. In that case your Lua table with the output samples should contain the `blocksize` samples for channel 1 followed by the `blocksize` samples for channel 2, etc., using the same layout as the tables for multi-channel input signals. We won't go into this here any further, but you can find a simple example for your perusal in the examples/multichannel folder in the Pd-Lua source.

## 3. GRAPHICS

Pd-Lua's new graphics API provides you with a way to equip an object with a static or animated graphical display inside its object box on the Pd canvas. Typical examples would be various kinds of wave displays, or custom GUI objects consisting of text and simple geometric shapes. To these ends, you can adjust the size of the object box to any width and height you specify. Inside the box rectangle you can then draw text and basic geometric shapes such as lines, rectangles, circles, and arbitrary paths, through stroke and fill operations using any `rgb` color.

In order to enable graphics in a Pd-Lua object, you have to provide a `paint` method. This receives a graphics context `g` as its argument, which lets you set the current color, and draw text

and the various different geometric shapes using that color. In addition, you can provide methods to be called in response to mouse down, up, move, and drag actions on the object, which is useful to equip your custom GUI objects with mouse-based interaction.

Last but not least, the `set_args` method lets you store internal object state in the object's creation arguments, while `get_args` lets you retrieve those arguments. This is useful if you need to keep track of persistent state when storing an object on disk (via saving the patch) or when duplicating or copying objects. Also, their companion `canvas_realizeddollar` method allows you to expand symbols containing `"$"` patch arguments like `$0`, `$1`, etc. These three are often combined, but they can also be used separately, and they work just as well with ordinary Pd-Lua objects which don't utilize the graphics API.

We use a custom GUI object, a simple kind of dial, as a running example to illustrate most of these elements in the following subsections. To keep things simple, we will not discuss the graphics API in much detail here, so you may want to check the graphics subpatch in the main `pdlua-help` patch, which contains a detailed listing of all available methods for reference.

### 3.1. A basic dial object

Let's begin with a basic clock-like dial: just a circular *face* and a border around it, on which we draw a *center point* and the *hand* (a line) starting at the center point and pointing in any direction which indicates the current *phase* angle. See Fig. 4.

<figure><img src="image_75.png" /><figcaption>Figure 4: Basic dial.</figcaption></figure>

Following the clock paradigm, we assume that a zero phase angle means pointing upwards (towards the 12 o'clock position), while +1 or -1 indicates the 6 o'clock position, pointing downwards. Phase angles less than -1 or greater than +1 wrap around. Positive phase differences denote clockwise, negative differences counter-clockwise rotation. And since we'd like to change the phase angle displayed on the dial, we add an inlet taking float values.

Here's the code implementing the object initialization and the float inlet:

```python
local dial = pd.class("dial")

function dial:initialize sel, atoms
    self.inlets = 1
    self.outlets = 0
    self.phase = 0
    self:resize(127, 127)
    return true
end

function dial:in_1_float(x)
    self.phase = x
```
```R
self:repaint()
end
```

The `self:set_size()` call in the `initialize` method sets the pixel size of the object rectangle on the canvas (in this case it's a square with a width and height of 127 pixels). Also note the call to `self:repaint()` in the float handler for the inlet, which will redraw the graphical representation after updating the phase value.

We mention in passing that `self:repaint()` always re-draws everything, in this example the face, border, center point, and hand, even though only the hand will change with the phase angle. As of Pd-Lua 0.12.19, it is also possible to partition your graphics into *layers*, each with their own paint method, so that you only need to repaint layers that actually changed, which will improve rendering performance for complex drawings.

We still have to add the `dial:paint()` method to do all the actual drawing:

```R
function dial:paint(g)
    local width, height = self: get_size()
    local x, y = self: tip()

    -- object border, fill with bg color
    g: set_color(0)
    g: fill_all()

    -- dial face
    g: fill_ellipse(2, 2, width - 4, height - 4)
    g: set_color(1)
    -- dial border
    g: stroke_ellipse(2, 2, width - 4, height - 4, 4)
    -- center point
    g: fill_ellipse(width/2 - 3.5, height/2 - 3.5, 7, 7)
    -- dial hand
    g: draw_line(x, y, width/2, height/2, 2)
end
```

The existence of the `paint` method tells Pd-Lua that this is a graphical object. As mentioned before, this method receives a graphics context `g` as argument. The graphics context is an internal data structure keeping track of the graphics state of the object, which is used to invoke all the graphics operations. The `set_color` method of the graphics context is used to set the color for all drawing operations; in the case of `fill` operations it fills the graphical element with the color, while in the case of `stroke` operations it draws its border. There's just one color value, so we need to set the desired fill color in case of `fill`, and the desired stroke color in case of `stroke` operations. The color values 0 and 1 we use in this example are predefined, and indicate the default background color (usually white) and default foreground color (usually black), respectively.

It is possible to choose other colors by calling

`g: set_color(r, g, b)` with rgb color values instead, with each `r`, `g`, `b` value ranging from 0 to 255 (i.e., a byte value). For instance, the color "teal" would be specified as 0, 128, 128, the color "orange" as 255, 165, 0, "black" as 0, 0, 0, "white" as 255, 255, 255, etc. It's also possible to add a fourth `alpha` a.k.a. opacity value `a`, which is a floating point number in the range 0-1, where 0 means fully transparent, 1 fully opaque, and any value in between will blend in whatever is behind the graphical element to varying degrees. As of Pd-Lua 0.12.7, `alpha` values are fully supported in both `plugdata` and `Purr Data`. In vanilla Pd they are simply ignored at present, so all graphical objects will be opaque no matter what `alpha` value you specify.

Let's now take a closer look at the drawing operations themselves. We start out by filling the entire object rectangle, which is our drawing surface, with the default background color 0, using `g:fill_all()`.

This operation is special in that it not only fills the entire object rectangle, but also creates a standard border rectangle around it. If you skip this, you'll get an object without border, which may be useful at times.

We then go on to fill a circle with the background color, the dial's face. The graphics API has no operation to draw a circle, so we just draw an ellipse instead. The coordinates given to `g:fill_ellipse()` are the coordinates of the rectangle surrounding the ellipse. In this case the width and height values are what we specified with `self: set_size(127, 127)` in the `initialize` method, so they are identical, and thus our ellipse is in fact a circle. Also note that we make the ellipse a little smaller and put it at a small offset from the upper left corner, so the actual width and height are reduced by 4 and the shape is centered in the object rectangle (or square, in this case).

Note that we could have skipped drawing the face entirely at this point, since it just draws a white circle on white background. But we could make the face a different color later, so it's good to include it anyway.

After the face we draw its border, drawing the same ellipse again, but this time in the default foreground color and with a stroke width of 4. We then go on to draw the remaining parts, a small disk in the center which mimics the shaft on which the single hand of the dial is mounted, and the hand itself, which is just a simple line pointing in a certain direction.

Which direction? The line representing the hand goes from the center point width/2, height/2 to the point given by the `x`, `y` coordinates. Both width, height and `x`, `y` are calculated and assigned to local variables at the beginning of the `paint` method:

```R
local width, height = self: get_size()
local x, y = self: tip()
```

The `get_size()` call employs a built-in method which returns the current dimensions of the object rectangle; this is part of the graphics API. We could have used the constant 127 from the `initialize` method there, but we could change the size of the object rectangle later, so it's better not to hard-code the size in the `paint` method.

The `tip()` method we have to define ourselves. It is supposed to calculate the coordinates of the tip of the hand. We have factored this out into its own routine right away, so that we can reuse it later when we add the mouse actions. Here it is:

```R
function dial:tip()
    local width, height = self: get_size()
    local x, y = math: sin(self: phase * math: pi), -math: cos(self: phase * math: pi)
    x, y = (x/2*0.8+0.5) * width, (y/2*0.8+0.5) * height
    return x, y
end
```

This just converts the position of the tip from polar coordinates (1, phase) to rectangular coordinates (`x`, `y`) and then translates and scales the normalized coordinates to pixel coordinates in the object rectangle which has its center at (width/2, height/2). We also put the tip at a normalized radius of 0.8 so that it is well within the face of the dial. Moreover, the formula computing the `x`, `y` pair accounts for the fact that the `y` coordinates of the object rectangle are upside-down (0 is at the top), and that we want the center-up (a.k.a. 12 o'clock) position to correspond to a zero phase angle. Hence the sin and cos terms have been swapped and the cos term adorned with a minus sign compared to the standard polar-rectangular conversion formula.

So now that we hopefully understand all the bits and pieces, here's the Lua code of the object in its entirety again:

```R
local dial = pd.class("dial")

function dial:initialize sel, atoms)
    self.inlets = 1
```
```python
self.outlets = 0
self.phase = 0
self:phase = 0
self.set_size(127, 127)
return true
end

function dial:in_1_float(x)
    self.phase = x
    self:repaint()
end

-- calculate the x, y position of the tip
-- of the hand
function dial:tip()
    local width, height = self: get_size()
    local x, y = math. sin(self. phase * math. pi), -math. cos(self. phase * math. pi)
    x, y = (x/2*0.8+0.5)*width, (y/2*0.8+0.5)*height
    return x, y
end

function dial:paint(g)
    local width, height = self: get_size()
    local x, y = self: tip()

    -- object border, fill with bg color
g: set_color(0)
g: fill_all()

    -- dial face
g: fill_ellipse(2, 2, width - 4, height - 4)
g: set_color(1)
    -- dial border
g: stroke_ellipse(2, 2, width - 4, height - 4, 4)
    -- center point
g: fill_ellipse(width/2 - 3.5, height/2 - 3.5, 7, 7)
    -- dial hand
g: draw_line(x, y, width/2, height/2, 2)
end
```

## 3.2. Adding an outlet

We can already send phase values into our dial object, but there's no way to get them out again. So let's add an outlet which lets us do that. Now that the grunt work is already done, this is rather straightforward. First we need to add the outlet in the `initialize` method:

```python
self.outlets = 1
```

And then we just add a message handler for `bang` which outputs the value on the outlet:

```python
function dial:in_1_bang()
    self:outlet(1, "float", {self.phase})
end
```

Fig. 5 shows how our patch looks like now.

## 3.3. Mouse actions

Our dial now has all the basic ingredients, but it still lacks one important piece: Interacting with the graphical representation itself using the mouse. The graphics API makes this reasonably easy since it provides us with four callback methods that we can implement. Each of these gets invoked with the current mouse coordinates relative to the object rectangle:

<figure><img src="image_76.png" /><figcaption>Figure 5: Adding an outlet.</figcaption></figure>

* `mouse_down(x, y)`: called when the mouse is clicked

* `mouse_up(x, y)`: called when the mouse button is released

* `mouse_move(x, y)`: called when the mouse changes position while the mouse button is *not* pressed

* `mouse_drain(x, y)`: called when the mouse changes position while the mouse button is pressed

Here we only need the `mouse_down` and `mouse_drain` methods which let us keep track of mouse drags in the object rectangle in order to update the phase value and recalculate the tip of the hand. Here's the Lua code. Note that the `mouse_down` callback is used to initialize the `tip_x` and `tip_y` member variables, which we keep track of during the drag operation, so that we can detect in `mouse_drain` when it's time to output the phase value and repaint the object:

```lua
function dial:mouse_down(x, y)
    self/tip_x, self/tip_y = self:tip()
end

function dial:mouse_drain(x, y)
    local width, height = self: get_size()
    local x1, y1 = x/width-0.5, y/height-0.5
    -- calculate the normalized phase,
    -- shifted by 0.5, since we want zero
    -- to be the center up position
    local phase = math. atan(y1, x1)/math.pi + 0.5
    -- renormalize if we get an angle > 1,
    -- to account for the phase shift
    if phase > 1 then
        phase = phase - 2
    end

    self.phase = phase

    local tip_x, tip_y = self:tip();

    if tip_x =~ self/tip_x or tip_y =~ self/tip_y then
        self/tip_x = tip_x
        self/tip_y = tip_y
        self:in_1_bang()
        self:repaint()
    end
end
```
## 3.4. More dial action: clocks and speedometers

Now that our dial object is basically finished, let's do something interesting with it. The most obvious thing is to just turn it into a clock (albeit one with just a seconds hand) counting off the seconds. For that we just need to add a metro object which increments the phase angle and sends the value to the dial each second. See Fig. 6.

<figure><img src="image_77.png" /><figcaption>Figure 6: A clock.</figcaption></figure>

Pd lets us store the phase angle in a named variable (the `v phase` object) which can be recalled in an `expr` object doing the necessary calculations. The `expr` object sends the computed value to the `phase` receiver, which updates both the variable and the upper numbox, and the numbox then updates the dial. Note that we also set the variable whenever the dial outputs a new value, so you can also drag around the hand to determine the starting phase. And we added a `0` message to reset the hand to the 12 o'clock home position when needed.

Fig. 7 shows another little example, rather useless but fun, simulating a speedometer which just randomly moves the needle left and right.

<figure><img src="image_78.png" /><figcaption>Figure 7: A speedometer.</figcaption></figure>

We're sure that you can imagine many more creative uses for this simple but surprisingly versatile little GUI object, which we did in just a few lines of fairly simple Lua code. An extended version of this object, which covers some more features of the graphics API that we didn't discuss here to keep things simple, can be found as `dial.pd` and `dial.pdinsona` in the `pd-lua` tutorial examples.

## 4. REAL-WORLD EXAMPLES

To complement the previous examples, let's examine two more complex applications that demonstrate what's possible with Pd-Lua's graphics capabilities in real-world scenarios.

### 4.1. Multi-channel signal inspector

<figure><img src="image_79.png" /><figcaption>Figure 8: Screenshot of the `[show~]` object in Pure Data, showing a 31-channel signal of a spherical speaker array on a truncated icosahedron. The visualization reveals the delays applied for beam forming and vector based amplitude panning.</figcaption></figure>

The `[show~]` object combines a multi-channel waveform view with a panel on the right displaying RMS levels and average values of each channel [17]. The object implements several interactive features:

* Mouse-based zooming through dragging to inspect signals in detail or on a larger scale

* Channel highlighting on hover to focus on specific signals

This combination of multi-channel signal processing and interactive graphics makes it a simple, but valuable tool for analysis and debugging.

### 4.2. DualSense controller visualization

<figure><img src="image_80.png" /><figcaption>Figure 9: Screenshot of the `[dsshow]` object in Pure Data, displaying the controller state through input from a HID-based external.</figcaption></figure>

This graphical object provides an intuitive way to monitor and demonstrate controller interaction in Pure Data, creating a detailed visual representation of a DualSense controller that shows real-time state of buttons, analog sticks, touchpad, and gyro data [16].

The implementation demonstrates how Pd-Lua's graphics API can handle complex, responsive GUI elements. With the upcoming SVG support (see Section 5), it will be possible to create this kind of visualization in a much more efficient way.
## 5. FURTHER DEVELOPMENT

While the current graphics API is highly versatile, certain features could not be implemented due to inherent limitations in vanilla Pd's graphics engine. These limitations include the ability to fill gradients, have rotation transforms, or have paths intersections handled with a specific fill rule (even-odd or nonzero).

To address these limitations and make Pd-Lua truly a complete vector graphics renderer, a new function "draw SVG" is being added to the graphics API. This provides near-complete support for the Scalable Vector Graphics format, enabling aforementioned features such as gradients, path intersections and rotation transforms using Mikko Mononen's NanoSVG library [12]. The restrictions of vanilla Pd's rendering engine are overcome by rasterizing the SVG into an image. The API usage is as follows:

```R
function example:paint(g)
  local svg = [[
    <svg width="24" height="24" fill="none" viewBox="0 0 24 24"><path
d="M8 12a4 4 0 1 1 8 0 4 4 0 0 1-8 0Z" fill="#212121"/></svg>
  ]]]
  local x = 10, y = 10
g:draw SVG(svg, x, y)
end
```

More ongoing work includes expanding the mouse interaction API with `mouse_enter(x, y)` and `mouse_exit(x, y)` callbacks to track whether the mouse is currently over the object. Additionally, an option to choose text justification when rendering text is also in development.

## 6. CONCLUSION

Pd externals are usually programmed using C, the same programming language that Pd itself is written in. But novices may find C difficult to learn, and the arcana of Pd's C interface may also be hard to master. Programming your externals in Lua makes this much easier and opens up many possibilities.

To help with that, the new capabilities of Pd-Lua 0.12 described in this paper vastly extend the scope of Pd-Lua applications, as you can now program pretty much any kind of Pd object in Lua, covering both signal and control processing, as well as custom GUI objects. Thus, next time you run into a Pd programming problem which cannot be solved easily with a Pd abstraction or an existing external, you may want to consider using Pd-Lua to implement the functionality that you need.

There are some alternatives to pd-lua worth considering if you'd like to program Pd objects with signal processing and/or graphics functionality. py4pd is a relatively new project which is based on the Python scripting language and offers some facilities to draw scores and perform sound analysis [13]. If your main focus is on audio signal processing, then you should look at Grame's Faust programming language [1]. This is a high-level functional programming language specifically tailored to sound processing which compiles to efficient native code. Employing Faust's just-in-time compiler back-end based on the LLVM compiler framework, there's support for running Faust programs in both Max and Pd [2, 9, 10].

## 7. ACKNOWLEDGEMENTS

The authors would like to thank the anonymous reviewers for their suggestions.

## 8. REFERENCES

[1] Faust Programming Language. https://faust.grame.fr/.

[2] faustgen. https://github.com/grame-cncm/faust/tree/master-dev/embedded/faustgen.

[3] pd-lua. https://agraef.github.io/pd-lua/.

[4] Pd Manual - Chapter 2: Theory of operation. https://msp.ucsd.edu/Pd_documentation/resources/chapter2.htm#s2.5.6.

[5] purr-data. https://agraef.github.io/purr-data/.

[6] pure-data/deken, Feb. 2025. https://github.com/pure-data/deken.

[7] A. Gräf. pdx.lua - advanced live-coding support. https://agraef.github.io/pd-lua/tutorial/pd-lua-intro.html#pdxlua.

[8] A. Gräf. A Quick Introduction to Pd-Lua. https://agraef.github.io/pd-lua/tutorial/pd-lua-intro.html.

[9] A. Gräf. pd-faustgen2, Apr. 2025. https://github.com/agraef/pd-faustgen.

[10] P. Guillot. pd-faustgen: The FAUST compiler embedded in a Pd external. https://github.com/CICM/pd-faustgen.

[11] R. Ierusalimschy, L. H. d. Figueiredo, and W. Celes. The Implementation of Lua 5.0. JUCS - Journal of Universal Computer Science, 11(7):1159-1176, July 2005.

[12] M. Mononen. memononen/nanosvg, Mar. 2025. https://github.com/memononen/nanosvg.

[13] C. K. Neimog. py4pd: Bringing Python to PureData, May 2025. https://github.com/charlesneimog/py4pd.

[14] M. Puckette. Software by Miller Puckette. https://msp.ucsd.edu/software.html.

[15] T. Schoen. plugdata: A visual programming environment for audio experimentation, prototyping and education. https://plugdata.org/.

[16] B. Wesch. ben-wes/pd-dualsense, Mar. 2025. https://github.com/ben-wes/pd-dualsense.

[17] B. Wesch. ben-wes/pdlua-show_tilde, Mar. 2025. https://github.com/ben-wes/pdlua-show_tilde.
LAC-25 Paper Session 6
# RAPTOR, THE RANDOM ARPEGGIATOR VERSION 7

Albert Gräf

Computer Music Dept., Institute of Art History and Musicology

Johannes Gutenberg University (JGU) Mainz, Germany

aggraef@gmail.com

## ABSTRACT

The paper is about the latest version of Raptor, the author's randomizing arpeggiator. Version 7 is written in Lua and available as both an Ardour plugin and a Pd external and patch. We give a brief overview of Raptor's functionality, report on its latest advancements, and describe how it can be used in Ardour and Pd.

In memory of Clarence Barlow (27 December 1945 – 29 June 2023).

## 1. INTRODUCTION

Arpeggiators are a kind of auto-accompaniment function of synthesizers and other music-making gear which have been around at least since the 1970s and to this day are used heavily in many styles of electronic and computer music [8]. They are similar to step sequencers, typically offering a variety of different patterns and controls for tempo, octave range, velocity, note length (gate) and repeats, and sometimes probability. However, in contrast to most step sequencers, arpeggiators adjust automatically to the chords played by the musician. Raptor offers all the basic functions of an arpeggiator, but it generates notes in a semi-random fashion based on the mathematical music theories of the late contemporary composer and computer music pioneer Clarence Barlow. It thus offers a much greater variety of patterns and a lot of parameters to control the generated arpeggios.

Raptor is available as free software under the GNU Public License [4]. In the following, we give a brief account of what's new in the current Raptor version and an overview of Raptor's functionality and control parameters. We also illustrate its use in Ardour and Pd by way of some simple examples. More details can be found in the online documentation available under the GitHub link cited above.

## 2. WHAT'S NEW

Raptor has been around for quite some time. Its original stand-alone version (version 4) can be traced back to 2005. This was the first "production" version actually used in a concert within the framework of the "Kultursommer Rheinland-Pfalz" 2005. Version 5 is a port of this version to the author's Pure programming language which consists of a Pd external and main patch. In version 6, the external was then ported to Lua to facilitate its installation and improve cross-platform compatibility; in particular, this made it possible to run Raptor on ARM-based devices such as the Raspberry Pi.

Using Lua also opened up the possibility to implement Raptor as an Ardour plugin which is now included in Ardour as of Ardour version 8 [1], along with some other, simpler arpeggiator plugins also written in Lua [3]. This version 7 of Raptor also offers some important new features, such as latch mode and an improved looper (a.k.a. ostinato) function. The accompanying Pd version is a backport of the Ardour plugin, and thus 100% compatible in terms of the underlying arpeggiator core, as well as the parameters and factory presets. (Earlier Raptor versions offered a similar, but not fully compatible, set of parameters.) While it is ultimately based on Raptor 6, the Pd patch was completely rewritten to provide a better and simplified user interface,

and besides latch and looper modes also provides some new features specific to the Pd version, such as improved transport functions, a MIDI learn facility, and much improved built-in support for a bunch of popular MIDI controllers. This makes it easier to control the algorithm's many parameters in a live performance situation.

## 3. THE ARPEGGIATOR

A complete discussion of the underlying algorithm is beyond the scope of this paper, instead we refer the reader to Barlow's original work [5, 6] and the author's ICMC 2006 paper [7] for details. However, to make the paper mostly self-contained, we have also summarized the mathematical formulas for the measures underlying Barlow's methods in Appendix A.

Raptor is really an algorithmic composition program driven by note input. It keeps track of the chords you play like any arpeggiator, but the note output it generates is in general much more varied than just playing back the input notes in a given pattern. This functionality is enabled by the special "raptor" control. If it is turned off, Raptor will still apply some basic parameters to the velocities and note filters, and produce a traditional arpeggio from the notes you play. If it is turned on, however, Raptor will pick notes more or less at random from a set of *candidate notes* determined by various harmonic criteria. Depending on the parameter settings, the generated notes may be close to the input, or at least harmonically related, or they may be entirely different, so the output can change from tonal to atonal and even anti-tonal in a continuous way. Moreover, Raptor can also vary the tonal strength automatically for each step based on the corresponding pulse weights.

Raptor's note generation process is quite involved, but at the heart of it there are just two basic "musiquantical" notions and corresponding measures from Barlow's theories: *meter* (which determines a kind of one-to-one pulse weights called *indispensabilities*) and *harmonicity* (a measure for the consonance of intervals calculated from so-called *indigestibilities*). In fact, there are a lot of similarities between Raptor and Barlow's groundbreaking Autobusk program which is based on the same concepts [2]. Both programs can operate in real-time, but Autobusk is driven exclusively by parameter input and will happily produce a constant stream of notes as soon as you turn it on. In contrast, Raptor never becomes "creative" on its own, it *always* requires note input.

Thus, in Raptor harmonicsities are used to *filter* candidate notes in relation to its note input (the notes you play). In other words, they determine *what* to play. On the other hand, the pulse weights are used to *assign* various parameters to each step in the pattern, such as note velocities and probabilities, and gate values. That is, they determine *how* to play the notes picked by the filter. The arpeggiator orchestrates the entire process, by taking note input from the musician, feeding the required data to the various parts of the algorithm in each step, and playing back the resulting stream of notes.

One rather unusual feature of Raptor's algorithm is the way that the Barlow indispensabilities drive the entire process by modulating various other parameters. Like in Autobusk, the pulse strengths directly affect note velocity and probability, but in Raptor they also modulate harmonicity and thereby change
the note selection process itself. Another unique feature of the algorithm is the harmonic preference parameter, which lets you prioritize notes by harmonicity, and can also be modulated by pulse strength in an automatic fashion.

In the following we give a brief rundown of Raptor's control parameters as they pertain to both the Ardour and the Pd version. All parameters can be changed in real-time. This is also the data that gets stored in each Raptor preset. Raptor includes various factory presets for illustration purposes which you can use as a starting point for your own presets.

Ranges are given in parentheses. Switches are denoted 0/1 (off/on). Continuous values use the 0 - 1 or -1 - +1 range and can be arbitrary floating point values. These generally denote probabilities or other normalized values (such as indispensabilities, harmonicities, and modulation values) expressed as fractions; some of these (in particular, the modulation parameters) can also be negative to reverse polarity. Other ranges denote integer values, such as MIDI channel and program numbers, MIDI note offsets, or enumerations such as the pattern and pitch tracker modes.

## 3.1. MIDI Controls

These controls let you change the MIDI program and the MIDI channels for input and output.

* pgm (0-128): Sets the MIDI program (instrument sound) of a connected synthesizer. pgm = 0 (the default) means no change, otherwise a MIDI program change message is sent to the output channel.

* inchan, outchan (0-16): Sets the MIDI input and output channels. inchan = 0 (omni) means that notes on all channels will be received, otherwise MIDI input on all other channels will be ignored. outchan = 0 means that output goes to the input channel; otherwise it goes to the given MIDI channel. The default is inchan = outchan = 0, which means that MIDI input will be received on all channels and output goes to the last channel on which input was received.

* transp (-64 - 64): Transposes the output of the arpeggiator by the given number of semitones. (Pd only.)

## 3.2. Arpeggiator Modes

These controls are 0/1 switches which control various global modes of the arpeggiator.

* bypass, latch, mute (0/1): The bypass toggle, when engaged, suspends the arpeggiator and passes through its input as is. This provides a means to monitor the input to the arpeggiator, but can also be used as a performance tool. In latch mode, the arpeggiator keeps playing if you release the keys on your MIDI keyboard, which makes changing chords easier. The mute control suppresses note output of the arpeggiator (the arpeggiator still keeps tracking input notes, so that it is ready to go immediately when you unmute it again).

* raptor (0/1): Toggles raptor mode, which enables the advanced raptor controls discussed below.

* loop (0/1), loopsize (0-16): This engages Raptor's built-in looper which repeats the last few bars of output from the arpeggiator. The loopsize control specifies the number of bars to loop. If input runs short then the looper will use what it has, but it needs at least one complete bar to commence loop playback. You can toggle the loop control at any time and it will switch between loop playback and arpeggiator output immediately, which comes in handy if you need to get your hands free to record a generated pattern or play along on a different track. In the Pd version, the current loop can also be saved to disk as text and reloaded at a later time. (In the Ardour version, you can record the loop to a MIDI track instead.)

## 3.3. Arpeggiator Controls

These controls are always in effect (unless the looper is active), no matter whether raptor mode is on or off. The modulation controls (velmod, gatemod, pmod) vary the corresponding parameters according to normalized pulse weights (i.e., indispensabilities) of the current step. These values can also be negative which reverses polarity. E.g., velmod = 1 means that the note velocity varies from minvel to maxvel, *increasing* with the pulse weight, whereas velmod = -1 varies the velocities from maxvel to minvel, *decreasing* with the weight. If velmod = 0, then the velocity remains constant at the maxvel value. The other modulation controls work in an analogous fashion.

* tempo (0-240): The tempo measured in quarter beats per minute (bpm). (Pd only. In Ardour, the tempo is taken from the timeline.) See Section 3.5 below for details.

* meter (1-16): This is a pair of numbers which specifies the desired time signature. The first value (the numerator) specifies the number of beats in a bar. The second number (the denominator) specifies the base pulse a.k.a. the unit of the beat. Thus, e.g., 4 4 denotes common time, a.k.a. 4/4. The base pulse can be further divided into tuplets with the division parameter, see below. (Pd only. In Ardour, the meter is taken from the timeline.)

* division (1-7): Number of subdivisions of the base pulse. E.g., in a 4/4 meter a division value of 2 gives you duplets (eighth notes), 3 gives you (eighth) triplets, etc. The resulting number of steps you get is the numerator of the meter times the number of subdivisions.

* up, down (-2 - +2): Range of octaves up and down. Both values can also be negative. Typically, you'd use a positive (or zero) value for the up, and a negative (or zero) value for the down control. But you also might want to use positive or negative values for both, if you need to transpose the octave range up or down, respectively.

* mode (0-5): Sets the pattern mode. The collection is a bit idiosyncratic, but should nevertheless be familiar and cover most use cases: 0 = random, 1 = up, 2 = down, 3 = up-down, 4 = down-up, 5 = outside-in (alternate between low and high notes). Default is 1 = up.

* minvel, maxvel (0-127), velmod (-1 - +1): Sets minimum and maximum velocity. The actual velocity varies with the pulse weight, by an amount given by the velmod control.

* gain (0-1): This control ties in with the *velocity tracker*, a function which calculates a kind of envelope from the velocities of the notes that you play and adjusts the velocities of output notes generated by the arpeggiator accordingly. The gain value actually controls the mix between preset velocity values (minvel, maxvel) and the calculated envelope, ranging from 0 (all envelope) to 1 (all preset; this is also the default).

* gate (0-1), gatemod (-1 - +1): Sets the gate (length of each note as a fraction of the pulse length). The actual gate value varies with the pulse weight, by an amount given by the gatemod control. An actual gate value of 0 forces legato across rests if notes are being filtered out.

* wmin, wmax (0-1): Deterministic pulse filter. Pulses with a weight outside the given wmin - wmax range will be filtered out. As the deterministic pulse filter uses the pulse weights, it filters notes in a musical fashion by gradually removing less salient pulses while wmin is increased. Conversely, it can also give you syncopation a.k.a. an off-beat rhythm when decreasing wmax.
* pmin, pmax (0-1), pmod (-1 - +1): Probabilistic pulse filter. The given minimum and maximum probability values along with the corresponding modulation determine through a random choice whether a pulse produces any notes. You can disable this by setting pmax = 1 and pmod = 0 (which are the defaults).

## 3.4. Raptor Controls

These controls only affect the arpeggiator if it runs in raptor mode. The modulation controls (hmod, prefmod, smod, nmod) work as described above to vary the corresponding parameter with the pulse weight of the current step. The controls filter and order candidate output notes according to various criteria, which determines which notes are eventually output by the arpeggiator in each step.

* hmin, hmax (0-1), hmod (-1 - +1): This filters candidate notes by comparing their (modulated) average harmonicity with respect to the input chord to the given bounds hmin and hmax. The harmonicity function being used here is a variation of the one in Barlow's Autobusk program, please see my ICMC 2006 paper for details [7, Section 8]. Some interesting harmonicity thresholds are at about 0.21 (the 5th), 0.17 (the 4th), 0.1 (major 2nd and 3rd), and 0.09 (minor 7th and 3rd). To hear the effect, try varying the hmin value while playing a single note.

* pref (-1 - +1), prefmod (-1 - +1): This parameter doesn't affect the candidate notes, but sorts them according to harmonic preference. The arpeggiator will prefer notes with high harmonicity if the value is positive, notes with low harmonicity if it is negative, and simply choose notes at random if it is zero. The preference also varies according to the prefmod modulation control with the current pulse weight. Basically, a high pref value tends towards a tonal, a low one towards an anti-tonal, and zero towards an atonal style, but of course this also depends on the harmonicity range set with the hmin and hmax parameters. This parameter can be very effective if the arpeggiator has enough candidate notes to choose from. E.g., pref = 1 with hmax = 1 will give you a high degree of tonality even if hmin is very low.

* smin, smax (-12 - +12), smod (-1 - +1): Determines the minimum and maximum step size in semitones between consecutive steps. Note that these values can also be negative, allowing for down-steps in an “up” pattern, and the step size can be modulated according to pulse weight. Since Raptor picks notes more or less at random, these parameters give you some control over how large the steps can be. You can also effectively disable this (set smin = 0 and smax = 12), but in this case make sure to also enable uniq mode (see below), otherwise the arpeggiator may just play the same note over and over again.

* nmax (0-10), nmod (-1 - +1): Sets the maximum number of simultaneous notes (modulated according to pulse weight). Thus Raptor, unlike most arpeggiators, can play more than one note at a time, but this only works in raptor mode.

* uniq (0/1): When set to 1, makes sure that notes are not repeated between consecutive steps.

* pitchhi, pitchlo (-36 - +36), pitchtracker (0-3): Set this to extend the octave range (up/down) by the given number of semitones. This gives you finer control over the range of candidate notes in raptor mode. Also, using the pitchtracker control, you can have the arpeggiator follow the (highest and lowest) notes you play and automatically adjust the range of candidate notes accordingly, taking into account the up/down and pitchhi/pitchlo settings. The pitch tracker can be run in four different modes, 0 =

off, 1 = on (follow both high and low notes), 2 = treble (only follow the high notes), 3 = bass (only follow the low notes).

## 3.5. Time and Sync

The frequency at which pulses are being triggered depends on the meter (including division) and, of course, the tempo. One important thing to note here is that by convention, the tempo is *always* specified in *quarter* beats per minute, no matter what the actual base pulse of the meter is. This means that you can switch meters on a whim without having to constantly adjust the tempo when changing the base pulse.

E.g., at a tempo of 120 bpm, quarters run at 500 ms per step, 8ths at double speed (250 ms/step), 16ths at quadruple speed (125 ms/step), etc., no matter what the actual denominator of the meter may be. Raptor also allows you to use non-standard base units such as 6 (= quarter triplets), 12 (= 8th triplets), or even more exotic values such as 10 (= 8th quintuplets) which run at 333 ms, 166 ms, and 200 ms per step at 120 bpm, respectively. This makes it quite easy (or at least possible) to deal with really complex time signatures. However, if you just need tuplets, an easier and more traditional way to get these is to choose a “straight” base meter and adjust the division parameter instead.

In Ardour, both tempo and meter are determined by the timeline. The Pd patch has its own controls for these parameters, and can also sync to an external time source (MIDI clocks). In either case, the division parameter is controlled separately. Also, tempo, meter, and division are *not* preset data; i.e., they are not recorded in presets and thus won't change when recalling presets. (Tempo, meter, and division *are* saved by the looper in the Pd version, however.)

## 3.6. The Looper

Raptor's looper is (by design) quite basic. Its main purpose is to give you a quick way of putting a generated musical phrase on repeat while you have your hands free for playing along, twiddling knobs, or capturing that precious pattern before it vanishes forever. If you need more features, then I'd recommend running Raptor as an Ardour plugin, or the Pd patch alongside a DAW or external groovebox tailored to live usage. In particular, this also gives you the ability to record the *input* to the arpeggiator, which makes it much easier to tweak the results later.

# 4. RAPTOR IN ARDOUR

Ardour makes it very easy to run multiple instances of Raptor in concert, each with their own automation lanes to control Raptor's parameters in real-time. To these ends, you'd typically use one “master” MIDI track to feed note data to a number of different MIDI buses or tracks. (Using tracks rather than buses for the Raptor instances is useful, in particular, if you'd also like to record Raptor's output on each track.) E.g., Fig. 1 shows a little arrangement of this kind.

Here, the topmost “Raptor” track has all the note data being fed into the other tracks. The mixer strip on the left shows the “Organ” track running a Raptor instance into an instrument. The other tracks have been set up in a similar fashion. For details, please check the corresponding Ardour session which can be found in the raptor-freejazz.ardour-session-archive file (link). Note that this file can be opened as is in Ardour’s “Select session file” dialog, if you change the dropdown list at the bottom from “Ardour sessions” to “Session Archives”. The session assumes that you have the General MIDI Synth and setBfree instrument plugins installed, available from x42-plugins.com. There’s also an audio rendering of the session available that you can listen to if you don’t have Ardour installed (link).
<figure><img src="image_81.png" /><figcaption>Figure 1: Raptor Ardour session.</figcaption></figure>

## 5. RAPTOR IN PD

The Pd version includes a main patch with all the requisite controls to operate Raptor in a live performance. This includes transport and looper controls, a beat counter, a quick dial for the first 20 presets, and a panel with on-screen GUI elements for all the parameters, as can be seen in Fig. 2.

<figure><img src="image_82.png" /><figcaption>Figure 2: Raptor in Pd (Purr Data).</figcaption></figure>

While all parameters are accessible with mouse and keyboard using GUI controls in the main patch, Raptor also has built-in support for a number of popular MIDI controllers like the Novation Launchkey and Launchpad, the Novation Launch Control XL, the AKAI APC mini and MIDIMIX, and some of the Hercules DJ Control devices. Other MIDI devices can be used as well, by utilizing the built-in MIDI learn capability. This gives you easier access to at least the most important parameters in a live performance.

Multiple instances of the main patch can be run in concert, in which case one instance acts as the time master to which all other instances sync up, and it is also possible to sync Raptor to an external time source via MIDI clocks, so that it can be used alongside a DAW or stand-alone gear such as the AKAI MPC. Moreover, Raptor instances can be assigned to the decks of a DJ controller in order to control loop playback with the big jog wheels for some fun scrubbing action. Some examples can be found in the Raptor repository [4]. These consist of a main patch which embeds various Raptor subpatches for the parts of the ensemble. It also sets up various parameters in each subpatch such as the preset used in each part, so that the ensemble is ready to go as soon as the main patch has been loaded (cf. Fig. 3).

<figure><img src="image_83.png" /><figcaption>Figure 3: Raptor ensemble.</figcaption></figure>

## 6. CONCLUSIONS AND FUTURE WORK

Like other tools of its kind, Raptor is suitable both for live performances and compositional purposes. You'd probably want to use the Pd patch for the former, and the Ardour version for the latter, although the Pd patch can also be used as a stand-alone composition tool in combination with any DAW or external groovebox.

What makes Raptor stand out from the crowd, in my opinion, is its unusual approach using mathematical models for both harmony and meter as the underlying algorithmic composition engine, in lieu of classical music theory, traditional statistical methods such as Markov chains, or the generative AI techniques which have become popular in recent years. This makes its results quite predictable and very controllable in a live situation, while also providing a lot of musically interesting variation in its output. Other approaches often excel either in predictability or inventiveness, but not both at the same time.

It must be said, however, that arpeggiators available in MIDI keyboards, synthesizers, and software have also progressed quite substantially in the past decade, offering probabilistic elements and a lot of variation, going well beyond merely playing back the given notes across various octaves in some order. Therefore benchmarks and comparisons of Raptor with other existing approaches are definitely in order. To these ends, getting Raptor in the hands of many existing Ardour users will hopefully provide valuable feedback.

One downside of Raptor is its sheer number of parameters which goes way beyond what is on tap with more traditional arpeggiators. This is nothing new, however, complex synthesizer and effect units suffer from the same issue, and thus similar remedies can be applied here to tame the complexity. Having a good preset system is one obvious solution. Raptor already has this, but it could still be further refined to facilitate preset creation and recall in the Pd version. Also, Raptor currently has no built-in macro controls which would let you change multiple parameters at the same time. This could be added to the existing MIDI learn facility of the Pd version in a future release, but in order to also have it work in Ardour, macro controls would need to be added to the arpeggiator core itself.

## 7. ACKNOWLEDGEMENTS

Thanks are due to the anonymous reviewers for their suggestions.
## 8. REFERENCES

[1] Ardour 8.1 — What's new. https://ardour.org/news/8.1.html.

[2] Autobusk, 2000. https://www.musikinformatik.uni-mainz.de/Autobusk/.

[3] agraef/ardour-lua, Nov. 2024. https://github.com/agraef/ardour-lua.

[4] agraef/raptor7, May 2024. https://github.com/agraef/raptor7.

[5] C. Barlow. On the Quantification of Harmony and Metre. In *The Ratio Book*, pages 2–23. Feedback Papers, 1999.

[6] C. Barlow. *On Musiquantics*. Number 51 in *Musikinformatik & Medientechnik*. JGU, Mainz, 2012.

[7] A. Gräf. On Musical Scale Rationalization. In *Proceedings of the 2006 International Computer Music Conference, ICMC 2006, New Orleans, Louisiana, USA, November 6-11, 2006*, pages 91–98. Michigan Publishing, 2006.

[8] M. Vail. *The Synthesizer: A Comprehensive Guide to Understanding, Programming, Playing, and Recording the Ultimate Electronic Music Instrument*. Oxford University Press, USA, 2014.

## A. HARMONICITY AND INDISPENSIBILITY FORMULAS

Raptor uses a slightly modified version of Barlow's original harmony formula, $h(x) = 1/(g(x)+1)$, where for each *rational* interval (frequency ratio) $x = p_1^{a_1} p_2^{a_2} \cdots p_n^{a_n} > 0$ with prime factors $p_1 < p_2 < \cdots < p_n$ and their (positive or negative) multiplicities $a_1, a_2, \ldots, a_n$, $g(x) = |a_1| g(p_1) + |a_2| g(p_2) + \cdots + |a_n| g(p_n)$. That is, we simply sum up all the “prime indigestibilities” $g(p)$, taking into account their absolute multiplicities. The prime indigestibilities in turn are defined using Barlow's formula: $g(p) = 2(p-1)^2/p$. Thus, for instance, $g(5/4) = 2 \times (4^2/5) + 2 \times 2 \times (1^2/2) = 8.4$. (There are alternative formulas for the prime indigestibilities which can be used here, such as $g(p) = p-1$ which is related to Euler's “gradus suavitatis”; please see [7] for details.)

The above formula maps the harmonicity of the unison to 1 (rather than infinity, as in Barlow's original formula $h(x) = 1/g(x)$), which is more convenient to handle computationally. We also “factor out” octaves by just setting $g(2) = 0$, since in our application we are only concerned with tone classes rather than absolute pitches. Now, given some input notes $x_1, \ldots, x_n$, we can assign a harmonicity measure to a “candidate note” $y$ by computing the average harmonicity

$$ \bar{h}(x_1, \ldots, x_n; y) = \sqrt[q]{h(q(x_1, y)) \cdots h(q(x_n, y))}, $$

where $q(x, y)$ denotes the integer ratio assigned to the interval between some notes $x$ and $y$. (Note that we employ a geometric mean here, which appears to work better in this context than the arithmetic mean.) Of course, Raptor works with MIDI notes, to accommodate for that we can simply map the absolute difference between $x$ and $y$ (which is the number of semitones in the interval) to the corresponding integer ratio in a precomputed rationalization of the 12 tone equal temperament scale and then compute the corresponding harmonicity value from that ratio.

As is discussed in [7], the “disharmonicity” function $g$ provides a measure for the complexity of a rational interval, which is empirically related to its “sonance” (consonance or dissonance). In fact, it turns out that if we define $d(x, y) = g(x/y)$ for each rational scale points $x, y$, then the resulting harmonic distance function $d$ is a metric in the mathematical sense. This provides (heuristic) justification for our use of the harmonicity values derived from this metric for musical purposes.

For the pulse weights, we employ Barlow's indispensability formula from [5] as is. This formula is a lot more complicated,

as the goal is to provide a single formula which deals with even the most complex time signatures, and still yields pulse weights which sound convincingly. We assume that the meter is given in a “stratified” form, as a list of prime numbers $q_1, \ldots, q_n$, going from the topmost level $q_1$ of divisions of the bar, to the lowest level $q_n$ (the basic beat). $N = q_1 q_2 \cdots q_n$ is the total number of beats at the lowest level.

We then define the indispensability $I_{q_1, \ldots, q_n}(k)$ of beat $k = 1, \ldots, N$ as

$$ \sum_{i=1}^{n} q_1 \cdots q_{i-1} P_{q_i} \left( \left\lfloor \frac{(k-2) \bmod N}{q_{i+1} \cdots q_n} + 1 \right\rfloor \bmod q_i + 1 \right) $$

Here $P_q(k)$ are the prime indispensabilities ($k = 1, \ldots, q$), $q_1 \cdots q_{i-1}$ the number of beats above, and $q_{i+1} \cdots q_n$ the number of beats below the $i$th level ($i = 1, \ldots, n$). The prime indispensabilities in turn are defined as follows:

* If $q \le 3$, then for $k = 1, \ldots, q$:

$$ P_q(k) = (k - 2) \bmod q $$

* If $q > 3$, then for $k = 1, \ldots, q$:

$$ P_q(k) = \begin{cases} \lfloor q/4 \rfloor, & k = q - 1 \\ I'_{q-1}(k-1), & k = q \\ I'_{q-1}(k), & \text{otherwise} \end{cases} $$

The indispensabilities $I'$ for prime factor $q$ in the latter formula are defined recursively in terms of the indispensabilities for a down-by-1 number of subdivisions $q-1$ as follows:

$$ I'_{q-1}(k) = \begin{cases} I_{PF(q-1)}(k) + 1, & I_{PF(q-1)}(k) \ge \lfloor q/4 \rfloor \\ I_{PF(q-1)}(k), & \text{otherwise} \end{cases} $$

where $PF(q-1)$ are the prime factors of $q-1$ in descending order.

The basic idea is that in order to come up with the indispensabilities for a prime $q$, you take the (general) indispensabilities for $q-1$ beats stratified as $PF(q-1)$, add a new pulse with indispensability $\lfloor q/4 \rfloor$ at the second-last position, and increment all existing indispensabilities at or above this value by 1. This makes sure that all resulting weights are unique again. For instance, a meter with 5 beats will be reduced to 4 beats with weights 3-0-2-1 (as calculated recursively). Then we increment all weights $\ge \lfloor 5/4 \rfloor = 1$, yielding **4-0-3-2**, and insert a new pulse with weight $\lfloor 5/4 \rfloor = 1$ at the second-last position, which yields the final result **4-0-3-1-2**.

I don't think that anyone really knows how Barlow came up with this formula. All I can say is that it seems to work reasonably well, at least with the usual kind of complex meters I tried it with, involving prime subdivisions of up to 11. Note that I have taken apart the original formula from [5] into its constituent parts in order to make it more readable and clarify what is going on there. This formulation is also how the indispensability function is implemented in the Raptor program.
# REAL-TIME MUSICAL INSTRUMENTS RECOGNITION FOR SCENOGRAPHY PURPOSES

Hugues Kadi

Institut National des Sciences Appliquées (INSA)
Lyon, France
huguesk@attssystem.fr

Romain Michon

INRIA, CITI Laboratory, INSA Lyon, GRAME-CNCM
Villeurbanne, France
romain.michon@inria.fr

## ABSTRACT

In recent years, Artificial Intelligence (AI) has been increasingly applied across various domains, including music, where its adoption has grown significantly.

This study leverages modern deep learning architectures, implemented using the Rust programming language, to analyze audio streams. Specifically, the objective is to develop a deep learning model capable of real-time musical instrument recognition, optimized for execution on cost-effective hardware and published as an open-source tool.

This research aligns closely with the Real-Time Music Information Retrieval (RT-MIR) domain.

## 1. INTRODUCTION

### 1.1. Goals

Our primary objective is to provide a preliminary contribution to the field of musical instrument recognition, a specialized area within Music Information Retrieval (MIR). MIR has been extensively explored using AI methodologies, but real-time polyphonic instrument recognition remains a challenging and evolving domain.

This work is focused on investigating the potential of deep learning techniques to develop an initial, open-source solution for this task. To the best of our knowledge, there are currently no open-source solutions or commercial products specifically designed to address this task. While some AI-based tools are capable of separating instruments from audio signals—such as those used for generating STEMS—they typically do not operate under real-time constraints. Moreover, the goal of this work is not to extract the actual waveforms of individual instruments, which is a significantly more complex task.

The integration of this technology into scenography tools is considered a long-term and complex goal, serving as a practical motivation for the research. However, it is important to first clarify the concept of scenography and the potential benefits such a tool could bring to this field. Scenography involves designing and setting up the visual and spatial environment in which a performance takes place, in order to support and enhance the artistic expression. In the context of musical performances, scenography primarily relies on the use of lighting and pyrotechnic effects to complement the music. These effects are typically managed by professionals who either pre-program them to occur at specific moments or control them manually during the performance. The proposed AI tool could enable a new level of automation by triggering these scenographic effects based on the real-time detection of specific instruments. For example, the QLab software [1], which provides an API for cue management, could be used to automatically trigger lighting cues when a piano is detected. This capability would be particularly beneficial in

Mathis Devidal

Institut National des Sciences Appliquées (INSA)
Lyon, France
mathis.devidal@gmail.com

Johan Erbani

Institut National des Sciences Appliquées (INSA)
Lyon, France
johan.leydet@insa-lyon.fr

improvised or semi-structured performances, where traditional pre-programming is less feasible.

The focus of this study is to design a lightweight, easily trainable, and efficient model capable of recognizing multiple musical instruments in real-time from polyphonic audio streams. This solution is intended as a foundation for further improvements and evolution, encouraging contributions from the community to refine and expand its capabilities.

### 1.2. Constraints

Our research project is bounded by two main constraints:

*The real-time constraint* is associated with the necessity of achieving rapid instrument identification to maintain a real-time experience. Initially, a buffer size of 10 ms was chosen based on an interpretation of the latency limit mentioned in a study on Real-Time Percussive Technique Recognition [2]. However, this interpretation proved to be overly simplistic, as it did not account for the loss of information from waveforms with wavelengths exceeding 10ms. Consequently, this buffer size was deemed unsuitable for our purposes, as it failed to yield meaningful results. Instead, a more practical limit of approximately 30ms was adopted, aligning with established guidelines for maintaining a real-time perception [3]. It is important to note that inference time is not the only contributor to latency; the model must be sufficiently lightweight to ensure rapid inference, and additional delays may arise from the response times of scenography tools interfacing with the system [1].

*The hardware constraint* is associated with the deliberate choice of ensuring accessibility, making it unreasonable to require the use of HPC<fnref n="1" /> clusters for training or inference. The experiments were designed to be reproducible on consumer-grade devices. While training may benefit from gaming-level computers equipped with discrete GPUs,<fnref n="2" /> the inference process is intended to be feasible on systems without discrete GPUs, ranging from modern Raspberry Pi devices to older laptops.

### 1.3. State Of The Art

If deep learning has already reached impressive results in the field of image recognition, it is less the case in the field of musical instruments recognition. However, a specific domain emerged in the 2010s, MIR which stands for Music Information Retrieval. This domain even became more specific with RT-MIR which adds the Real-Time component of it.

Several studies have been identified [4] [5], dating back to the early 2010s and using deep-learning techniques, and more specifically CNN. These works are considered preliminary, as

<footnote n="1">¹High Performance Computing</footnote>

<footnote n="2">²Graphics Processing Unit</footnote>
they propose simple yet effective solutions. More recent publications have demonstrated improved inference performance, although the architectural details of the models employed are often not thoroughly documented.

In recent years, alternative approaches have been proposed, including hierarchical methods [6] (Hornbostel-Sachs classification) and various CNN-based techniques [7] [4] [2] [8]. Some studies have also explored the possibility of bypassing feature extraction preprocessing, such as FFT, in favor of raw data processing [9] (deep 34 layers CNN, raw waveform).

Datasets such as NSynth [10] (monophonic), CoCoPops [11] (hum and harm files), and Slakh2100 [12] (polyphonic) were considered. The selection of an appropriate dataset proved to be a challenging aspect of this research, and further details regarding this process are discussed in subsequent sections.

In the field of deep learning, Python is often considered the standard choice, as highlighted in [13]. However, after careful consideration, Rust was chosen for this project §1.4.

## 1.4. Why Rust For Real-Time Audio Processing

Initially, Python was considered as the primary programming language for this project. However, a transition to Rust, along with the Burn ML/DL framework [14], was made for several reasons:

* The performance and safety features of Rust were deemed essential for the development of a real-time application. These capabilities are attributed to Rust's lower-level nature and its robust memory safety mechanisms;

* The Rayon framework, available in Rust, facilitates efficient parallelization of the preprocessing pipeline applied to the datasets;

* Although not a primary objective of this project, Burn's ability to operate in a no_std environment allows for bare-metal execution of a significant portion of the process;

* A limitation of the Rust/Burn combination is its relative immaturity compared to Python-based alternatives. However, this is mitigated by the support of a rapidly growing community.

## 1.5. Paper Organization

This paper is structured as follows: Section 2 describes our dataset selection criteria and the implemented preprocessing pipeline. Section 3 presents our methodology for quantitative performance evaluation. Section 4 details the iterative development process leading to our final model architecture. Finally, we conclude with an analysis of experimental results and propose directions for future research.

# 2. THE DATASET

## 2.1. Choosing Inputs

Identifying a suitable dataset is often a challenging task in AI research. In the context of audio processing, where datasets are less prevalent compared to fields like computer vision, this challenge is even more pronounced. As an initial approach, we used the NSynth dataset [10], which includes 305,979 musical notes, each characterized by unique pitch, timbre, and envelope. These notes are stored as 4-second-long WAV files. To augment the dataset, we divided each sample into smaller segments of 10ms. This segmentation aimed to increase the effective number of training samples and enhance the model's ability to generalize, as the model was initially designed to perform real-time inference on 10ms audio chunks.

Subsequently, we transitioned to the Slakh dataset [12], which offers a more complex and comprehensive collection of audio data. The Slakh dataset consists of approximately 145

hours of music stored in FLAC format. Each track includes both the full mix and individual stems corresponding to the separate instruments played. This structure enabled us to detect the presence of specific instruments in the mix by analyzing the stems in parallel. If a stem contained non-zero waveform data, the corresponding instrument was deemed present in the audio segment. Initially, we applied the same 10ms chunking strategy as with NSynth. However, the primary advantage of the Slakh dataset lies in its stem separation, which facilitated the development of a robust preprocessing pipeline.

## 2.2. Our Preprocessing Pipeline

<figure><img src="image_84.png" /><figcaption>Figure 1: Preprocessing pipeline. Looping through all tracks' directory then cleaning data and applying FFT.</figcaption></figure>

The preprocessing pipeline is a significant component of this research, as it evolved substantially throughout the project. While the initial objective was to extract relevant features from the dataset, it became increasingly evident that additional steps were necessary. Specifically, a considerable portion of the preprocessing effort was dedicated to addressing dataset imbalance.

The following section focuses on the preprocessing pipeline applied to the Slakh dataset, which serves as the primary dataset for this study. This pipeline builds upon and extends the simpler approach initially developed for the NSynth dataset. A visual representation of the complete pipeline is provided in Fig. 1.

## 2.3. Reduce The Dataset

As previously mentioned, one of the primary objectives of this research was to ensure that the model could be trained on consumer-grade hardware. While the hardware used during this study included a professional workstation equipped with an A3000 GPU and an 11th generation Intel i7 CPU with 16 cores, it is important to note that even such machines were incapable of processing the entire dataset. The Slakh dataset comprises approximately 145 hours of music, making it impractical to train on the full dataset due to the excessive computational time required.
To tackle this challenge, we devised a strategy to reduce the over-all size of the input dataset while maintaining sufficient diversity for the model to generalize effectively. This was achieved by introducing a set of hyperparameters specifically designed for dataset reduction:

*   **CHUNK JUMP:** This parameter, expressed in seconds, defines the temporal interval between consecutive audio chunks selected for training.

*   **TRACKS_LIMIT:** This parameter specifies the number of tracks to be included in the training process.

*   **TRACKS_START:** This parameter indicates the duration, in seconds, to be skipped at the beginning of each track before selecting audio segments for training.

*   **TRACKS_LENGTH:** This parameter determines the duration, in seconds, of the audio segment to be used for training after the initial skip defined by TRACKS_START.

These hyperparameters were instrumental in reducing the dataset size while maintaining a diverse set of inputs, thereby enabling the model to achieve a balance between computational feasibility and generalization capability.

## 2.4. Extract Features

Feature extraction was primarily left to the model, which is common in deep learning. However, after conducting several experiments, two key preprocessing transformations were applied to the input data:

*   The audio signal was downsampled by a factor of 8. This step effectively reduced the dimensionality of the input data while retaining sufficient auditory information for instrument recognition. The downsampling factor was empirically determined to ensure minimal perceptual loss of instrument characteristics by ear. Preliminary tests indicated that this reduction had negligible impact on the model's performance.

*   A Fast Fourier Transform (FFT) was applied to transform the audio signal from the time domain to the frequency domain. This transformation enhances the model's ability to identify patterns in the spectral representation, which are more distinct and informative for the task of instrument recognition.

## 2.5. The Balancing Logic

The final component of our preprocessing pipeline is the dataset balancing mechanism. Ensuring a balanced dataset was critical to mitigate overfitting and to enhance the model's ability to generalize to unseen inputs. Initial experiments revealed significant overfitting issues, particularly for instruments that were overrepresented in the dataset.

To address this issue, we implemented a straightforward yet effective balancing strategy. Consider a scenario with $I = 5$ instruments: $X^1, X^2, X^3, X^4$, and $X^5$. The dataset is processed in $I$ iterations, with each iteration introducing a small variation to ensure that the same tracks are analyzed but not identical audio segments. During the first iteration, items containing only one instrument are generated. In the second iteration, items containing two instruments are created, and so forth, until the $I$-th iteration, where items with all $I$ instruments are included.

For each iteration, instruments not selected for analysis are treated as noise and removed. The remaining instruments are shuffled, and only the first instrument is retained in the first iteration, the first two instruments in the second iteration, and so on. Additionally, counters are maintained to track the frequency of each instrument across the entire output dataset. Instruments are excluded from further inclusion if their frequency exceeds

the average frequency of all instruments by 50 or more, thereby preventing overrepresentation.

This iterative and controlled approach ensures a balanced representation of instruments in the dataset. A visual representation of this balancing strategy is provided in Fig. 2.

<figure><img src="image_85.png" /><figcaption>Figure 2: Balancing strategy. Included in preprocessing. Basically going through the dataset multiple times without reading the same chunks.</figcaption></figure>

# 3. MEASURING PERFORMANCE

In this section, we present our methodology for evaluating our model's performance. Multiple complementary metrics were developed and used throughout our research to ensure robust assessment of the model's capabilities. We explore both traditional evaluation approaches and novel metrics specifically tailored to address the unique challenges of multi-label instrument recognition in real-time audio processing.

## 3.1. A Naive Approach

An initial approach involved consulting the Burn documentation, which guided us toward employing the Hamming score. This metric is commonly used in multi-label classification tasks and is defined mathematically as follows:

$$HammingScore = \frac{1}{N} \sum_{i=1}^{N} \frac{|y_i \cap \hat{y}_i|}{|y_i \cup \hat{y}_i|}$$

Where $N$ is the total number of samples, $|y_i \cap \hat{y}_i|$ is the intersection of the true labels $y_i$ and predicted labels $\hat{y}_i$ for the $i$-th sample and $|y_i \cup \hat{y}_i|$ is the union of the true and predicted labels for the $i$-th sample.

We combine it with the binary cross entropy loss whose formula is:

$$BCE = - \frac{1}{N} \sum_{i=1}^{N} [y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i)]$$

Where $N$ is the total number of samples, $y_i$ the true label for the $i$-th sample, $\hat{y}_i$ is the predicted probability for the $i$-th sample. Explanations about its usage are given in 4.5.

However, the Hamming score exhibits certain limitations, particularly in the context of imbalanced datasets. It tends to penalize rare labels disproportionately and emphasizes the accuracy of individual labels rather than the correctness of the entire prediction vector. These shortcomings occasionally led to misleadingly high scores (exceeding 85%), even when the model's predictions were biased towards the most frequent instrument
in the dataset (e.g., the guitar), rather than providing accurate multi-label predictions.

## 3.2. Our Own Metric: Exact Match Ratio

To address the limitations of the initial metric, we introduced an alternative approach, referred to as the Exact Match Ratio (EMR). This metric is conceptually straightforward and is defined as follows:

$$EMR = \frac{FT}{N}$$

Where N is the total number of samples and FT the number of sample *FullyTrue* (where all labels were guessed correctly).

While this metric provides additional insights, it remains limited in its ability to capture the full performance of the model. Furthermore, unlike the Hamming score, the Exact Match Ratio can yield low values even when the model demonstrates meaningful predictive capabilities.

## 3.3. Our Great Saviour: The TCM

One common performance criterion for a deep learning model is the confusion matrix. Such a tool enables one to:

* identify which instruments are confused with which ones

* extract other performance criterion, like the accuracy, the class-wise recall and precision (and hence F-scores, discussed later)

Confusion matrices are traditionally designed for single-label classification tasks. To extend this concept to multi-label classification, we have implemented the Transport Confusion Matrix (TCM), as proposed by Johan Erbani in his work [15]. The TCM enables the computation of confusion matrices and associated metrics for multi-label classification problems.

The TCM is grounded in the principles of optimal transport theory. While a detailed exploration of its mathematical properties is beyond the scope of this paper, interested readers are encouraged to consult the referenced work for an in-depth analysis. For the purposes of this study, we provide a concise summary of the key definitions and formulations used in the implementation.

Giving the following notations:

* C, the number of instruments

* N, the number of tracks samples used to train our model

* y, a vector in {0, 1}^C giving the present instruments for a track sample

* ŷ, a vector in {0, 1}^C giving the predicted instruments for a track sample

* ||·||_1, the l₁ norm

* min(u, v), a vector having, element-wise, the minimum of u and v

* diag(v), the diagonal matrix whose diagonal is equal to vector v

* ⊗, the outer product

* λ(y, ŷ), a contribution weight

For a single observation, represented by a pair of y and ŷ, we define:

$$m = \min(\frac{y}{||y||_1}, \frac{ŷ}{||ŷ||_1})$$

$$D = \operatorname{diag}(m)$$

$$\bar{D} = \frac{(\frac{y}{||y||_1} - m) \otimes (\frac{ŷ}{||ŷ||_1} - m)}{||\frac{y}{||y||_1} - m||_1}$$

$$\pi^*(y, \hat{y}) = \begin{cases} D & y = \hat{y} \\ D + \bar{D} & y \neq \hat{y} \end{cases}$$

This is true for one observation. Then, a TCM is no more than the sum of the previous $\pi^*(y, \hat{y})$ matrix for all observations. With a superscript n ∈ [0, N] indicating the index of the observation in our train set, we finally have:

$$TCM = \sum_{n=0}^{N} \lambda(y^n, \hat{y}^n) \pi^*(y^n, \hat{y}^n)$$

The contribution weight could allow us to put more importance on the predictions, the targets or having both the same weight. We have decided to give predictions and targets the same weight, so $\lambda(y^n, \hat{y}^n) = 1$. From this TCM, we can extract the accuracy:

$$accuracy = \frac{\sum_{k=1}^{C} TCM_{kk}}{\sum_{i,j=1}^{C} TCM_{ij}}$$

As told before, recall and precision are computed class wise. For a class (instrument) k, we have:

$$recall_k = \frac{TCM_{kk}}{\sum_{j=1}^{C} TCM_{kj}}$$

$$precision_k = \frac{TCM_{kk}}{\sum_{i=1}^{C} TCM_{ik}}$$

$$Fscore_k = 2 \times \frac{recall_k \times precision_k}{recall_k + precision_k}$$

As observed, the F score represents the harmonic mean of recall and precision, computed on a per-class basis. Once the C F scores are calculated, they can be aggregated using various methods. These include macro, weighted, and micro averaging. In this study, macro averaging is employed, leveraging the preprocessing pipeline, which ensures class balance within the dataset. Consequently, the final formulation is as follows:

$$Fscoremacro = \frac{1}{C} \sum_{k=1}^{C} Fscore_k$$

We now have three metrics: the TCM, the accuracy and the macro F score.

# 4. EVOLUTION OF MODEL ARCHITECTURE

## 4.1. Using Time Domain

Initially, we aimed to use the inherent feature extraction capabilities of deep learning by minimizing the amount of preprocessing applied to the input data. Consequently, apart from reducing the dataset size, we provided the model with raw audio waveforms as input for training.

However, this approach quickly proved to be ineffective. While deep learning is capable of autonomously extracting features, the raw waveform data did not provide sufficient relevance under the specific conditions of our experiments, leading to sub-optimal performance.

## 4.2. Using Frequency Domain

The rationale for transitioning from the time domain to the frequency domain is grounded in the following observation. In the frequency domain, represented by a spectrum, it becomes significantly more intuitive for a human observer to distinguish between different instruments. This is due to the visibility of their characteristic frequencies and harmonics. Based on this premise, it was hypothesized that a deep learning model might similarly benefit from the enhanced distinguishability offered by the frequency domain, thereby improving its ability to identify and classify musical instruments.

During our experiments with frequency input data, we opted to use Convolutional Neural Networks (CNNs) due to their established efficiency in processing image data. A spectrogram,
which represents the frequency content of an audio signal over time, can be interpreted as a two-dimensional image. This characteristic makes CNNs particularly well-suited for analyzing such data. To leverage this capability, we initially applied CNNs to colored spectrograms, as they were the most intuitive and accessible representation of the frequency domain available to us at the time.

Subsequently, based on expert recommendations, we explored the use of Mel-Frequency Cepstral Coefficients (MFCCs), which are widely regarded as effective features in traditional machine learning for audio processing tasks. However, the implementation of MFCCs presented several technical challenges, which hindered their integration into our pipeline. Ultimately, we reverted to employing the Fast Fourier Transform (FFT) as the primary feature extraction method. This decision was motivated by the FFT's computational efficiency and its ability to provide sufficiently relevant information about the audio signal, as previously discussed.

### 4.3. Enjoying All Possibilities

Initially, the model was trained using 10ms audio samples, sampled at a frequency of 44.1kHz. However, as previously discussed, this buffer size was found to be suboptimal, as it resulted in the loss of significant low-frequency information, effectively acting as a high-pass filter.

To address this limitation and recover the lost low-frequency content, the criteria for determining the buffer size were revised. First, a downsampling factor of 8 was introduced to reduce the input size, thereby making the data more computationally efficient to process. Additionally, based on expert recommendations, the Fast Fourier Transform (FFT) was configured to use 1024 points.

Given these adjustments, the new buffer size can be calculated as follows:

$$ 
\begin{aligned}
sample\_duration &= FFT_{size} \times downsampling\_factor \times \frac{1}{f} \\
&= 1024 \times 8 \times \frac{1}{44100} \approx 186ms
\end{aligned}
$$

Furthermore, given the symmetric nature of the resulting FFT around zero, only the positive frequency components were retained.

### 4.4. Our Final Model Architecture

The final architecture of our model is illustrated in Fig. 3. Throughout the course of this research, several alternative approaches were explored, including the use of transformers and ResNet architectures. However, we ultimately opted for a more traditional convolutional neural network (CNN) design.

This decision was motivated by both practical and performance considerations. The alternative architectures proved to be significantly more complex to implement and required substantially longer training times. In contrast, the CNN-based approach provided a balance between computational efficiency and predictive performance, yielding satisfactory results in specific scenarios, as detailed in the next section.

### 4.5. About Training Procedure

Throughout all architecture evaluations, we maintained the same training hyperparameters and did not attempt to optimize them for performance.

The optimizer employed was Adam, configured with the following parameters:

* $\beta_1 = 0.9$

* $\beta_2 = 0.999$

<figure><img src="image_86.png" /></figure>

<figure><img src="image_87.png" /><figcaption>Figure 3: Model architecture. Simple 8 layers CNN.</figcaption></figure>

$$ \epsilon = 1.0e^{-5} $$

No weight decay (i.e., regularization) or gradient clipping was applied during training.

Additionally, no learning rate scheduling strategy was implemented; instead, a fixed initial learning rate was provided, and its adaptation was managed internally by the Burn framework.

The loss function used was the binary cross-entropy loss, as previously defined in Section 3.1. This loss was selected due to its widespread use in multi-label classification tasks, where it is beneficial that the loss for each class is computed independently of the others. This property is appropriate for our application, as the presence of one instrument (e.g., drums) does not necessarily imply the presence of another (e.g., piano).

**Note:** In the following sections, particularly in the tables and curves, the term "Loss" refers specifically to the binary cross-entropy loss.

### 4.6. Obtained Results

**Important**: for this part, our parameters were the following:

* `CHUNK JUMP` = 1000

* `TRACKS_LIMIT` = 64

* `TRACKS_START` = 10000

* `TRACKS_LENGTH` = 70000

* `EPOCHS` = 5

* `LEARNING_RATE` = $1.0e^{-5}$

The performance of the model is influenced by the number of instruments included during training. In addition to the tables, we provide loss curves for both the training and validation phases corresponding to each instrument set, generated using Python and Matplotlib. These curves provide insight into the model's capacity to learn and generalize. Other performance metrics are omitted, as they follow similar trends. To enhance readability and reduce noise, the curves were smoothed using a
<table><caption>Table 1: Results for 2 instruments: piano and drums</caption><thead><tr><th>Split</th><th>Metric</th><th>Min</th><th>Epoch</th><th>Max</th><th>Epoch</th></tr></thead><tbody><tr><td rowspan="4">Train</td><td>Loss</td><td>0.299</td><td>3</td><td>NaN</td><td>5</td></tr><tr><td>Hamming</td><td>80.468</td><td>1</td><td>87.840</td><td>5</td></tr><tr><td>EMR (%)</td><td>62.112</td><td>1</td><td>76.445</td><td>5</td></tr><tr><td>Accuracy</td><td>0.807</td><td>1</td><td>0.878</td><td>5</td></tr><tr><td></td><td>F score</td><td>0.800</td><td>1</td><td>0.872</td><td>5</td></tr><tr><td rowspan="4">Valid</td><td>Loss</td><td>0.478</td><td>4</td><td>0.589</td><td>2</td></tr><tr><td>Hamming</td><td>73.863</td><td>2</td><td>81.873</td><td>5</td></tr><tr><td>EMR (%)</td><td>53.376</td><td>2</td><td>66.471</td><td>5</td></tr><tr><td>Accuracy</td><td>0.738</td><td>2</td><td>0.819</td><td>5</td></tr><tr><td></td><td>F score</td><td>0.727</td><td>2</td><td>0.815</td><td>5</td></tr><tr><td rowspan="3">Test</td><td>EMR (%)</td><td>/</td><td>/</td><td>79.340</td><td>/</td></tr><tr><td>Accuracy</td><td>/</td><td>/</td><td>0.893</td><td>/</td></tr><tr><td>F score</td><td>/</td><td>/</td><td>0.892</td><td>/</td></tr></tbody></table>

<table><caption>Table 2: Results for 3 instruments: Piano Guitar Drums</caption><thead><tr><th>Split</th><th>Metric</th><th>Min</th><th>Epoch</th><th>Max</th><th>Epoch</th></tr></thead><tbody><tr><td rowspan="4">Train</td><td>Loss</td><td>0.438</td><td>5</td><td>0.559</td><td>1</td></tr><tr><td>Hamming</td><td>70.430</td><td>1</td><td>78.020</td><td>5</td></tr><tr><td>EMR (%)</td><td>34.209</td><td>1</td><td>45.159</td><td>5</td></tr><tr><td>Accuracy</td><td>0.679</td><td>1</td><td>0.740</td><td>5</td></tr><tr><td></td><td>F score</td><td>0.674</td><td>1</td><td>0.735</td><td>5</td></tr><tr><td rowspan="4">Valid</td><td>Loss</td><td>0.633</td><td>5</td><td>0.690</td><td>1</td></tr><tr><td>Hamming</td><td>49.789</td><td>1</td><td>57.754</td><td>5</td></tr><tr><td>EMR (%)</td><td>11.151</td><td>1</td><td>21.383</td><td>4</td></tr><tr><td>Accuracy</td><td>0.429</td><td>1</td><td>0.525</td><td>4</td></tr><tr><td></td><td>F score</td><td>0.213</td><td>1</td><td>0.434</td><td>5</td></tr><tr><td rowspan="3">Test</td><td>EMR (%)</td><td>/</td><td>/</td><td>7.188</td><td>/</td></tr><tr><td>Accuracy</td><td>/</td><td>/</td><td>0.496</td><td>/</td></tr><tr><td>F score</td><td>/</td><td>/</td><td>0.418</td><td>/</td></tr></tbody></table>

sliding window (mean) with a width corresponding to 3% of the total number of batches.

Initially, the final architecture was evaluated using two distinct instruments: drums and piano. The results, presented in Table 1, indicate relatively strong performance in the test phase. This outcome can be attributed to the distinct characteristics of these two instruments, which likely made them easier for the model to differentiate. The corresponding training loss curve, shown in Fig. 4, indicates effective learning behavior, as evidenced by a downward trend in loss values, although the difference between the highest and lowest values remains relatively small compared to what can be observed for other AIS tasks. Similarly, the validation loss exhibits a general decrease across epochs, suggesting improved generalization over time.

The NaN value observed in the table is a result of an exceptional case encountered within the Rust framework [14] used for this study. Unfortunately, this issue could not be resolved during the course of the research.

<figure><img src="image_88.png" /><figcaption>Figure 4: Loss evolution during training and validation phases for piano and drums</figcaption></figure>

Subsequently, a model was trained to recognize three instruments: piano, guitar, and drums. As shown in Table 2, the results demonstrate a decline in performance compared to the two-instrument training scenario. This reduction in accuracy can be attributed to the increased complexity of distinguishing between three instruments, which poses a greater challenge for the model. Figure 5 reflects this difficulty, as the training loss decreases more slowly and remains higher compared to the two-instrument

case shown in Fig. 4. Additionally, the validation loss does not exhibit a substantial overall decline throughout the training process.

**Note:** We present only the loss evolution for the piano, guitar, drums set, as the trends observed for the piano, bass, drums configuration are similar.

<figure><img src="image_89.png" /><figcaption>Figure 5: Loss evolution during training and validation phases for piano, guitar and drums</figcaption></figure>

It is interesting to compare results between piano, guitar and drums with piano, bass and drums.

During the test split, outside the EMR, both accuracy and F score are higher. This could have been explained by the fact that guitar and piano have closer sound properties than bass and piano are on a FFT, but the confusion matrices below (see Fig. 6 & Fig. 7) show that our model confuses more piano and bass than piano and guitar. The brighter the square is, the more the instruments on the corresponding line and column are confused. Of course, the goal is to have a white diagonal and black squares elsewhere. We acknowledge that this behavior warrants a more in-depth analysis; however, we chose to focus our efforts on evaluating the model's performance with respect to the number of instruments considered.

To further validate the observed trend of decreasing performance as the number of instruments increases, we present the results obtained for a model trained on four instruments: piano, brass, guitar, and drums.

The results demonstrate a noticeable decline in performance across all metrics as the number of instruments increases. This
<table><caption>Table 3: Results for 3 instruments: Piano Bass Drums</caption><thead><tr><th>Split</th><th>Metric</th><th>Min</th><th>Epoch</th><th>Max</th><th>Epoch</th></tr></thead><tbody><tr><td rowspan="4">Train</td><td>Loss</td><td>0.348</td><td>5</td><td>0.521</td><td>1</td></tr><tr><td>Hamming</td><td>73.149</td><td>1</td><td>83.992</td><td>5</td></tr><tr><td>EMR (%)</td><td>37.739</td><td>1</td><td>58.774</td><td>5</td></tr><tr><td>Accuracy</td><td>0.703</td><td>1</td><td>0.812</td><td>5</td></tr><tr><td></td><td>F score</td><td>0.699</td><td>1</td><td>0.809</td><td>5</td></tr><tr><td rowspan="4">Valid</td><td>Loss</td><td>0.635</td><td>4</td><td>0.708</td><td>1</td></tr><tr><td>Hamming</td><td>54.718</td><td>1</td><td>66.156</td><td>4</td></tr><tr><td>EMR (%)</td><td>17.458</td><td>1</td><td>28.535</td><td>4</td></tr><tr><td>Accuracy</td><td>0.495</td><td>1</td><td>0.681</td><td>5</td></tr><tr><td></td><td>F score</td><td>0.369</td><td>1</td><td>0.593</td><td>5</td></tr><tr><td rowspan="3">Test</td><td>EMR (%)</td><td>/</td><td>/</td><td>4.862</td><td>/</td></tr><tr><td>Accuracy</td><td>/</td><td>/</td><td>0.587</td><td>/</td></tr><tr><td>F score</td><td>/</td><td>/</td><td>0.468</td><td>/</td></tr></tbody></table>

<figure><img src="image_90.png" /><figcaption>Figure 6: TCM: piano, bass, drums</figcaption></figure>

<figure><img src="image_91.png" /><figcaption>Figure 7: TCM: piano, guitar, drums</figcaption></figure>

<figure><img src="image_92.png" /><figcaption>Figure 8: Loss evolution during training and validation phases for piano, brass, guitar and drums</figcaption></figure>

observation is supported by Fig. 8 which, despite having a lower total number of training batches compared to previous experiments, shows that the loss does not exhibit a clear decreasing trend during either the training or validation phases.

Additionally, a consistent trend is observed where the performance metrics for the validation and testing splits are lower compared to the training split. This indicates that the model struggles to generalize effectively to unseen data. A potential solution to address this issue would be to increase the number of tracks used during training. However, this approach would necessitate access to higher computational resources, which were not available for this study.

## 4.7. Putting Into Practice

It is important to address the apparent discrepancy between the sample duration of approximately 186 ms and the real-time constraint of 30 ms identified earlier. To reconcile this, a sliding window approach was implemented. This method processes audio samples spanning the most recent 186 ms, effectively creating a continuous stream of overlapping input data.

Inference is performed during a callback function, which is triggered by the audio processing pipeline. This pipeline uses

<table><caption>Table 4: Results for piano, brass, guitar and drums</caption><thead><tr><th>Split</th><th>Metric</th><th>Min</th><th>Epoch</th><th>Max</th><th>Epoch</th></tr></thead><tbody><tr><td rowspan="4">Train</td><td>Loss</td><td>0.544</td><td>5</td><td>0.644</td><td>1</td></tr><tr><td>Hamming</td><td>61.101</td><td>1</td><td>71.689</td><td>5</td></tr><tr><td>EMR (%)</td><td>20.522</td><td>1</td><td>25.504</td><td>5</td></tr><tr><td>Accuracy</td><td>0.573</td><td>1</td><td>0.654</td><td>5</td></tr><tr><td></td><td>F score</td><td>0.567</td><td>1</td><td>0.638</td><td>5</td></tr><tr><td rowspan="4">Valid</td><td>Loss</td><td>0.712</td><td>1</td><td>0.724</td><td>4</td></tr><tr><td>Hamming</td><td>45.708</td><td>2</td><td>49.456</td><td>5</td></tr><tr><td>EMR (%)</td><td>0.0</td><td>1</td><td>3.700</td><td>5</td></tr><tr><td>Accuracy</td><td>0.0</td><td>1</td><td>0.568</td><td>5</td></tr><tr><td></td><td>F score</td><td>0.0</td><td>1</td><td>0.170</td><td>5</td></tr><tr><td rowspan="3">Test</td><td>EMR (%)</td><td>/</td><td>/</td><td>0.665</td><td>/</td></tr><tr><td>Accuracy</td><td>/</td><td>/</td><td>0.365</td><td>/</td></tr><tr><td>F score</td><td>/</td><td>/</td><td>0.134</td><td>/</td></tr></tbody></table>

an external Rust crate, *cpal*, over which we had limited control. Empirical observations revealed that the callback function was invoked at intervals of approximately 42 ± 0.5 ms. Despite the longer sample duration, this implementation provided a seamless real-time experience, as confirmed through subjective evaluation.

## 5. CONCLUSION

As discussed throughout this paper, a promising direction for future work would be to train the model on a larger dataset. Due to computational constraints, a significant portion of the input data was excluded to ensure reasonable training times. Given the model's demonstrated ability to perform passably with a limited number of instruments, achieving medium results on the training dataset but only poor performance on the validation and testing datasets, it is reasonable to hypothesize that training on the entire dataset could substantially improve the model's generalization capabilities. However, this would inevitably lead to a significant increase in training time, which was beyond the scope of this study, as the primary objective was to avoid reliance on high-performance computing clusters.

Another avenue for future exploration would involve training and testing the model using real instrument recordings instead of synthetic audio data. This would provide valuable insights into the practical applicability of the proposed solution in real-world scenarios and could further enhance its usability.

Additionally, with more time and resources, it would be worthwhile to revisit the potential of leveraging temporal data from the audio inputs. Specifically, exploring the use of Recurrent Neural Networks as an alternative to Convolutional Neural Networks could yield interesting results. Furthermore, examining research in the domain of voice recognition, which has faced similar challenges for a longer period, may provide valuable methodologies and insights applicable to this work.
This research represents a preliminary exploration into a cutting-edge domain of artificial intelligence, specifically within the field of deep learning. We successfully developed a straightforward model architecture capable of recognizing a limited set of musical instruments. While the implementation required significant effort, the final solution relies on fundamental techniques:

* The model architecture consists of a simple design featuring eight 1D convolutional layers, complemented by max pooling and dropout mechanisms to enhance performance and generalization.

* The preprocessing pipeline employs the Fast Fourier Transform (FFT) as the primary method for transforming raw audio data into a more suitable representation for the model.

In retrospect, while evaluating the initial objectives of this research, it is evident that the outcomes are partially aligned with the intended goals. Specifically, the objective of enabling model training on consumer-grade hardware was not fully achieved. Although the training process was conducted on such hardware, the resulting performance metrics were only moderate, indicating room for improvement in this aspect.

However, the second objective, focused on real-time inference on resource-constrained devices, was successfully accomplished. The proposed model architecture is lightweight and optimized for efficient inference, enabling real-time performance even on modest hardware. Empirical tests conducted on the training machines, including scenarios where the GPU was used in battery mode (which typically impacts performance), demonstrated the system's ability to deliver real-time inference, as perceived by human users.

The complete source code for this project, along with the materials referenced in this report, is available in our public Git repository at https://git.attssystem.fr/attssystem/rrtir. The repository is distributed under the open-source MIT License, ensuring accessibility and encouraging further contributions from the research community.

## 6. REFERENCES

[1] Figure 53, "QLab," Available at: https://qlab.app/, 2024, Visited on 2024-12-05.

[2] Andrea Martelloni, Andrew P. McPherson, and Mathieu Barthet, "Real-time percussive technique recognition and embedding learning for the acoustic guitar," in Proceedings of the 24th International Society for Music Information Retrieval Conference, Milan, Italy, 2023, pp. 121–128, ISMIR, Available at: https://doi.org/10.5281/zenodo.10265236, Visited on 2024-11-18.

[3] Stefani Domenico and Luca Turchet, "On the challenges of embedded real-time music information retrieval," 2022, number 17 in Proceedings of the 25th International Conference on Digital Audio Effects (DAFx20in22), Available at: https://www.dafx.de/paper-archive/2022/papers/DAFx20in22_paper_17.pdf, Visited on 2024-11-15.

[4] Arun Solanki and Sachin Pandey, "Music instrument recognition using deep convolutional neural networks," International Journal of Information Technology, vol. 14, no. 3, pp. 1659–1668, 2022, Available at: https://link.springer.com/10.1007/s41870-019-00285-y, Visited on 2024-11-18.

[5] Maciej Blaszke and Bozena Kostek, "Musical instrument identification using deep learning approach," Sensors, vol. 22, pp. 3033, 2022, Available at: https://www.researchgate.net/publication/360046712_Musical_Instrument_Identification_Using_Deep_Learning_Approach, Visited on 2024-11-18.

[6] Dylan Sechet, Francesca Bugiotti, Matthieu Kowalski, Edouard d'Hérouville, and Filip Langiewicz, "A hierarchical deep learning approach for minority instrument detection," 2024, number 60 in Proceedings of the 27th International Conference on Digital Audio Effects (DAFx24), Available at: https://www.dafx.de/paper-archive/2024/papers/DAFx24_paper_60.pdf, Visited on 2024-11-15.

[7] Kleanthis Avramidis, Agelos Kratimenos, Christos Garoufis, Athanasia Zlatintsi, and Petros Maragos, "Deep convolutional and recurrent networks for polyphonic instrument classification from monophonic raw audio waveforms," IEEE, 2021, ISSN: 2379-190X, Available at: https://ieeexplore.ieee.org/document/9413479, Visited on 2024-11-18.

[8] Jongpil Lee, Taejun Kim, Jiyoung Park, and Juhan Nam, "Raw waveform-based audio classification using sample-level CNN architectures," no. arXiv:1712.00866, 2017, Available at: http://arxiv.org/abs/1712.00866, Visited on 2024-11-18.

[9] Wei Dai, Chia Dai, Shuhui Qu, Juncheng Li, and Samarjit Das, "Very deep convolutional neural networks for raw waveforms," no. arXiv:1610.00087, 2016, Available at: http://arxiv.org/abs/1610.00087, Visited on 2024-11-18.

[10] Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, and Mohammad Norouzi, "Neural audio synthesis of musical notes with WaveNet autoencoders," 2017, Available at: http://arxiv.org/abs/1704.01279, Visited on 2024-11-03.

[11] Claire Arthur and Nathaniel Condit-Schultz, "The coordinated corpus of popular musics (CoCoPops): A metacorpus of melodic and harmonic transcriptions," in Proceedings of the 24th International Society for Music Information Retrieval Conference, Milan, Italy, 2023, pp. 239–246, ISMIR, Available at: https://doi.org/10.5281/zenodo.10265267, Visited on 2024-11-18.

[12] Ethan Manilow, Gordon Wichern, Prem Seetharaman, and Jonathan Le Roux, "Cutting music source separation some (Slakh): A dataset to study the impact of training data quality and quantity," in Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WAS-PAA). IEEE, 2019, Available at: http://www.slakh.com/, Visited on 2024-11-18.

[13] Stefani Domenico, Luca Turchet, and Simone Peroni, "A comparison of deep learning inference engines for embedded real-time audio classification," 2022, number 16 in Proceedings of the 25th International Conference on Digital Audio Effects (DAFx20in22), Available at: https://dafx.de/paper-archive/2022/papers/DAFx20in22_paper_16.pdf, Visited on 2024-11-15.

[14] Tracel Inc., "Burn," Available at: https://burn.dev/, Visited on 2024-11-18.

[15] Johan Erbani, Pierre-Édouard Portier, Elöd Egyed-Zsigmond, and Diana Nurbakova, "Confusion Matrices: A Unified Theory," IEEE Access, pp. 1–1, 2024, Available at: https://ieeexplore.ieee.org/document/10769075/?arnumber=10769075, Visited on 2024-12-06.
LAC-25 Paper Session 7
# SPATIAL ROOM IMPULSE RESPONSE EXTRAPOLATION USING THE IMAGE SOURCE METHOD

Zhenxian Li

INSA Lyon, LVA, UR677
Villeurbanne, France

zhenxian.li@insa-lyon.fr

Thomas McKenzie

Acoustics and Audio Group
University of Edinburgh, UK

thomas.mckenzie@ed.ac.uk

## ABSTRACT

This paper presents a method for the spatial extrapolation of a single higher-order Ambisonic spatial room impulse response (SRIR) measurement to any desired source—receiver position in the room. An assumption is made that in an augmented / virtual reality scenario, the user will likely be wearing technology that is able to scan and create a model of the room. The method of extrapolation therefore uses the expected geometry and source—receiver positions with the image source method, to estimate the direction, time and amplitude of the direct sound and early reflections. The measured SRIR is then processed to rotate, delay and apply a gain to each image source to extrapolate to the target position. In a numerical comparison, the proposed method is shown to effectively extrapolate direct sound and some early reflections, though the early reflection extrapolation is less accurate and more sensitive to any differences between the real geometry and the one used in the image source model.

## 1. INTRODUCTION

Recent advancements in virtual reality (VR) and augmented reality (AR) have underscored the importance of spatial audio in enhancing realism and immersion. Spatial audio is pivotal for creating a compelling virtual environment, provides realistic sound cues that enhance presence [1], and is crucial for applications requiring real-time, low-complexity audio delivery such as video games and virtual meetings.

A room's reverberation, which can be measured as a room impulse response (RIR) comprises of direct sound, early reflections, and diffuse reverberation. Among these, the direct sound and early reflections play essential roles in auditory localisation. The direct sound, defined as the acoustic component traveling straight from the sound source to the receiver without significant environmental reflections, carries primary cues regarding source direction, distance, and identity. It is fundamental in creating the initial perception of spatial orientation and auditory presence. Early reflections, arriving shortly after the direct sound through reflections from objects and surfaces such as walls, floor and ceiling. Early reflections are important for speech intelligibility by effectively increasing the signal-to-noise ratio [2], and allow for the inference of the geometry and size of the space [3]. Accurate reproduction of the direct sound and early reflections is thus critical in immersive spatial audio rendering.

Efficient spatial audio reproduction that aligns with real environments requires concise and computationally efficient rendering [4]. Parameterized rendering addresses this by encoding the acoustic field offline into a model that can be decoded in real-time, allowing for flexible adjustments to various acoustic scenarios.

Spatial audio reproduction often involves spatial room impulse responses (SRIRs) to capture the acoustic characteristics of an environment, which can then be used to synthesise binaural audio by convolving signals with these SRIRs [5] and rendering the output over headphones. However, this technique requires extensive measurements, and is therefore likely not suitable to wider applications with unknown rooms.

Recent research on spatial audio reverberation rendering using minimal measurements has been published, such as the Six Degrees of Freedom Parameterised Spatial Audio (6DoF-RIR) by Arend et al. [6], which use a parameterised model based on a single monophonic RIR for efficient real-time synthesis with 6DoF head tracking. Alternatively, Tsunokuni et al. showed it is possible to extrapolate the early part of RIRs in a small room using sparse equivalent sources and image source method [7], though this also used a monophonic RIR.

Building on these approaches, using an SRIR instead of a monophonic RIR may offer advantages, as the directional information of the direct sound and early reflections is available and can be taken from the measurement. This may lead to improved rendering accuracy and a more immersive experience, as the original spatial data does not need to be inferred or extrapolated from a single-channel source.

This study presents a method for spatial extrapolation of a single higher-order Ambisonic SRIR within a room to any desired source—receiver position. The image source method is used to calculate the direction, distance and time of arrival of the direct sound and early reflections, with which the SRIR measurement is processed accordingly to a new source—receiver position.

The paper is organised as follows. Section 2 presents the methodology for SRIR extrapolation, detailing the image source model and the extrapolation processes. Section 3 then presents a numerical evaluation of the method using time-domain, frequency domain and spatial domain analysis. The results of the evaluation are discussed in Section 4, identifying the strengths and weaknesses of the current methodology. Finally, the paper is concluded in Section 5 along with future work.

## 2. METHODOLOGY

This section describes the methodology of this study, including the image source model to predict information about the direct sound and early reflections and the processing of the SRIR measurement from the original position to a target position.

### 2.1. SRIR Extraction and Spatial Verification

The initial step involves the extraction of SRIRs from a dataset provided in SOFA (Spatially Oriented Format for Acoustics) format. The dataset is in Ambisonics format and is used for its capability to create immersive audio experiences by capturing and reproducing sound in a full-sphere surround setup, utilizing spherical harmonics for detailed audio encoding in three dimensions [8]. This method supports various playback configurations, including multi-speaker setups and binaural decoding for headphones, providing a comprehensive 3D audio experience [9].

The positions of sources and receivers are obtained, and their spatial coordinates are explicitly defined. Subsequently, a three-dimensional geometric representation of the environment is generated to visually verify the spatial relationship between source and receiver positions, ensuring data integrity and spatial accuracy.
<figure><img src="image_93.png" /><figcaption>Figure 1: Illustration of the source (S) and receiver (R) positions used in the evaluation. Microphones and loudspeakers are oriented facing north and south, respectively, according to the illustration orientation. Original S-R pair (to be extrapolated) shown by blue arrow; target S-R pair shown by red arrow.</figcaption></figure>

## 2.2. Diffuse Field Assumption and Subspace Decomposition

In this implementation, SRIR extrapolation is performed by separately processing the direct sound and early reflections, while assuming the diffuse late reverberation to be spatially invariant within a room—at least from a perceptual standpoint. To aid in this, the SRIR measurement is processed using direct and residual subspace decomposition<fnref n="1" />, following the approach by Depisch et al. [10]. A generalised singular value decomposition (GSVD) is performed to effectively isolate the direct sound and salient early reflections from the diffuse reverberation, which is treated as the residual. This decomposition leverages the fact that direct sound and early reflections occupy a limited subspace within the full signal space captured by the microphone array, thus enabling their extraction through low-rank approximations.

Let $h_{orig}(t)$ denote the measured higher-order Ambisonic SRIR at the original position. It is first decomposed into a salient component (comprising direct sound and early reflections) and a diffuse component (late reverberation):

$$h_{\text{orig}}(t) \xrightarrow{\text{GSVD}} \begin{cases} h_{\text{sal}}(t), & (\text{direct + early reflections}) \\ h_{\text{diff}}(t), & (\text{diffuse reverberation}) \end{cases} \quad (1)$$

Only $h_{sal}(t)$ will be manipulated in the spatial extrapolation process; $h_{diff}(t)$ is kept unchanged under the assumption that late reverberation is approximately position-invariant within the same room [11]. This assumption would not apply in rooms with more complex geometry, however, such as L shaped rooms [12] or between coupled rooms [13]. While the separation of salient and diffuse parts of the SRIR is not an entirely necessary step in the extrapolation method, it means that when processing arrivals, the residual is untouched and theoretically, only the part of the SRIR in the direction of interest is processed.

## 2.3. Image-Source Model for Arrival Time and Direction

The Image-Source Model (ISM) is used to simulate the direct sound and early reflections for a known room’s geometry. This is done for both the original and target source and receiver positions. The ISM is used to calculate the coordinates, propagation delays, distances, and arrival directions for the direct sound and early reflections at both original and target positions. [14]

<figure><img src="image_94.png" /></figure>

<figure><img src="image_95.png" /><figcaption>Figure 2: Two-dimensional plot of the calculated 2nd order image sources (black stars) with source (green triangle) and receiver (orange circle).</figcaption></figure>

To use the ISM derived data in the extrapolation, it is essential that the measured data and theoretical predictions are aligned in terms of spatial direction and arrival timing [14]. In practice, the measured SRIR might have been truncated to begin exactly at the theoretical zero time, necessitating temporal alignment adjustments (counter-delay), and it may be that the coordinate systems between the measurement and the ISM are not the same. Therefore, a series of alignment checks are performed at this stage of the method.

Given the known room geometry, virtual images of the sound source are created by mirroring it across each boundary (e.g., walls, floor, ceiling) according to the reflection order of interest. Let $x_S^{(n)}$ denote the position of the n-th image source, and $x_R^{(\text{orig})}$, $x_R^{(\text{tar})}$ be the coordinates of the original and target receiver positions, respectively.

**Time-of-Arrival** For each image source $n$, the corresponding propagation distances and hence time-of-arrival (ToA) are obtained. Specifically,

$$\tau_n^{(\text{orig})} = \frac{\|\mathbf{x}_R^{(\text{orig})} - \mathbf{x}_S^{(n)}\|}{c}, \quad \tau_n^{(\text{tar})} = \frac{\|\mathbf{x}_R^{(\text{tar})} - \mathbf{x}_S^{(n)}\|}{c}, \quad (2)$$

where $c$ is the speed of sound in air, set in this study as 343 m/s. These form the direct and early-reflection arrival times at the

<footnote n="1">¹https://github.com/thomasdeppisch/SRIR-Subspace-Decomposition</footnote>
original and target positions, respectively.

**Arrival Directions** Once the positions of each image source have been determined, the direction-of-arrival (DoA) for each reflection can be computed via

$$ \Omega_n^{(\text{orig})} = \text{dir}(\mathbf{x}_R^{(\text{orig})}, \mathbf{x}_S^{(n)}), \quad \Omega_n^{(\text{tar})} = \text{dir}(\mathbf{x}_R^{(\text{tar})}, \mathbf{x}_S^{(n)}), \quad (3) $$

where $\text{dir}(\cdot)$ maps the coordinates of the receiver and the (image) source to spherical angles $(\varphi, \theta)$ (azimuth and elevation), and so equation (3) can be rewritten as

$$ \Omega_n^{(\text{orig})} = (\phi_n^{(\text{orig})}, \theta_n^{(\text{orig})}), \quad \Omega_n^{(\text{tar})} = (\phi_n^{(\text{tar})}, \theta_n^{(\text{tar})}). \quad (4) $$

**Propagation Distances and Sorting** Similarly, the propagation distances for reflection $n$ at each position are

$$ r_n^{(\text{orig})} = \|\mathbf{x}_R^{(\text{orig})} - \mathbf{x}_S^{(n)}\|, \quad r_n^{(\text{tar})} = \|\mathbf{x}_R^{(\text{tar})} - \mathbf{x}_S^{(n)}\|. \quad (5) $$

The image sources are sorted based on the mean of $r_n^{(\text{orig})}$ and $r_n^{(\text{tar})}$ (or equivalently $\tau_n^{(\text{orig})}$ and $\tau_n^{(\text{tar})}$), prioritising arrivals that are shorter in distance (and thus earlier in time). The rationale for sorting in this way is so that the most perceptually relevant arrivals are processed first, as arrivals from the SRIR measurement are removed to be processed (see Section 2.4).

## 2.4. SRIR Extrapolation Using Arrival Manipulation

Each identified arrival — comprising either the direct sound or an early reflection — is individually processed through a sequence of operations. From $h_{\text{sal}}(t)$, each significant early arrival $n$ is isolated by windowing around $\tau_n^{(\text{orig})}$:

$$ k_n(t) = w(t - \tau_n^{(\text{orig})}) \cdot h_{\text{sal}}(t), \quad (6) $$

where $w(\cdot)$ is a short temporal window (set in this study as 4 ms for the direct sound and 1 ms for an early reflection) around each arrival that is applied to isolate it from the original SRIR at the calculated arrival time of the original position. The isolated arrival is then removed, leaving the rest of the salient SRIR. The temporal window is characterized by linear ramps with ascending and descending phases of 5 and 10 samples, respectively.

The residual SRIR, denoted as $h_{\text{res}}(t)$, is obtained by removing all isolated early arrivals $k_n(t)$ from the original salient SRIR $h_{\text{sal}}(t)$:

$$ h_{\text{res}}(t) = h_{\text{sal}}(t) - \sum_{n=1}^{N_{\text{early}}} k_n(t). \quad (7) $$

This operation effectively eliminates the contributions of all early arrivals calculated by Image source model, yielding a signal that contains the remaining components of the $h_{\text{sal}}(t)$.

For each extracted arrival $r_n(t)$, three adjustments are computed to map from the original to the target positions:

### 1. Time shift:

$$ \Delta\tau_n = \tau_n^{(\text{tar})} - \tau_n^{(\text{orig})}. \quad (8) $$

### 2. Amplitude scaling (assuming free-field distance law):

$$ g_n = \frac{r_n^{(\text{orig})}}{r_n^{(\text{tar})}}. \quad (9) $$

### 3. HOA-domain rotation to align directions: To rotate the $n$-th reflection from its original direction to match the target direction in the HOA domain, the angle offsets are computed as

$$ \Delta\phi_n = \phi_n^{(\text{tar})} - \phi_n^{(\text{orig})}, \quad \Delta\theta_n = \theta_n^{(\text{tar})} - \theta_n^{(\text{orig})}. \quad (10) $$

$$ \mathbf{k}'_n(t) = R(\Delta\phi_n, \Delta\theta_n)\mathbf{k}_n(t), \quad (11) $$

where $\mathbf{k}_n(t)$ is the vector of HOA channels for the $n$-th arrival, and $R$ is the HOA rotation matrix for Ambisonic order $N$) [8].

Combining these, the transformed excerpt $k'_n(t)$ is inserted back into the SRIR timeline at $\tau_n^{(\text{tar})}$:

$$ k'_n(t) = g_n \left[ R(\Delta\phi_n, \Delta\theta_n) \mathbf{k}_n(t) \right]_{t \to t + \Delta\tau_n}. \quad (12) $$

After processing all major early arrivals $n = 1, \dots, N_{\text{early}}$, they are summed into a single HOA signal to form the new salient component at the target position:

$$ h_{\text{sal}}^{(\text{tar})}(t) = h_{\text{res}}(t) + \sum_{n=1}^{N_{\text{early}}} k'_n(t). \quad (13) $$

Finally, as the diffuse portion remains unchanged:

$$ h_{\text{extrap}}(t) = h_{\text{sal}}^{(\text{tar})}(t) + h_{\text{diff}}(t). \quad (14) $$

Thus, $h_{\text{extrap}}(t)$ serves as the spatially extrapolated SRIR for the target source–receiver configuration.

## 3. EVALUATION

To evaluate the proposed method, a dataset of SRIR measurements was used from the Arni variable acoustics room at Aalto University's Acoustics Lab, encompassing three source positions and seven receiver positions, detailed in Fig. 1. These positions, carefully chosen to mimic realistic performance scenarios, involve a linear array of receivers spaced 1.25 meters apart, with some near the room's perimeter, and sources directed towards a common "front." The variable acoustics 6DoF dataset<fnref n="2" /> (as described in [15]). The measurements used in this study to evaluate the SRRI extrapolation method were those with medium reverberance, with octave-band [500 Hz, 1 kHz, 2 kHz] RT60 values of [0.50, 0.54, 0.68] s, recorded at a background noise level of 20.5 dB SPL(A). This dataset was recorded at 24-bits with a 48 kHz sampling rate.

The proposed method is evaluated by taking an SRIR measured at one position in the room (herein referred to as the original position) and applying the extrapolation process to transform it to another position in the room (herein referred to as the target position). The original and extrapolated SRIR are then compared to a ground truth SRIR measurement made at the target position.

The pair of room acoustics measurements used in the evaluation correspond to the source – receiver (S–R) positions as follows (and illustrated in Fig. 1):

* Original: S2–R4 (blue)

* Target: S3–R2 (red)

These were chosen as a number of factors change from the original to target position: the distance between the source and receiver increases, and also the direction of the source changes.

### 3.1. Time-domain

To evaluate the amplitude and temporal nature of the extrapolation method, Fig. 3 presents a time-domain comparison of the first 50 ms of the original, extrapolated and target SRIRs (omni-directional channel). The main point of observation here is that the direct sound is clearly well extrapolated in both time and level. Secondly, the first (and most major) early reflection has moved in time from the original position, which matches the target position, however it has not necessarily moved to the correct place or level. This is likely due to a mismatch between the ISM

<footnote n="2">²https://doi.org/10.5281/zenodo.5720724</footnote>
<figure><img src="image_96.png" /><figcaption>(a) Original</figcaption></figure>

<figure><img src="image_97.png" /><figcaption>(b) Extrapolated</figcaption></figure>

Figure 3: Time-domain comparison overlaying the target SRIR on the original (top) and extrapolated (bottom) SRIRs, omnidirectional channel. The original and target source–receiver pairs are S2–R4 and S3–R2, respectively (as labelled in Fig. 1).

and the real room measurements. Some of the other reflections have been moved too, and inaccuracies clearly exist in these too.

To get an understanding of the effect of extrapolation on loudness, the root-mean-square (RMS) amplitude of the omnidirectional channel of the SRIRs was calculated. This was done for the entire duration of the SRIR. The RMS amplitude of the original, extrapolated and target SRIRs is -40.5dB, -44.8dB and -45.0dB, respectively. This shows that the method extrapolates the amplitude effectively.

## 3.2. Frequency-domain

To assess the effect of extrapolation on the frequency response of the SRIRs, the original, extrapolated and target SRIRs were first rendered binaurally. This was done using dual-band Ambisonic decoding (mode-matching below 1.8 kHz, max-$r_E$ above), with time-alignment above 1.8 kHz [16] and Ambisonic diffuse-field equalisation [17] pre-processing<fnref n="3" />, and non-individualised Neumann KU100 HRTFs [18].

Fig. 4 presents the frequency responses of the left and right binaurally rendered SRIRs, overlaying the extrapolated response on the original and target responses. Firstly, it is again evident how the extrapolation method matches the level well. Additionally, it is clear that the frequency response above 1 kHz is very well matched for the extrapolated and target responses, for both left and right, with differences largely falling within 1 dB between 1 kHz and 10 kHz. This is a significant change from the original response. The binaural levels are well extrapolated when considering the direction of the source has moved from front-right to front-left. Below 1 kHz however, inaccuracies are evident in the extrapolated response. This could be due to mismatch between the geometry used in the ISM and the real geometry of the measured room, or could be due to the simplifications inherent in the ISM: namely the geometric reflection modeling which neglects wave effects such as mode buildup, resonance and diffraction.

To predict how similar the extrapolated SRIRs will be perceived in timbre, the PBC-2 auditory model [19] was used to predict the colouration between the binaural signals. The PBC-

<figure><img src="image_98.png" /><figcaption>(a) Original</figcaption></figure>

<figure><img src="image_99.png" /><figcaption>(b) Extrapolated</figcaption></figure>

Figure 4: Frequency-domain comparison overlaying the target SRIR on the original (top) and extrapolated (bottom) SRIRs, which have been rendered binaurally. The original and target source–receiver pairs are S2–R4 and S3–R2, respectively (as labelled in Fig. 1).

<figure><img src="image_100.png" /></figure>

Figure 5: Estimated direction-of-arrival of direct sound and early reflections for the original, extrapolated and target SRIRs. The original and target source–receiver pairs are S2–R4 and S3–R2, respectively (as labelled in Fig. 1).

2 model combines signal rectification, high frequency smoothing, equivalent rectangular bandwidth frequency weighting, and signal-dependent weighting of the left and right colouration values to produce a single binaural colouration prediction. The version used in this study is available in the Auditory Modeling Toolbox under the model name *mckenzie2025*. PBC-2 calculations were made with 1/3 octave smoothing applied at all frequencies. The PBC-2 value between the binauralised original and target SRIRs was 91.4, and the PBC-2 between the binauralised extrapolated and target SRIRs was 37.4 (where 100 is highly coloured and 0 is no colouration). This shows that colouration is significantly reduced by the extrapolation method. However, the difference between the extrapolated and target SRIRs is still quite high, and it's highly likely this would be audible.

## 3.3. Direction Of Arrival

To evaluate the effect of extrapolation on the directionality of the direct sound and early reflections, the direction-of-arrival (DoA) was estimated above 3 kHz in two degree resolution for the original, extrapolated and target SRIRs. This was done by steering a 4th-order hyper-cardioid beamformer (sometimes referred to as normalised plane wave decomposition) in all directions on the sphere [20] and calculating the power, which can be used to deduce directional information and the relative level of the direct sound and loudest early reflections.

<footnote n="3">³https://github.com/thomas-mckenzie/binaural_ambisonic_preprocessing</footnote>
<figure><img src="image_101.png" /><figcaption>Figure 6: Normalised power response of the original, extrapolated and target SRIRs showing the seven highest arrivals. The original and target source-receiver pairs are S2-R4 and S3-R2, respectively (as labelled in Fig. 1).</figcaption></figure>

Firstly, the seven locations with the highest power were extracted, which likely estimate the horizontal DoA of the direct sound and first order early reflections. This is presented in Fig. 5. Both the DoA and power of most arrivals appears to be extrapolated fairly effectively, although some (e.g. around 180° and -135° in the target) are poorly matched by the extrapolation.

Secondly, to get a three-dimensional view of the spatial qualities of the extrapolation, Fig. 6 presents the normalised power responses in the same 2° resolution over the sphere. The DoA of the direct sound appears well steered, as was observed in Fig. 5, however in this plot the other arrivals are harder to see, and it appears that some directions are less clear. Again, the arrival around 180° in the target seems to have been poorly steered by the extrapolation method. The reason for this is likely a mismatch between the ISM and measurements, which has meant that the arrival was extracted at the wrong time.

## 4. DISCUSSION

The results show that the proposed method for spatial extrapolation of a single SRIR measurement using ISM is reasonably effective as a simple method. The ISM is able to predict the direction, time and amplitude of the direct sound and early reflections for both the original and target source-receiver positions, which are used to augment an SRIR measured at the original source-receiver position. Comparing the original SRIR

and extrapolated version of the original SRIR to the target SRIR, the RMS amplitude is significantly improved, and the predicted colouration is significantly reduced.

The direct sound appears to be the most accurately extrapolated, in all three measures of direction, time and amplitude. The first early reflections are somewhat accurately extrapolated, though the inaccuracies increase as time goes on.

A likely cause of much of the errors in the method is where the image source and the measurement are mismatched. A number of options exist for what causes this mismatch. The simplest of these is that the geometry, source positions and receiver positions were not accurately measured. Mismatch in these would mean that the image sources are in the wrong positions, leading to errors that compound as the order of reflection increases. Similarly, the image source method used here assumes that walls are perfectly flat and perfectly reflective. In reality, both the geometry and materials making up the room are more complex: there are objects in the room and the walls have texture. Finally, the current method does not account for source directivity. This will likely affect the high frequency timbre the most. Whilst the high frequencies were shown to be relatively well extrapolated in the example in this study, the inaccuracies would likely be more noticeable in cases where one of the receiver positions is behind or strongly to the side of the source position.

Future work will look at how the ISM can be matched closer to the real room. It may be possible to alter the geometry and source-receiver positions through a comparison of the ISM simulation and the measurement at the original source-receiver position.

Finally, for each arrival, the proposed method extracts the entire signal at the desired time window. The arrivals instead could be extracted using a beamforming approach, in order to only extract in the direction of the original arrival. This would likely improve results where two arrivals occur at a similar time.

Though results are only presented here for a single extrapolation, multiple source receiver positions were tried and it worked similarly well for all. Future evaluation will report results for all as well as look at different datasets, and also how the method works perceptually using listening tests. The present study is limited to static source-listener pairs; however, in AR/VR scenarios continuous motion could cause the current distance-based resorting to swap image-source indices and thus generate audible artefacts. Future work will therefore extend the algorithm with a *reflection-tracking* layer: (i) each image source will be assigned a persistent identifier derived from its mirror vector ($m_x, m_y, m_z$), allowing its parameters to evolve smoothly, and (ii) a soft-selection gate will fade paths in and out according to energy or delay thresholds. Collectively, these two measures are expected to ensure artefact-free extrapolation along moving listener or source trajectories.

## 5. CONCLUSIONS

This paper has presented a method for higher-order Ambisonic spatial room impulse response (SRIR) extrapolation from a single measurement, where the room geometry and source-receiver positions are known. The method uses the room geometry information along with the image source method to simulate the direction-of-arrival, time-of-arrival, and distance of the direct sound and early reflections of the original and target positions, before calculating the difference. The subspace decomposition method is used to separate out the salient part of the spatial room impulse response measurement from the diffuse part. The direct sound and early reflections are then windowed out from the salient part of the measurement, processed (to the target time-of-arrival, gain and direction-of-arrival), and summed back into the response.

The proposed method has been evaluated numerically by comparing a measurement at an original source-receiver position.
tion and an extrapolated version of the measurement to a separate measurement made at the target source–receiver position. The method has been shown to effectively reproduce the correct direct sound, and to somewhat effectively reproduce some of the first early reflections, though errors exist. Likely, these errors stem from where the image source and the measurement are mismatched.

Future work will look to improve the methodology through tuning of the geometry and source–receiver position parameters used in the image source method by comparing the measurement and image-source simulations. Also, by including source-directivity in the model. The method will also be more thoroughly evaluated numerically, as well as perceptually.

A Matlab implementation of the presented SRIR extrapolation method, *srir_extrapimgSource_subspace_LAC.m*, along with a demonstration script which reproduces the figures presented in this paper, is available to download at https://github.com/ZhenxianLi/srir_extrap_LAC2025_ZL_TM.

## 6. REFERENCES

[1] T. J. MacGillivray, W. Ellis, and S. D. Pye, “The resolution integral: Visual and computational approaches to characterizing ultrasound images”, *Physics in Medicine and Biology*, vol. 55, no. 17, pp. 5067–5088, 2010. DOI: 10.1088/0031-9155/55/17/012. (visited on 04/12/2023).

[2] J. S. Bradley, H. Sato, and M. Picard, “On the importance of early reflections for speech in rooms”, *The Journal of the Acoustical Society of America*, vol. 113, no. 6, pp. 3233–3244, 2003.

[3] D. Khaykin and B. Rafaely, “Acoustic analysis by spherical microphone array processing of room impulse responses”, *J. Acoust. Soc. Am.*, vol. 132, no. 1, pp. 261–270, 2012. DOI: 10.1121/1.4726012.

[4] P. Stitt, E. Hendrickx, J.-C. Messonnier, and B. Katz, “The role of head tracking in binaural rendering”, in *29th Tonmeistertagung, International VDT Convention*, 2016.

[5] B. Rakerd and W. M. Hartmann, “Localization of sound in rooms. v. binaural coherence and human sensitivity to interaural time differences in noise”, *J. Acoust. Soc. Am.*, vol. 128, no. 5, pp. 3052–3063, 2010. DOI: 10.1121/1.3493447. (visited on 08/17/2023).

[6] J. M. Arend, S. V. A. Garí, C. Schissler, F. Klein, and P. W. Robinson, “Six-degrees-of-freedom parametric spatial audio based on one monaural room impulse response”, *Journal of the Audio Engineering Society*, vol. 69, no. 7, pp. 557–575, 2021. DOI: 10.17743/jaes.2021.0009. (visited on 04/26/2023).

[7] I. Tsunokuni, K. Kurokawa, H. Matsuhashi, Y. Ikeda, and N. Osaka, “Spatial extrapolation of early room impulse responses in local area using sparse equivalent sources and image source method”, *Applied Acoustics*, vol. 179, p. 108027, 2021.

[8] F. Zotter and M. Frank, *Ambisonics: A Practical 3D Audio Theory for Recording, Studio Production, Sound Reinforcement, and Virtual Reality* (Springer Topics in Signal Processing). Cham: Springer International Publishing, 2019, vol. 19, ISBN: 978-3-030-17206-0 978-3-030-17207-7. DOI: 10.1007/978-3-030-17207-7. (visited on 06/14/2023).

[9] L. McCormack, A. Politis, T. McKenzie, C. Hold, and V. Pulkki, “Object-based six-degrees-of-freedom rendering of sound scenes captured with multiple ambisonic receivers”, *Journal of the Audio Engineering Society*, vol. 70, no. 5, pp. 355–372, May 11, 2022. DOI: 10.17743/jaes.2022.0010. (visited on 06/27/2023).

[10] T. Deppisch, S. V. A. Garí, P. Calamia, and J. Ahrens, “Direct and residual subspace decomposition of spatial room impulse responses”, *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 31, pp. 927–942, 2023.

[11] C. Kirsch, J. Poppitz, T. Wendt, S. Van De Par, and S. D. Ewert, “Spatial Resolution of Late Reverberation in Virtual Acoustic Environments”, en, *Trends in Hearing*, vol. 25, p. 23 312 165 211 054 924, Jan. 2021. DOI: 10.1177/23312165211054924. (visited on 03/11/2025).

[12] L. Savioja and U. P. Svensson, “Overview of geometrical room acoustic modeling techniques”, en, *The Journal of the Acoustical Society of America*, vol. 138, no. 2, pp. 708–730, Aug. 2015. DOI: 10.1121/1.4926438. (visited on 03/11/2025).

[13] T. McKenzie, S. J. Schlecht, and V. Pulkki, “Auralisation of the transition between coupled rooms”, in *Int. Conf. on Immersive and 3D Audio*, Online: IEEE, Sep. 2021, pp. 1–9. DOI: 10.1109/I3DA48870.2021.9610955.

[14] J. B. Allen and D. A. Berkley, “Image method for efficiently simulating small-room acoustics”, *The Journal of the Acoustical Society of America*, vol. 65, no. 4, pp. 943–950, Apr. 1979, ISSN: 0001-4966. DOI: 10.1121/1.382599. [Online]. Available: https://doi.org/10.1121/1.382599 (visited on 03/16/2025).

[15] T. McKenzie, L. McCormack, and C. Hold, “Dataset of spatial room impulse responses in a variable acoustics room for six degrees-of-freedom rendering and analysis”, in *arXiv preprint*, 2021, pp. 1–3. DOI: 10.48550/arXiv.2111.11882. arXiv: 2111.11882.

[16] M. Zaunschirm, C. Schörkhuber, and R. Höldrich, “Binaural rendering of Ambisonic signals by head-related impulse response time alignment and a diffuseness constraint”, *J. Acoust. Soc. Am.*, vol. 143, no. 6, pp. 3616–3627, 2018. DOI: 10.1121/1.5040489.

[17] T. McKenzie, D. T. Murphy, and G. C. Kearney, “Diffuse-field equalisation of binaural Ambisonic rendering”, *Appl. Sci.*, vol. 8, no. 10, Oct. 2018. DOI: 10.3390/app8101956.

[18] B. Bernschütz, “A spherical far field HRIR / HRTF compilation of the Neumann KU 100”, in *Fortschritte der Akustik – AIA-DAGA 2013*, 2013, pp. 592–595.

[19] T. McKenzie and F. Brinkmann, “Toward an Improved Auditory Model for Predicting Binaural Coloration”, *J. Audio Eng. Soc.*, no. Accepted for publication, 2025. DOI: 10.17743/jaes.2022.0192.

[20] A. Politis, “Microphone array processing for parametric spatial audio techniques”, PhD Thesis, Aalto University, 2016.
# TAPE SIMULATION

Fons Adriaensen

Zita Audio

fons@linuxaudio.org

## ABSTRACT

Before the days of digital audio, music was produced using analog magnetic tape recorders. It is sometimes claimed that this gave the result a quality – usually decribed in rather subjective terms – that today’s all-digital production environment doesn’t provide. So there is a market for analog tape emulator plugins. The *zita-ts1* tape simulator will be released around the time of the conference. This paper introduces some of the technical aspects of analog magnetic tape audio recording, and relates how these influenced the design of the digital simulation algorithm.

## 1. INTRODUCTION

Analog audio tape recording technology was developed in the period just before and during the second world war. The German *Telefunken* company created the first recorders that offered sufficient quality for music production and broadcasting. After the war some of these *Magnetophon* machines were taken to the USA, where the *Ampex Corporation* picked up the technology and further developed it.

By the early 1950s tape recorders had become standard equipment in all professional audio production environments. The first multitrack machines appeared in the 60s, and ten years later all major music recording studios were using 16 or 24 tracks. Some even synchronised two machines to provide even more tracks.

Apart from quality and performance improvements very little happened until digital technology became available in the early 80s. Some digital audio tape machines were developed, but very soon computer based systems recording on hard disk became the standard.

Today there are still some studios that use analog tape, mainly some big ones that can afford to maintain their legacy machines. Very few audio engineers have the necessary knowledge and skills to keep them aligned and in good working order, and as the author found out, even reliable technical documentation can be hard to find.<fnref n="1" /> This shouldn’t be a surprise considering that most of the theory and technology of analog audio tape recording was developed more than 50 years ago.

## 2. ANALOG MAGNETIC TAPE RECORDING

### 2.1. Magnetisation

Ferromagnetic materials consist of microscopically small domains that can change their magnetic orientation when subjected to an external magnetic field. The stronger the external field, the more domains will align with it and the net result is magnetisation of the material. For almost all ferromagnetic materials the relation between the field strength *H* and the resulting magnetisation *M* is not only nonlinear, it also exhibits a memory effect called *hysteresis*.

A relation $y = F(x)$ is nonlinear if the derivative $dy/dx$ depends on $x$. It has hysteresis if $dy/dx$ depends on the sign of $dx$.

Figure 1 shows an example. Assume that originally the material is fully demagnetised, point **A**, and we apply a field *H* proportional to a sine wave starting at zero. As the field strength *H* increases, the magnetisation *M* increases in a non-linear way until it reaches a maximum value at **B**. In this example *H* is also strong enough to drive *M* into saturation. As *H* returns back to zero *M* will follow the top curve to **C**. To reduce *M* to zero at **D** requires a negative *H*. After reaching the maximum negative value, *M* will follow the bottom curve and the cycle repeats.

The magnitude of *M* at **C** is called the *remanence*. Ferromagnetic materials with a high remanence are called *hard*, those with less or no hysteresis *soft*.

<figure><img src="image_102.png" /><figcaption>Figure 1: *Example magnetic hysteresis curve.*</figcaption></figure>

To store binary information the response as shown in Fig.1 is near ideal. All that is needed is to apply a positive or negative field which will result in a remanence with the same polarity. The magnetisation can be driven into saturation without any ill effect. This is how digital tape recorders and 'core memory' as used in early computers work. To record analog data a strictly linear relation between the signal and the magnetisation is required. How this can be done will be discussed in the following sections.

### 2.2. The Recording Process

Magnetic tape consist of a long narrow strip of some plastic material, coated on one side with very fine particles of a 'hard' ferromagnetic material, e.g. iron oxyde. The tape is moved at a constant speed over a number of *magnetic heads*. Professional tape recorders normally have three of these to respectively, erase, record and play back the tape.

Figure 2 shows how a magnetic head is built. The *core* is made of a 'soft' ferromagnetic material. An electric wire coil is wound around it. At one side there is a narrow gap made of a physically hard non-magnetic material, e.g. ceramic. This is where the tape is in contact with the head.

<footnote n="1">¹Much of the information used in this paper was found in a collection of documents [1] from the early years. Most of these are by Jay McKnight who worked first at Ampex and later founded Magnetic Reference Laboratory.</footnote>
<figure><img src="image_103.png" /><figcaption>Figure 2: Magnetic head and tape geometry (not to scale).</figcaption></figure>

When recording, an electric current in the coil will generate a magnetic field *H* concentrated at the gap and this will locally magnetise the tape coating. The playback head senses the magnetisation *M* and an electrical signal proportional to *dM/dt* is generated by the coil.

For the record and playback heads the gap size must be very small. It acts as a rectangular window in the time domain, resulting in a $sin(x)/x$ shaped frequency response. In practice the gap size must not be much larger than half the signal wavelength on tape. To put this in perspective, at a tape speed of 19 cm/s (the lowest one used professionally) a 20 kHz signal has a wavelength of less than 10 micrometer.

## 2.3. HF Bias

As explained in 2.1, the relationship between the magnetisation and the field that creates it is highly nonlinear, and trying to record an analog waveform on magnetic media will result in severe distortion.

It was discovered quite early on that adding a strong high frequency component to the signal to be recorded would remove most of the nonlinearity, with only a high amplitude saturation effect remaining [2]. A typical frequency for this *bias* signal would be around 100 kHz.

The way bias works is sometimes explained in an informal context by pointing out that if the bias amplitude is high enough this forces the signal peaks to fall in the more or less linear regions of the hysteresis curve, and assuming that the average magnetisation will be linear as well. While this explanation appeals to intuition, it is wrong and it doesn't explain what is actually observed.

What happens in reality is a bit more complicated. While the magnetic field at the record head is certainly concentrated at the head gap, it does extend a bit on both sides. How fast it decays as a function of distance depends on the head geometry near the gap and on the thickness of the tape coating.

Now imagine recording a low frequency audio signal that can be assumed to be constant during the time a small fragment of the tape passes near the recording head gap. The top part of Fig.3 shows the magnetic field this fragment is subjected to, with zero on the time axis corresponding to passing the gap. The bottom plot show the resulting magnetisation. In this example the bias frequency is 100 kHz, and the constant signal amplitude is half the bias amplitude. Figure 4 shows the trajectory in the *H*, *M* plane, starting at time zero in Fig.3.

The signal that finally remains on the tape is the value the magnetisation converges to as the tape fragment moves away from the recording head and the field strength decreases to zero, at around 150 microseconds in this example. Note that this is not at all the average of *M* when near the gap.

Note that the high values of *M* at the gap completely obliterate whatever came before – only the part after passing the head

<figure><img src="image_104.png" /><figcaption>Figure 3: Simulated magnetic field (top) and magnetisation (bottom) as a tape fragment moves past the recording head gap. The magnetisation converges to its final value around 150 µs after passing the recording head gap.</figcaption></figure>

<figure><img src="image_105.png" /><figcaption>Figure 4: Trajectory in the *H*, *M* plane corresponding to Fig.3, starting at time zero.</figcaption></figure>

matters. Erasing the tape before recording is based on the same principle: the erase head is driven by a high amplitude HF signal and as the tape moves away from it and the field strength decreases, the magnetisation converges to zero.

The example shown in Fig.3 illustrates what happens for a relatively low frequency audio signal. At higher audio frequencies, the signal can not be considered constant during the time the record head field decreases, and things become again a lot more complicated.

It is of course possible to simulate this.<fnref n="2" /> Fig.5 shows magnetisation level as a function of signal amplitude for a set of signal frequencies. Fig.6 show the same data as a frequency re-

<footnote n="2">²For all the simulations shown in this section, the intensity of the *H* field is assumed to be proportional to the inverse square of the distance from the record head gap, decaying to half its maximum value at 0.012 mm. This values was obtained by trial and error, comparing the results to published HF loss data.</footnote>
sponse for a set of signal amplitudes.

<figure><img src="image_106.png" /><figcaption>Figure 5: *Simulated magnetisation level as a function of signal level for a set of frequencies, tape speed 381 mm/s.*</figcaption></figure>

<figure><img src="image_107.png" /><figcaption>Figure 6: *Simulated magnetisation level as a function of frequency for a set of signal levels, tape speed 381 mm/s.*</figcaption></figure>

Several observations can be made from these plots. Magnetisation is reduced significantly at higher audio frequencies. These also saturate at a lower amplitude. The resulting compression is not really 'soft', but it is clearly very frequency dependent. What actually matters is not frequency, but wavelength on tape, so these effects will be stronger at lower tape speeds.

Other effects are revealed by examining the actual magnetisation waveforms. Low frequency signals saturate by clipping softly. Higher frequencies don't, they are just attenuated. This is called the *self erasure effect*.

High level, high frequency signals also produce considerable amounts of aliasing due to signal harmonics mixing in a non-linear way with the bias signal.

### 3. TAPE RECORDING STANDARDS

A practical requirement when using tape recorders is of course that a tape recorded on one machine should play back correctly on any other one using the same tape format, track layout and speed. Two more things need to be standardised for this to be the case:

* The absolute magnetic flux on tape corresponding to some reference output level (e.g. 0 VU) and frequency.

* The frequency response of the output signal relative to the magnetic flux on tape.

Standards have evolved over time, but by the 1970s the most popular ones where the NAB<fnref n="3" /> one used by American manufacturers, and the IEC<fnref n="4" /> one used in Europe and Japan. Both cover the 7.5 and 15 IPS (19 and 38 cm/s) tape speeds. At the higher speed of 30 IPS (76 cm/s) the AES<fnref n="5" /> standard was often used.

The absolute flux level (measured in Weber/meter) is not relevant for a software emulation, but the equalisation certainly is. As shown in section 2.3, higher frequencies saturate at a lower magnetisation level. That means that they must be boosted on playback – it would not be possible to obtain a reasonable output level otherwise. This is contrary to e.g. FM radio which uses HF pre-emphasis in the transmitter and de-emphasis in the receiver. The consequence of this is that any playback EQ will be compromise between signal to noise ratio and the capability to handle high level HF signals.

All standards define the required high frequency equalisation as the response of a first order analog filter. Such a filter is fully defined by a time constant *T* or by its 3 dB frequency *F* = 1/(2π*T*). The table below lists these parameters and Fig.7 shows the corresponding frequency responses.

<table><thead><tr><td>Standard</td><td>T</td><td>F</td></tr></thead><tbody><tr><td>NAB 19 and 38 cm/s</td><td>50 us</td><td>3180 Hz</td></tr><tr><td>IEC 19 cm/s</td><td>70 us</td><td>2275 Hz</td></tr><tr><td>IEC 38 cm/s</td><td>35 us</td><td>4550 Hz</td></tr><tr><td>AES 76 cm/s</td><td>17.5 us</td><td>9100 Hz</td></tr></tbody></table>

The NAB standard also specifies a low frequency roll off on playback with *T* = 3180 us, *F* = 50 Hz. To compensate, bass frequencies are boosted when recorded, leading to a more pronounced compression effect at high levels. For the IEC and AES standards the +3 dB frequencies are proportional to tape speed and hence to wavelength. Up to around 10 kHz they are actually quite close to the inverse of the high frequency recording losses for small amplitude signals at all three speeds. The higher speed of 76 cm/s provides the best HF response and was sometimes used for master tapes, but it had its own problems in the form of irregularities in the very low frequency range.

<figure><img src="image_108.png" /><figcaption>Figure 7: *Playback equalisation standards.*</figcaption></figure>

<footnote n="3">³National Association of Broadcasters</footnote>

<footnote n="4">⁴International Electrotechnical Commission</footnote>

<footnote n="5">⁵Audio Engineering Society</footnote>
Note that the actual playback equaliser in any machine will be quite different from what these standards specify. It also needs to compensate for the +6 dB per octave response of the playback head, and for any losses due to the finite head gap size. The record equalisation is not standardised, it should be whatever is required to have an end-to-end flat frequency response.<fnref n="6" />

Professional tape recorders needed quite a lot of care and maintenance. Apart from cleaning and de-magnetising the tape path and checking the mechanical alignment of the record and playback heads, this included periodic tuning of the recording and playback levels and equalisers. Playback EQ was adjusted using a *test tape* with carefully calibrated magnetisation levels. On most machines this involved adjusting the playback gain and HF response. Test tapes were produced by standard laboratories and could be quite expensive. Record alignment depended on the type of tape, and had to be redone at least each time a new batch of tape was used. Usually three parameters needed to be set: bias level, record gain, and HF response. So for a 24-track machine the complete alignment procedure would require at least 120 adjustments for each tape speed.

## 4. REAL-TIME IMPLEMENTATION

For low signal frequencies it would be possible to precompute the resulting magnetisation as function of the signal amplitude and turn this into a form that can be evaluated efficiently, once for each output sample. This, plus some equalisation, is in fact all that typical tape simulation plugins actually do. The high frequency effects are then simulated by a static lowpass filter.<fnref n="7" /> Clearly this is not very realistic.

The only way to accurately simulate the high frequency effects is to go through the entire process illustrated by Figs. 3 for each output sample. With a bias frequency of 100 kHz and since the process is highly nonlinear, the sample rate would need to be around 1 MHz. Then, using the same parameters as for Fig. 4 each sample would require around 150 evaluations of a function that computes the magnetisation, and double that number at the lower tape speed of 190 mm/s. There are several mathematical models of hysteresis, but they all are share one quality – they are not simple, and doing this in real time on a general purpose computer seems out of the question.

The simulations that produced Figs. 5 and 6 were done using Python and Numpy, but with the inner loop – a simplified version of the Preisach [4] magnetic hysteresis model – coded in optimised C++. On the author's 3.6 GHz CPU this was running around 15 times slower than real-time. So clearly some pragmatic alternative solution is required.

Several potential solutions have been examined, and at the time of writing this paper it is not at all clear which one will be used for *zita-ts1*. It's not too difficult to create a relatively simple algorithm – a combination of filters and a non-linear element – that produces results similar to those shown in Figs. 5 and 6. Figure 8 shows one example, this should be compared to Fig. 5. But this only covers the case of simple sine waves, and how accurate this is for more complex signals remains to be seen.

One interesting potential solution would be to use a neural network trained by data created using the slow full simulation. Assuming this could run at 48 kHz sample rate, it would need to take around 14 samples of input and produce a single floating

<figure><img src="image_109.png" /><figcaption>Figure 8: Response of a relatively simple algorithm emulating high frequency magnetisation loss.</figcaption></figure>

point output. Finding out how far this could go would certainly be an interesting future project.

## 5. OTHER TAPE ARTEFACTS

*Tape noise* is not just additive, it is also dependent on the signal, i.e. there is a multiplicative component. Still this is relatively easy to emulate.

*Wow* refers to periodic and random tape speed variations, resulting in perceptible pitch modulation. It is not really an issue on professional tape machines.

*Flutter* is the result of small defects in the tape coating, or of dirt or dust affecting the contact of the tape with the head gap. It manifest itself as random short 'dropouts' affecting mainly high frequency content. It's not a nice effect, not even in small quantities, so one may question if there is any point at all in simulating it.

## 6. CONCLUSIONS

As should be clear by now, simulating the exact characteristics of analog magnetic tape recording by digital signal processing is not an easy exercise. All solutions known to the author (including *zita-ts1*) provide only an approximation of the 'real thing'. Probably the easiest *practical* solution in case such processing is wanted is to just use a real analog tape recorder.

The question remains: *is it worth it?* Opinions are divided. Some will tell you it's magic while others will say the whole thing is just superstition and hype.

The author was a sound engineer at the time when the music recording and broadcasting world made the transition from analog to digital, and still remembers the enthusiastic reaction of many musicians when hearing a digital recording of themselves for the first time. They just loved the fidelity and quality that digital provided, and all the talk of it being 'cold' and 'clinical' only came much later. Most of that actually originated in *audiophile* circles before creeping into the professional world.

During the analog tape era audio engineers did not adjust their machines to obtain a 'softly compressed warm sound' – they calibrated them for maximum quality. Individual tracks (guitar, bass, drums, ...) were sometimes recorded at very high level because the resulting compression could actually be beneficial and sound nice. But today there are much simpler ways to achieve that effect.

<footnote n="6">⁶For some examples of the actual frequency response of professional machines, see https://www.endino.com/graphs/</footnote>

<footnote n="7">⁷ This includes the well known Chow Tape. Although the paper [3] about this plugin gives the impression of describing a real physical model, it completely fails to consider the process described in section 2.3 and how this affects higher frequencies.</footnote>
Even the term 'compression' itself is somewhat misleading. A real audio compressor has attack and release times. The gain changes, even if they can be fast, are a smooth functions of time. No such thing happens with tape, whatever is done to the signal depends only on its intantaneous value. In other words, what you get is distortion, not compression.

Also consider that compression or limiting will only bring any benefit if the original signal has some nontrivial dynamic range to start with. At the time the *loudness wars* had not yet escalated to the really insane level we see today. What makes absolutely no sense is to first squeeze all dynamics out by aggressive compression and limiting and then sending the result through a tape emulator plugin – it’s a sure way to ruin your mix.

## 7. ACKNOWLEDGMENTS

Many thanks to the anonymous reviewers for their comments and suggestions for improving this paper.

## 8. REFERENCES

[1] Jay McKnight e.a., "MRL documents online," http://mrltapes.com/mrl-documents-online.html.

[2] Jay McKnight, "Biasing in magnetic tape recording," *Electronics World*, pp. 34–37, Aug. 1967, http://mrltapes.com/#Tech%20Papers.

[3] Jatin Chowdhury, "Real-time physical modelling for analog tape machines," in *Proc. of DAFx-19, Birmingham UK*, Sept. 2019, https://archive.org/details/dafx-2019-paper-3.

[4] Gustav Mörée and Mats Leijon, "Review of hysteresis models for magnetic materials," *Energies*, vol. 16, May 2023, https://www.doi.org/10.3390/en16093908.
LAC-25 Paper Session 8
# WEBPDL2ORK: BRINGING THE LINUX LAPTOP ORCHESTRA TO THE BROWSER

William Furgerson

Virginia Tech
Blacksburg, USA
williamfurgerson@vt.edu

Justin Kerobo

Virginia Tech
Blacksburg, USA
justinkerobo@vt.edu

Ivica Bukvic

Virginia Tech
Blacksburg, USA
ico@vt.edu

Bradley Davis

Virginia Tech
Blacksburg, USA
brdavis4@vt.edu

## ABSTRACT

In the following paper, we introduce WebPdL2Ork, a new browser-based port of Pd-L2Ork. We present a brief history of Pd-L2Ork and the Linux Laptop Orchestra (L2Ork), the motivation for a Web-based implementation, and its benefits. We explain how we adapted Pd-L2Ork to run in the browser using Emscripten and WebAssembly and detail the technical hurdles we encountered along the way as well as our solutions. We highlight the L2Ork-Tweeter Pd-L2Ork app, a core component of the Linux Laptop Orchestra since 2020 and the most complex patch packaged with Pd-L2Ork, running in WebPdL2Ork with near-parity to its desktop counterpart. Finally, we list the remaining shortcomings of WebPdL2Ork as well as our plans for future work to address its shortcomings.

## 0.1. Funding and Copyright Notice

This work relates to Department of Navy award N00014-22-1-2164 issued by the Office of Naval Research. The United States Government has a royalty-free license throughout the world in all copyrightable material contained herein.

## 1. INTRODUCTION

The Linux Laptop Orchestra (L2Ork) [1], was founded in 2009. To meet the specific needs of L2Ork, Pd-L2Ork [2] was developed as a custom version of Pure-Data (Pd). Building on the defunct Pd-Extended, Pd-L2Ork has continued to evolve and expand alongside L2Ork, as well as finding use in K-12 education [3]. Although Pd-L2Ork was initially developed as a Linux-only application, it has since been ported to both Windows and macOS, broadening its reach, while also adopting a nw.js front-end. The introduction of libPd [4] [5], a lightweight embeddable version of Pure Data, has since shown the feasibility of integrating Pd-based processing into external applications. This approach gained traction in both commercial and experimental settings, including use in Electronic Arts' game Spore [6]. More recently, Web technologies have made it possible to bring Pd-based environments to the browser. A notable example is PdWebParty [7], created by Zack Lee during the 2020 Google Summer of Code (GSOC), which leveraged libPd and WebAssembly [8] to allow for basic execution of Pd patches within a Web browser.

Building on these foundations, this paper introduces WebPdL2Ork, a Web-based implementation of Pd-L2Ork which extends PdWebParty's capabilities. By implementing many of the features that PdWebParty lacks, WebPdL2Ork enables near-complete execution of Pd-L2Ork patches directly inside a Web browser. This development significantly broadens Pd-L2Ork's reach potential, reducing installation barriers, particularly in the K-12 sector, and increases its usefulness.

## 2. MOTIVATION

The motivation for a Web-based version of Pd-L2Ork stems from a desire to resolve compatibility and consistency challenges. The Linux Laptop Orchestra has historically required that users install and configure a full desktop version of Pd-L2Ork in order to run L2Ork Tweeter (A Pd-L2Ork patch used by L2Ork for real-time composition and performances). This friction has limited the diversity of platforms that can be readily used with it. By adding a Web-based option, we hope to eliminate many of these barriers and allow participants to use L2Ork Tweeter instantly without any installation process, from any device with a Web browser. L2Ork Tweeter has also served as a crucial benchmark due to its size and complexity. As such, it has become a stress and regression test platform for the ongoing development of both Pd-L2Ork and WebPdL2Ork.

Such an option would enable even broader participation in L2Ork by lowering the technical requirements in two ways. First, users would not have to worry about how to install and/or update Pd-L2Ork on their systems. Second, we would not need to worry as much about edge cases that only affect certain system configurations. Since Web browsers are fairly standard in how they behave, testing and fixing bugs becomes easier for browser-based applications than their native counterparts, where every system is potentially a unique environment.

Finally, this motivation aligned with another one of our research efforts focused on making Pd-L2Ork more accessible for Science, Technology, Engineering, and Math (STEM) K-12 education. For that particular research, we were interested in making Pd-L2Ork accessible for use in middle and high school curricula to teach wave physics, where schools face challenges installing and maintaining software due to limited IT resources. To address this, we were already exploring an online runtime platform, and given that the same solution would greatly benefit L2Ork Tweeter, it made sense to pursue a fully featured Web-based port of Pd-L2Ork.

## 3. IMPLEMENTATION

### 3.1. Foundation

Our new extension to Pd-L2Ork is based on two prior works by other authors. libPd [4, 5] serves as the foundation of WebPdL2Ork, allowing the use of Pd-L2Ork's core as a library. Using libPd's API, we can control Pd-L2Ork's core from the outside, tell it to open, close, or run patches, and add hooks to receive events when data changes, allowing us to instrument the state of Pd-L2Ork and perform additional processing. We make extensive use of these hooks for our new GUI and compatibility layer, WebPdL2Ork. We also make use of PdWebParty [7], a proof-of-concept bare-bones Web UI based on libPd, as a foundation for WebPdL2Ork. PdWebParty uses Emscripten [9] to compile Pd-L2Ork's core (and libPd) into WebAssembly [10],
<figure><img src="image_110.png" /><figcaption>Figure 1: WebPdL2Ork Architecture</figcaption></figure>

so that it can run inside a browser. It implements a very lim-
ited set of core GUI objects (Bang Buttons, Toggles, Sliders,
Radio buttons, Canvases, and Comments), all vanilla non-GUI
objects, and a subset of external libraries (externals), and as a
result has significant limitations on what patches it can support.
We based WebPdL2Ork on the foundation provided by PdWeb-
Party, added the full set of GUI objects from Pd-L2Ork, added
nearly all remaining Pd-L2Ork externals, and addressed almost
all of PdWebParty's limitations on what patches are able to run.

3.2. Architecture

As a Web application, WebPdL2Ork is split into a client and a
server. We serve WebPdL2Ork using Express.js [11, 12] and
Node.js [13] behind an nginx [14] proxy. The server hosts
WebPdL2Ork's static JavaScript and WebAssembly code, which
is then run by the user's browser when they visit WebPdL2Ork.
WebPdL2Ork's server also provides patches to the client. It
hosts a small collection of patches itself and also supports load-
ing patches from external URLs. In either case, the client will
request that the server fetch the patch for it, and the server will
serve that patch to the client for it to open in the browser. Any
static files used by a patch are handled in the same way. We
maintain a docker [15] image and sample docker-compose file
in our GitHub repository [2] as well as documentation, so that
others who are interested in running an instance of WebPdL2Ork
can do so.

## 3.3. Limitations of PdWebParty

While PdWebParty offered a solid foundation, as a proof-of-
concept implementation it has several major shortfalls which
limit what patches it can run. It does not support many GUI ob-
jects, such as drop-downs, text boxes, links, images, arrays, and
others. It cannot handle abstractions (additional patches loaded
from another patch at run-time) or graph-on-parent (displaying
GUI objects which live inside a subpatch or an abstraction), both
of which are used heavily in L2Ork Tweeter. It is missing about

20 external libraries, many of which (cyclone, for instance) are
used heavily in many Pd-L2Ork patches, like L2Ork Tweeter.
It does not support networking, meaning that L2Ork Tweeter's
main purpose as a collaborative telematic platform for musick-
ing is lost. Finally, it suffers from a major limitation of libPd.
In Pd-L2Ork, there are two kinds of connections between ob-
jects, so called wired and wireless. Wired connections consist
of an explicit connection between two specific objects. Every
wire connects exactly two objects together, bridging the output
of one object with the input of another. Wireless connections, on
the other hand, can be thought of as transmitters and receivers.
Objects can transmit wirelessly using a specific address, and any
objects that are listening using the same address will receive the
data that was sent, as if on a wire. Any number of objects can
transmit or receive data using the same address. Most objects in
Pd-L2Ork only support wired connections, with the notable ex-
ception of GUI objects. Most GUI objects can be connected ei-
ther using wires or using wireless connections, but libPd's hooks
only work on wireless connections. Any data send over wired
connections cannot be monitored by libPd, or PdWebParty by
extension, since it receives all of its state information via hooks,
Therefore, any GUI objects that use wired connections - the most
common type of connection by far - will not work properly in
PdWebParty.

## 3.4. GUI Objects

In WebPdL2Ork, we have added support for all non-dePRECATED
GUI objects that are actively maintained in Pd-L2Ork and were
missing from PdWebParty, including those introduced after its
initial release. Since the core (which is compiled from C to
WebAssembly) is responsible for audio processing and patch
logic and does not handle any GUI operations, all user inter-
action must be managed externally with LibPd serving as the
interface through which external code communicates with the
core. As a result, we implemented all GUI object interactivity
in WebPdL2Ork from the ground up using JavaScript (the native