# BuGUI: A Widget System for Button Grid Controllers

Thibaud Keller, Victor Lazzarini, Gordon Delap, Jean-Michaël Celerier

► To cite this version:

Thibaud Keller, Victor Lazzarini, Gordon Delap, Jean-Michaël Celerier. BuGUI: A Widget System for Button Grid Controllers. Proceedings of the 19th Linux Audio Conference, Jun 2025, Lyon, France. hal-05096026

HAL Id: hal-05096026

https://hal.science/hal-05096026v1

Submitted on 3 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# BUGUI:
A WIDGET SYSTEM FOR BUTTON GRID CONTROLLERS

Thibaud Keller

Ossia team
Music Dept Maynooth, Ireland

thibaud.keller.2025@mumail.ie

Gordon Delap

Music Dept Maynooth, Ireland

gordon.delap@mu.ie

Victor Lazzarini

Music Dept Maynooth, Ireland

victor.lazzarini@mu.ie

Jean-Michel Celerier

Ossia team
SAT Montreal, Canada

jmcelerier@sat.qc.ca

## ABSTRACT

BuGUI is a library intended to facilitate user interface design on creative hardware, especially button grid controllers. Our approach is to consider these devices as very low resolution touch screens, with some additional controls potentially (buttons, encoders, actual screens). To that effect, the library is structured as a simplistic yet general-purpose UI system.

This effort to keep our implementation generic makes it easily extensible and eventually capable of covering a wider range of both hardware interfaces and software use cases.

This paper details our current, early-stage development, on which we base a plugin for the *ossia score* software sequencer. As we aim to provide a full-featured hardware interface for score, this plugin also offers BuGUI an extensive case study.

## 1. INTRODUCTION

BuGUI<fnref n="1" /> acts as an abstraction layer between software and hardware. Our motivation stems from observing the wide variety of uses for grid controllers. Unfortunately, these creative uses, in most cases, remain specific to a particular piece of software. Whether it is in the form of standalone, fully integrated systems, or desktop applications compatible with external controllers, most projects reimplement their own hardware interactions. This approach not only limits hardware compatibility, as each project must maintain their own catalogue of available controllers, it also duplicates software implementations of similar use cases.

We propose a generic approach to hardware interfaces, inspired by traditional step-sequencer affordances, where these devices are essentially considered in the same way that screen, mouse and keyboard are in regular GUI development.

### 1.1. Ossia score

*ossia score* [1] is an interactive sequencer, based on a branching temporal model. Durations are defined in independent intervals that can play sequentially or concurrently, and otherwise overlap freely. These intervals can also be set to a fixed duration, or a relative one, depending on user-defined conditions.

Scenarios in *ossia score* are defined with a set of generic temporal elements, summarised by the following list:

* *States*: a single point in time

* *Events*: a group of simultaneous *States*

* *Syncs*: a group of simultaneous *Events*

* *Transitions*: instantaneous jump between two distinct *Syncs*

* *Intervals*: flexible duration connecting two distinct *Syncs*

* *Scenarios*: spanning the duration of an *Interval*, it implements recursion and ordering

Furthermore, the user defined conditions that make these elements interactive take one of two forms:

* *Triggers*: extends or interrupts the predefined duration of *Intervals*

* *Conditions*: allows for selecting between parallel *Events* sharing the same *Sync*

For comparison, the set of temporal elements used in hardware sequencers could be defined as follows:

* *Steps*: a single point in time

* *Sequences*: a circular chain of *Steps*, spaced in multiples of a predefined musical subdivision

* *Patterns*: a group of concurrent *Sequences*

* *Songs*: a linear arrangement of *Patterns*

Temporal control requires additional elements, defining for example the direction of the sequence: forward, backward, mirrored (forwards and backwards alternatively), random, etc.

This list features a one-to-one relationship with hardware interfaces elements: either *Steps* or *Patterns* are represented with a single backlit button, and *Sequences* or *Songs* with one or several rows of backlit buttons.

Similarly, *ossia score* provides graphical representation of its temporal elements with dots (*States*), horizontal (*Intervals*) and vertical lines (*Events*, *Syncs*), along with simple icons (*Triggers*, *Conditions*).

The *ossia score* plugin for button grids we describe here proposes first a translation of the UI, simplified down to an arrangement of coloured cells, rows and columns. From our initial assessment, this hardware representation is enough to convey the structure of a scenario. Current developments focus on providing basic control of this structure with tactile inputs on the grid.

The representation and manipulation of interactive scenarios as defined in *ossia score* can then lead to the design of a unique grid interface, demonstrating further the versatility of these devices.

*ossia score*'s GUI also features many common place widgets like sliders, spin boxes, and a nodal patcher, that have been introduced on grid based products. A full-featured hardware interface for it, as we aim to provide with this plugin, would also offer a cohesive, open-source implementation of these widgets on hardware.

<footnote n="1">¹https://codeberg.org/bugui/bugui</footnote>
<figure><img src="image_2.png" /><figcaption>Figure 1: *Representation of a scenario on the Launchpad Pro. Syncs and Events displayed only when a child State is pressed*</figcaption></figure>

## 1.2. Similar projects

In 2017, Van Haaren presented OpenAV's Ctlra[2], a great project we initially considered for the purposes of *ossia score's* hardware integration. BuGUI shares Ctlra's objective of centralizing support for many devices in a single library, as well as abstracting away the communication protocols.

It is worth noting however that Ctlra and BuGUI mainly target different hardware. We aim to provide integration with common grid controllers that are, in most cases, simple MIDI devices. Although Ctlra can work with generic MIDI controllers, this aspect is still marked as "in progress, but not complete" in the code repository<fnref n="2" />. Its primary targets are "Modern Controllers", featuring large numbers of input controls, LED feedback, often high-resolution screens, and communicating over the HID protocol.

The "scripts" used by Ctlra to interface with a particular device are meant to be customizable, and potentially unique to each user, even on the same software. As we'll elaborate further, we took a more fixed approach and limited hardware specifications to an arguably simpler format.

The work of Hunt[3] is also closely related to our endeavour. His Universal Grid Sequencer (UGS) allows creating independent sequences executed simultaneously on one or more button grid controllers. It leverages hardware interactions for more tangible feedback of potentially complex temporal and rhythmical relationships.

Instrumental patterns created with the UGS framework can fall in and out of phase or synchronisation according to predefined numbers of steps and tempos. Such patterns are designed in two distinct stages. Before starting the system, multiple instances are set up in software, with hard-coded voices, steps and tempo. The content of each pattern can then be edited in real-time once the system is running.

Although very different from the more general purpose library we describe here, this work echos our main use case: hardware integration for the *ossia score* sequencer. Button grid controllers are the UGS's central focus, and we also note that the sequences rendered on the devices are defined and behave similarly to common widgets.

## 1.3. Why button grids ?

Button grids have become ubiquitous in digital live performance. More and more of these devices are released every year (see Figure 2 in [5]). Button grids are defined as backlit button matrices, that offer both individual tactile input and light feedback for each cell.

The interaction they afford is closely tied to the usage of hardware step sequencers and drum machines. However, various innovative implementation enable a greater variety of interfaces in different products. Here is a non-exhaustive list of available button grid interfaces:

* Keyboards / drum pads

* Step sequencers

* Discrete sliders

* "Clip" launchers (Ardour, Ableton Live)<fnref n="3" />

* Text displays (Novation Launchpad)

* Basic spin boxes (Novation Circuit)

* Arrangement views (Synthstrom Deluge)

* Routing matrixes (Synthstrom Deluge)

* Rudimentary waveform croppers (Synthstrom Deluge)

* Nodal patchers (Empress Zoia)

These interactions are in most cases specific to a particular hardware, or software.

A notable exception in this field is the company Monome<fnref n="4" /> that pioneered it. They offer an open-ended environment, technically compatible with a wider range of devices, but intended mainly to run on their Norns system, connected to their Grids controller. Over the years, Monome has garnered a dedicated community of artists and developers, due in part to their commitment to open-source.

Norns enable programming Lua scripts that encompasses audio processing, graphical user interface, and hardware integration. As such, this system enables very involved controller interactions and graphical renderings, but does not seem to be suited for hardware integration in other software. For example, the port of the Orca sequencer to Norns has apparently required a full Lua rewrite<fnref n="5" />.

## 1.4. Existing literature

Button grid practices have been thoroughly documented in recent years at LMU Munich by Beat Rossmy[4], with Alexander Wiethoff[5], Maximilian Rauh[6] and Sebastian Unger[7]. A taxonomy of grid interaction was also published in 2021[8], consolidating further what we perceive as a strive for a certain standardisation of the practice. The research conducted at LMU also shows, among many insights, a clear upward trend in the release of devices featuring a button grid interface since the late nineties.

The Avendish project[9] is a major influence, as we attempt to mirror its declarative style for our hardware specification classes. Avendish leverages C++ reflection to create generic, framework-agnostic media processors and UI. Users list control parameters in C++ aggregates, and write their processes in basic functions, without any dependency or reference to the environment they will eventually be compiled for. The same generic C++ code can then make use of various wrappers to produce media processors for *ossia score*, Python, PureData and other environments. The naming scheme of these aggregates and functions suffices to retrieve all necessary information at compile-time.

<footnote n="2">²https://github.com/openAVproductions/openAV-Ctlra</footnote>

<footnote n="3">³Desktop applications integrated with external controllers</footnote>

<footnote n="4">⁴see Monome's website: https://monome.org</footnote>

<footnote n="5">⁵see the forum entry for the projecthttps://llllllll.co/t/orca</footnote>
This approach of abstracting away the particulars of the im-
plementation is also applied to the definition of custom graphical
user interfaces. Our use of this technique is currently limited, but
very relevant to our needs, as we will demonstrate.

## 2. IMPLEMENTATION

This section details BuGUI's chosen architecture, in its current, early stage development. We will present this bottom-up design, starting from hardware specifications, then moving to core functionality and finally describing the API.

### 2.1. General device parameters

We are focussing first on supporting the Novation Launchpad Pro<fnref n="6" />, as it is a fairly well-known, common and full-featured controller. We defined its specification in a simple C++ aggregate, containing the following static functions:

```cpp
static consteval auto sysex_prefix ()
{
    return std::array<uint8_t, 6>
        {240, 0, 32, 41, 2, 16};
}

static consteval uint8_t sysex_sufix ()
{
    return 247; }

static consteval auto clear ()
{
    return std::array<uint8_t, 2>{14, 0}; }

static consteval uint8_t light_prefix ()
{
    return 10; }
```

These functions are marked consteval to ensure that the value they return is available at compile-time. For communication to the device, we elected to use system exclusive messages (Sys-Sex). These types of flexible MIDI messages can carry an unspecified number of data bytes. They allow concatenating several instructions together, reducing the number of messages sent to achieve the same result. Several features of the device are also only available through SysEx, allowing for example to turn off all LED feedback with a single message. We take advantage of this feature when defining the clear function, which returns the specific part of the SysEx message to call this functionality on the device. This pair of values can then be concatenated between the SysEx prefix and suffix, returned by their respective functions, to create the required message and clear the device once sent to the appropriate port.

We use SysEx messages in the same way to set the colour of individual LEDs on the device. The same procedure of concatenating values between SysEx prefix and suffix is employed. The specific content of this message starts with the returned value of the color_prefix function, followed by the identifier of the chosen button, and its desired colour. The full message takes this form<fnref n="7" />:

240,0,32,41,2,16,10,<LED>,<Colour>,247

Additional information about the device are defined in nested aggregates as follows.

### 2.2. Identifiers

At present, we only defined a few buttons of interest for the Launchpad Pro. Here we borrowed a technique from Avendish that uses empty ints as compile-time *flags*. Several C++ concepts can be defined to determine the existence of such ints<fnref n="8" />, and differentiate them by name.

```cpp
struct buttons {
    enum cc{};
```

```cpp
static consteval uint8_t shift_id ()
{ return 80; };
static consteval uint8_t stop_id ()
{ return 8; };
static consteval uint8_t up_id ()
{ return 91; };
...
```

In this context, we use this technique to determine that the type of midi messages the Launchpad's buttons are sending is MIDI Control Change (CC). The static functions in this aggregate simply return the unique identifier of a particular button.

The grid aggregate is structured in the same way. However in this case, instead of individual functions per button, a single function returns a two-dimensional array of all identifiers for the grid's buttons, so that they can be accessed with a two-dimensional index.

```cpp
static consteval auto ids () {
    return std::array<std::array<uint8_t, width()
        >, height()>
        {
            std::array<uint8_t, width()>
                {81, 82, 83, 84, 85, 86, 87, 88},
            std::array<uint8_t, width()>
                {71, 72, 73, 74, 75, 76, 77, 78},
            ...
        }
}
```

### 2.3. Colours

The last nested aggregate necessary for the device specification is reserved for a colour lookup table. The palette function returns a map of hex formatted colours predefined in the device, associated to an integer value. In this map, the colour is the key and the value is the associated integer, from 0 to 127. Technically, the Launchpad Pro's LED can be set with messages containing RGB value, but this option significantly increases message size, and is not supported for every functionality that takes colour arguments. For the sake of simplicity and coherence, we decided to only support colour specification in the form of a lookup table, at least for now.

### 2.4. Handling the painter

bugui::midi_hardware is a class that offers a generic interface to a specific device. It enables drawing on a grid and retrieving input events, similarly to a more traditional GUI library drawing on screen and retrieving mouse clicks and key presses. The handle_painter function is passed a reference to a painter object that contains the desired state of the grid. This state is expressed as a generic two-dimensional vector of colours.

```cpp
if constexpr(sends_sysex<Device>)
{
    const auto p{Device::sysex_prefix()};
    libremidi::midi_bytes
        b{p.cbegin(), p.cend()};
    b.push_back(Device::light_prefix());
    ...
    b.push_back(Device::sysex_sufix());
    m_output.send_message(b);
}
```

Device refers here to midi_hardware's first template argument. At the moment, it can only be substituted by either of two Launchpad_Pro types, corresponding to two different modes of the device (we provide both options for testing purposes).

The function starts by determining if the device specification allows for sending SysEx messages. The control structure here could be extended to support devices that only expects plain midi messages, without any runtime overhead, since the sends_sysex concept is determined at compile-time.

This code snippet shows the previously stated method of concatenating together parts of a SysEx message. In this case,

<footnote n="6">⁶https://novationmusic.com/products/launchpad-pro</footnote>

<footnote n="7">⁷as seen in the Launchpad Pro's user guide:
https://media.discopiu.com/files/2020/1/29/648553-original.pdf</footnote>

<footnote n="8">⁸see include / concepts / message.hpp in the source code</footnote>
the colour values are determined from the content of the painter object (not shown here), and the rest of the message is retrieved from the Device type.

## 2.5. Handling inputs

The challenge in handling inputs from the device is that we need to match the incoming control identifier with its associated function call in the most efficient way. To that effect, we first differentiate between the type of MIDI message we receive. In the case of the Launchpad Pro, we previously saw that buttons outside the grid all send control changes, whereas the grid only sends on/off notes. Other devices may not offer such a clear-cut distinction however, so this part of the code in particular needs to be easily extendable.

To avoid repetition, the following control structure is defined in the `case_msg_type(C)` macro, where C is used for text substitution, to call the appropriate concept. When a control change message is received, for example, calling `case_msg_type(cc)` guarantees that only off-grid button identifiers will be evaluated for a match. This use of concepts enables the compiler to completely ignore the possibility of testing identifiers of the grid buttons against an incoming control change message.

Member functions testing for these matches return false if no association is found between the given input and a definition in the controller class (see 2.6). In this case, the control structure would continue to the next condition. To handle more fine-grained distinctions between input message types, additional conditions can then easily be defined in future development.

```c
ifnoon
    (is_##C<ottypename Device:: buttons>)
{
    if (on_button_received
        (ctrlr, message bytes))
        return;
}

ifconstexpr
    (is_##C<ottypename Device:: grid>)
{
    if (on_grid_received
        (ctrlr, message bytes))
        return;
}
```

## 2.6. The controller class

`bugui:: controller` is the main class users are supposed to inherit. It can be thought of in the same way a `mainwindow` class operates in a regular GUI library. It is the root of the widget hierarchy, and provides generic hardware access through a unique instance of `midi_hardware`.

In the derived class, we propose a system to facilitate hardware integration of common application functionalities like *shift*, *play*, *stop*, etc. Most devices also feature dedicated buttons labelled with these functionalities. As shown previously in the device specification, we named the static functions returning button identifiers accordingly.

Several functions can be defined in the controller's derived class, and directly mapped to the appropriate button press and release. These functions are not overridden. Their name must prefix the button label with `on_`. They must also be void and take a single boolean argument (i.e. `void on_shift(bool pressed)`).

To enable a certain button press/release to call their associated user defined function, we first establish the existence of such button and function at compile-time. If the user has defined no such function, or if the device does not feature this particular button, this possibility won't be compiled into the project, reducing the number of conditions to evaluate at run-time. Here

again, we place the following constraint in a macro to avoid code repetition.

At present, we consider that the static function returning the button identifier is valid so long as it is not void. We aim to correct this shortcoming soon and establish a set of sensible identifier types.

```c
#define feature_exists(F)
if (constexpr
    (requires (Controller c, bool b)
        {
            std::is void v <type (c. on_##F(b))&&
            !std::is void v < declaration
            (Device:: buttons::F##_id()) >;
        })
```

## 2.7. C++ as a DSL

In our approach, the C++ classes required to support a specific device act as configuration files. We do consider supplying device specification through JSON files to be more practical, expressive and less verbose. However, relying on this format would limit us to loading hardware specification at run-time. We would have to rely only on dynamic types, and perform more operations when interacting with the device. We would have to account for all possibilities at run-time, having no prior knowledge of existent, or missing features of a particular device.

Thus, we are focussing on C++ aggregates, and compile-time programming. The first advantage is performance, and the ability to rule out certain interactions not supported by the device or the application. Keeping these C++ classes declarative also protects users against run-time errors such as memory issues, etc. As we reserve all logic and memory allocation to our hardware class, a user contribution to support a particular device can be limited to defining simple aggregates and static functions. Ill-formed classes will be caught by static type checking and cause compile errors. They may then be addressed early.

## 2.8. Widget tree

After deriving a custom class from `bugui:: controller`, creating widgets is the second major part in `BuGUI`'s API. Users are only meant to interact directly with the `controller`, the `baseWidget` and `containerWidget` classes, as well as the painter and the presser objects.

The `controller` already inherits from `containerWidget`, enabling it to handle button presses, to draw on the device, as well as adding, owning and deleting its children widget. In turn, these user defined widgets can be parent of other widgets, also able to draw, handle presses and so on.

Each `widget`, `controller` included, can override the paint function, where the `draw_cell` and `draw_line` method of the painter can be called. The `controller` owns the painter object. Upon update, a reference to this object is first passed to the controller's paint function, then to all the controller's children widgets. Every `widget` behaves this way recursively, as to make the painted cells from their paint function the background for their children's.

Custom `widgets` can also override the `onPress`, `on_double Press` and `on_drain` functions. When one of these events is received from the device, its `x` and `y` position on the grid is determined and stored in the presser object. A reference to this object is then passed to the `controller`, and recursively to the first `widget` whose geometry encompasses the `{x, y}` coordinates in question. Depending on the event, either of these three overridden function will be called on the last `widget` that meets this criteria, so long as none of its children does too.
# 3. SCORE'S HARDWARE PLUGIN

In this section, we discuss the first project utilising BuGUI: *ossia score*'s hardware integration<fnref n="9" />. This plugin is a work in progress, and statements made here may not all apply in the near future.

## 3.1. Proof of concept

Our first attempt at displaying *ossia score*'s interface on a button grid was based on previous versions of BuGUI that did not feature a widget system, but only a painter. *Intervals* created and manipulated in *ossia score*'s main *Scenario* were directly painted on the grid, but the grid did not offer any access to these *Intervals*.

As expected, it was quickly confirmed that readability would be greatly compromised if every element of the *Scenario* were to be displayed on the grid. Indeed, small elements such as *Intervals* and *Events*, represented in the GUI by horizontal blue lines and vertical white lines, take very little space on screen. They can be drawn very close to one another without there being much confusion where one element ends and another starts.

On the grid however, we are working with a significantly smaller area. Additionally, even small elements, like a single dot on screen, cannot appear any smaller than one cell thick on the grid. This early version made it clear that we ought to only show on the grid elements that are strictly necessary to the current interaction (see Figure 1). Before our efforts to display a *Scenario* on the device went any further, we decided to first implement some level of access to the *Scenario* from the grid, if only to identify possible interactions, and draw accordingly.

## 3.2. Navigation

As well as aiming to de-clutter the display, we also needed to easily navigate inside the *Scenario* and zoom in on specific parts. Navigation also helps us avoid rendering the full span of the UI on our limited grid-estate (as in [5]).

Our currently supported hardware, the Launchpad Pro, features navigation buttons above the grid. We defined the `on_left`, `on_right`, `on_up` and `on_down` functions in the controller as previously described (see 2.6).

These functions apply an offset and a scale amount to the coordinates of the painter. Navigation and zoom features were then trivial to implement before we introduced the widget system. The added complexity in our current state of the project stems from a widget's coordinates being relative to their parent. Offset and scale now also have to apply to the presser object.

Double clicks with the mouse are also fundamental when navigating a *Scenario* in *ossia score*. They are used to put the content of an interval in full-view, and easily move in and out of nested *Scenarios*. Although we don't yet support this type of navigation in the hardware plugin, detecting double presses is functional in BuGUI. When identified, it is set as a flag in the presser and passed to the `on_doublePress` function of the relevant widget.

## 3.3. Intervals first

*Intervals* are central to *ossia score*'s structures. They are at root of its object hierarchy, and can hold any *Process*, whether it be a piece of media, a signal processing graph, or a *Scenario* (remember that *Intervals* can hold more *Processes*).

The first visible widget made with BuGUI is a representation of an *Interval*. The first user interaction we enabled for these widgets was the ability to be moved vertically, as one would on the GUI by dragging up or down with the mouse.

To that effect we implemented drag detection in BuGUI. The expected behaviour is similar to a touch screen, where users can

press down on a specific graphical element that will follow the position of their finger until they lift it up. On a button grid, we can provide a discrete version of this interaction. Since we could already detect presses on the same button in quick succession for double press, drag detection simply employed the same principle with button presses one cell apart. The coordinates of both the first and the second button are passed together to the presser object. We found this interaction quite intuitive, and are able to move *Intervals* up and down smoothly. We are also able to produce these movement in parallel with both hands, moving two *Intervals* independently at the time, something that is not possible with the mouse on the GUI.

## 3.4. Free from the GUI

Going further with *ossia score*'s hardware integration, it became clear that this plugin could not simply be a companion piece to the graphical interface. Initially, we made liberal use of Qt signals and slots from the GUI, propagating and notifying of value changes. This approach required the hardware and the GUI to always display the same *Scenario*. Navigating to a sub *Scenario*, or attempting to open a *Scenario* without the GUI (in "headless" mode), rendered the device wholly unresponsive.

We also currently retrieve *ossia score*'s colour palette from the GUI. The palette is loaded at startup, unless it is running in "headless" mode.

To circumvent our reliance on the GUI's signals and slots, we implemented recursive functions to propagate changes to relevant elements of the *Scenario*. Currently, these functions allow basic edits on an existing *Scenarios*, but are not yet integrated into its undo/redo system.

To address the colour palette issue, we'll have to detect if it was previously loaded by the GUI, and load it ourselves in the plugin if not. More and more, proper hardware integration is starting to look like designing a new, complete interface for *ossia score*.

Our ultimate goal is to provide users with a full experience of creating, editing, and performing *Scenarios* without requiring the GUI. It would be a testament to the viability of grid interaction, not just for *ossia score*, but, we believe, for almost any intermedia authoring software. Although this perspective remains further ahead into the future, it is guiding our design choices at every step.

# 4. FUTURE WORK

In this section, we will detail the project's current objectives, and lay out its future milestones. Keeping with our current workflow for now, additional features in BuGUI will continue to be added as they are needed in *ossia score*'s hardware plugin.

Along with future development, we hope to gain access to a variety of controllers. The hardware specification file described in this paper (see 2.1–2.3) is optimized for being written easily. This would enable us to quickly provide support for additional devices, and open the door for contribution from a wider audience, maybe even by not necessarily technically minded users.

## 4.1. Ideal interaction

For our interactions to be intuitive, we aim to mirror established gestures common in regular touch screens (as in [5][7]). We also need to remain consistent with the use of cursor actions in *ossia score*, such as double clicks, right clicks, drag and drops or mouse hovering.

We are currently in the process of implementing the double click behaviour from the GUI, with double presses on the grid. This action performed on an *Interval* would display the processes contained in that interval in full view. Double pressing the background of a *Scenario* would create a floating *State*.

<footnote n="9">⁹https://codeberg.org/bugui/scoreaddon-hardware</footnote>
Creation via double presses could also be extended with the introduction of held widgets, equivalent to a mouse selection. Pressing down and holding one or more *State* widgets, followed by a double press elsewhere on the grid, could create new *Intervals*. These *Intervals* would connect the held *States* to the *State* that received the double press. If a double press is performed in the background of the *Scenario*, a new *State* would first be created before creating the *Intervals* connecting it to the *States* currently held.

Holding widgets on the grid could also offer an equivalent to mouse hovering in the GUI. For example, hovering over an *Interval* in *ossia score* reveals a *Mute* icon and a slider to set its speed. Similarly, holding down an *Interval* widget on the grid could show a discrete slider as a row of cells. If a *Mute* button is available on the device, it could be mapped to the *Mute* function of that particular *Interval* when held.

## 4.2. Additional Features

Currently, only a subset of common buttons outside of grids are supported. We hope to extend this subset and support additional controls like sliders, knobs and encoders.

We also note that not all controls are labelled on every controller, and many are simply arranged in one dimensional arrays. Our widget system should also target these arrays.

Some devices feature an actual screen of varying resolution, from basic dot matrixes to modern OLED displays. Supporting screen rendering on these devices would mitigate certain issues such as displaying menus or parameter names. Ideally, we would like to support all types of interfaces, even text based, directly on the grid, but have the option to offload particular interfaces that are better served by a screen, if one is available.

Screens could provide some of the functionalities defined in the application, but without a dedicated control on the device. The Launchpad Pro for example, does not feature a *Play* button. If a screen was available to us, possibly the screen of another connected device, and this other device was also missing this button, *Play* ought to be accessible through the screen. This method, if generalized, would provide a “decentralized” UI, where features on screen are always replaced with hardware control if available.

## 4.3. Optional dependencies

As previously mentioned, similar open-source projects like libmonome and Ctrla already support several devices, and provide hardware integration APIs. Both also offer embedded screen rendering. We will consider the feasibility of adding these libraries as optional dependencies.

Although the scope of these projects may differ from ours, one could conceive of a wrapper making use of certain functionalities of interest. It would provide a unified, minimal, modern C++ interface, using the libremidi<fnref n="10" /> backend as we currently do for generic MIDI devices, a Ctrla backend for HID devices, and a libmonome backend for SerialOSC devices like Monome's Grids.

## 4.4. Beyond *ossia score*

The generic widget system we are building could serve many other applications than *ossia score*. Our current focus on Scenarios is very specific to the software's interface, but nodal patchers, automation curves and step sequencers will also be part of our plugin. After integrating these processes, we intend to upstream as much code as possible into BuGUI. In turn, this library may provide a generic implementation of nodal patchers for example, that could be used directly in environments like PureData or Cardinal. In effect, we would provide the user interface basis for an open-source equivalent to the Empress Zoia.

## 4.5. Embedded systems

One of the possible outcomes of this project would be to enable the full use of *ossia score*, or any software integrated with BuGui, on embedded devices. With a grid and a few additional controls, in lieu of a mouse, a keyboard and a regular monitor, this library would ensure no loss in usability. The user base for these embedded grid devices has already shown its dedication through the success of Monome's Norns (with Grids), Empress's Zoia or Sythstrom's Deluge. Not only were these products adopted in significant numbers, they also formed communities of contributors, creating new scripts, patches and revisions to the firmware, expanding their potential further.

We hope a sufficiently extensive version of BuGUI could soon benefit from the feedback of this community. We also hope that embedding *ossia score* in one of the aforementioned grid devices, or a comparable device of our making, opens the software to the same sort of “hands-on”, performance-driven and immediate creative experience for users.

# 5. CONCLUSION

This paper presents BuGUI, a library enabling efficient interaction with hardware grid interfaces. It enables the creation and manipulation of a basic widget tree on such interfaces. The library can also determine automatically the correspondence of a specific button and its appropriate function call.

By leveraging C++ concepts, we reduce run-time overhead and aim to simplify the process of supporting new devices.

The design of the widget system is meant to remain generic and function similarly to general purpose graphical user interface frameworks.

We put forward a plugin for the *ossia score* sequencer based on this library, and are guiding its evolution. Although still in early development, the plugin can already render elements of its *Scenario* tangibly. Basic interactions currently available reveals the potential of translating a common drag motion on a touch screen to a hardware grid.

# 6. ACKNOWLEDGMENTS

We would like to thank the Linux Audio community for their great work and remarkable contribution to open-source software. We are indebted to all of you for making Linux a more than simply viable option for professional audio.

The research conducted in this publication was funded by the Irish Research Council under grant number EP-SPG/2024/1176. It is supervised by Victor Lazzarini, Gordon Delap and Jean-Michel Celerier. Many thanks to them, as well as Raphael Marczak and Simon Archipoff for their continued support!

<footnote n="10">¹⁰see libremidi’s documentation: https://celtera.github.io/libremidi</footnote>
7. REFERENCES

[1] Jean-Michael Celerier, *Authoring interactive media : a log-ical & temporal approach*, phdthesis, Université de Bordeaux, Mar. 2018.

[2] Harry Van Haaren, “OpenAV Ctrla: A Library for Tight Integration of Controllers,” in *Linux Audio Conference*, Jean MONNET University, Saint-Etienne – France, 2017, pp. 5–11.

[3] Samuel J. Hunt, “Exploring polyrhythms, polymeters, and polytempi with the universal grid sequencer framework,” in *Proceedings of the 15th International Audio Mostly Conference*, New York, NY, USA, Sept. 2020, AM ’20, pp. 101–106, Association for Computing Machinery.

[4] Beat Rossmy, “Buttons, Sliders, and Keys – A Survey on Musical Grid Interface Standards,” in *International Conference on New Interfaces for Musical Expression*, June 2022.

[5] Beat Rossmy and Alexander Wiethoff, “Musical Grid Interfaces: Past, Present, and Future Directions,” in *International Conference on New Interfaces for Musical Expression*, June 2021.

[6] Beat Rossmy, Maximilian Rauh, and Alexander Wiethoff, “Towards User Interface Guidelines for Musical Grid Interfaces,” in *International Conference on New Interfaces for Musical Expression*, June 2022.

[7] Beat Rossmy, Sebastian Unger, and Alexander Wiethoff, “TouchGrid – Combining Touch Interaction with Musical Grid Interfaces,” in *International Conference on New Interfaces for Musical Expression*, June 2021.

[8] Stefan Püst, Lena Gieseke, and Angela Brennecke, “Interaction Taxonomy for Sequencer-Based Music Performances,” in *International Conference on New Interfaces for Musical Expression*, June 2021.

[9] Jean-Michaël Celerier, “Rage Against The Glue: Beyond Run-Time Media Frameworks with Modern C++,” in *International Computer Music Conference*, Limerick, Ireland, July 2022.