# Brain-Body Digital Musical Instrument Work-in-Progress

Atau Tanaka, Anne Sèdes, Alain Bonardi, Stephen Whitmarsh, David Fierro, Francesco Di Maggio

► To cite this version:

Atau Tanaka, Anne Sèdes, Alain Bonardi, Stephen Whitmarsh, David Fierro, et al. Brain-Body Digital Musical Instrument Work-in-Progress. ISEA 2023 - 28th International Symposium on Electronic Art, Le Cube Garges; Ecole Nationale Supérieure des Arts Décoratifs; Forum des Images, May 2023, Paris, France. hal-04259596

HAL Id: hal-04259596

https://hal.science/hal-04259596v1

Submitted on 26 Oct 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Brain-Body Digital Musical Instrument Work-in-Progress

Atau Tanaka, Anne Sèdes, Alain Bonardi, Stephen Whitmarsh, David Fierro, Francesco Di Maggio

Maison des Sciences de l'Homme Paris Nord

CICM/MUSIDANSE Université Paris 8

Institut du Cerveau et de la Moelle épinière

Paris, FRANCE

{atau.tanaka, anne.sedes, alain.bonardi, stephen.whitmarsh, david.fierro, francesco.dimaggio}@mshparisnord.fr

## Abstract

The Brain/Body Digital Musical Instrument (BBDMI) will create a prototype for a digital musical instrument system that uses physiological signals from the human body: from the brain and muscles. The instrument system will be validated in a range of musical settings from concerts to the conservatoire, with a diverse range of musicians. The project brings together an interdisciplinary team of researchers, designers, musicians and engineers that will mutually inform each other through a mixed methods approach. The consortium represents the spectrum of research: from low-level technical development to innovative user-centered design, and the integration of state-of-the-art methods integrating neuroscience and musical practice. The project will hold a workshop and concert during the ISEA2023 affiliated conference, the Journées d'Informatique Musicale (JIM) 2023 at the MSH Paris Nord, the week following ISEA, 24 May.

## Keywords

Electroencephalogram, electromyogram, digital musical instrument (DMI), brain-computer interface (BCI)

## Introduction

The use of physiological signals from the human brain and body in human-machine interaction has come of age, constituting the research field of physiological computing. Advances in microelectronics have enabled low-cost consumer brain-computer interfaces (BCIs) and muscle interfaces for gaming and virtual reality applications. Musicians have been at the forefront of the development of digital musical instruments (DMIs) that adopt physiological interfaces, but their efforts can be isolated from mainstream scientific and musical communities, limiting how knowledge and practice inform wider art, music and science interests [2,5]. BBDMI seeks to address this gap.

In 1934, the neurophysiologist and Nobel laureate E. D. Adrian, together with B. H. C. Matthews, first sonified the human EEG, discovered years earlier by Hans Berger (1929), into audio signals. In 1965, the American composer

Alvin Lucier, together with engineer Edwards Dewan, created the first musical work for live brain activity and percussion instruments. With the advent of electronic synthesizers in the 1970s, experimental musicians such as David Rosenboom used EEG to explore musical compositions [10]. Well known performance artists Laurie Anderson and Pamela Z have used the Bodysynth system in the 1990s to integrate electromyogram (EMG) interaction into their multimedia stage performances[4]. Yoichi Nagahima has used EMG to extend the performance of Japanese traditional music and interactive media art [6].

## The Project

The Brain/Body Digital Musical Instrument (BBDMI) project will create a digital musical instrument system using physiological signals from the human body – the electromyogram (EMG) from muscle contraction and the electroencephalogram (EEG) from brain activity. It combines musical organology with biomedical technologies in a modular and accessible manner to make a system that will be useful for electronic artists, musicians, and educators. Its development methodology is based on iterative feedback between creative practice, design and research activities. The instrument system will be validated in a range of musical settings from concerts with a diverse range of musicians.

The consortium is led by la Maison de Sciences de l'Homme Paris Nord (MSH), with as industry partner the company, Soixante Circuits, and as partners, Paris Brain Institute (ICM), and the Centre de Recherche Informatique et Création Musicale (CICM/MUSIDANSE) Université Paris 8. The consortium represents the spectrum of research: from low-level technical development to innovative user-centered design, and the integration of state-of-the-art methods integrating neuroscience and musical practice.

## Project Objectives

The objectives of the BBDMI project are threefold, to:
1. Create a musical instrument hybridising EMG and EEG
2. Make accessible our technologies, through open source, open hardware, and Open Science ethos

3. Develop user centered methodologies that engage with expert musicians, novices, in a context of neurodiversity

## Technical: EEG and EMG

Electroencephalography (EEG) is the recording of the electrical activity of the brain from the scalp, measured in microvolts (µV). Brain oscillations in the alpha frequency (~10Hz) are remarkably stable over time and across different testing situations, promoting its usefulness in brain-computer interfaces (BCIs) [8]. Alpha oscillations reflect one of several attentional mechanisms in the brain and its amplitude correlates with our subjective experience of attention [15].

Electromyography (EMG) is the recording of muscle activity and correlates with the strength of muscle contraction, even in the absence of overt movement. Saponas has demonstrated that EMG can find practical use in HCI applications [11]. We have shown that EMG can be associated with a user's sense of physical effort [1].

Interface systems based on either brain or muscle signals have limitations when treated separately. Brain-based systems suffer from small signal amplitudes and low accuracy, interference and noise, and the variability between the users' brains that require adapting to individual signal features. Muscle-based interfaces suffer from subject fatigue and spasticity, especially in users that have less motor control (but who may benefit most). By combining EEG and EMG, their individual strengths become complementary: while muscle activity can be fast and accurate, EEG signals can indicate fluctuations of attention; EMG can provide an index of physical effort, while EEG can provide an index of cognitive effort. We describe this hybridization of EEG and EMG by adopting the term, ExG.

## EAVI ExG Hardware

We have developed a biosignal hardware prototype, the EAVI ExG that is designed from the ground up as a musical instrument device. It is based on the Texas Instruments ADS129x, a single chip solution for biosignal amplification and digitization. This front end is paired with a STM32F427 Cortex-M4 microcontroller. The board communicates with over Bluetooth LE 4.2 and also USB where it registers with the host as a class compliant audio and MIDI device.

Embedded signal processing is implemented using the OWL framework [14] and combines biosignal and audio signal processing into a single DSP pipeline. The OWL platform offers tools to develop “patches” in C++, or using FAUST [7], or Pure Data [9] music programming environments.

## Software

We have adopted a multi-strand software development strategy. One goal is to see how much signal processing can be run in the OWL firmware directly on the EAVI board, making it a standalone biosignal music synthesizer. For heavier tasks, functionality is divided between the EAVI and software running on a host computer. In this scenario, the EAVI handles biosignal pre-processing and feature extraction. By using FAUST, we have a multi-platform software development pipeline where sound synthesis can be prototyped on a range of music host environments (including Max, VST), then be embedded in the EAVI, microcontroller power permitting.

On the computer side, the multi-platform strategy includes, in addition to FAUST, Python and Cycling'74 Max. This includes, for example porting earlier work on EEG sonification developed in Python<fnref n="1" />.

We propose a modular architecture that allows users to flexibly interchange bespoke sound and modulation stages for creating multimodal instruments with a range of different devices. Modules deal with both lists of numeric data and multichannel audio signals, depending on the phase of data processing and the specifics of the acquisition device. The library consists of the following module types: *input*, *control processing*, *feature extraction*, *machine learning*, *sound synthesis*, *utilities* and *output*. The architecture follows a node-based paradigm, nesting multiple streams of data into single connectors to simplify patching for the end-user. We combine this with a novel graphic user-interface (GUI) design where the node blocks can expose their control parameters for direct on-screen manipulation.

In FAUST we have implemented a range of signal processing tasks, including utilities such as an EMG/EEG signal simulation and an instrumental noise generator. We have also developed a new powerful live audio granulator that can be instantiated in multichannel and be controlled by neural network-based regression models.

Multichannel output from synthesis modules are treated by spatial processing using abclib<fnref n="2" /> [3] and HOA[13] in a way where spatialization is not a separate, post-production step, but where immersive elements like ambisonic decorrelation become a core part of the synthesis itself. We call this *spatial synthesis* [12].

## Users and Events

BBDMI events so far have consisted of four public workshops, three user studies, and two concerts.

## Workshops

The first workshop took place on 19-20 January 2022 and served as a project kick-off event. It brought together research team members, members of the advisory board, and

<footnote n="1">¹ https://github.com/eegsynth/eegsynth www.EEGsynth.org</footnote>

<footnote n="2">² https://github.com/alainbonardi/abclib</footnote>
project partners. The workshop program consisted of sessions on technical overviews, summaries of existing artistic practices, and demos. The technical program included overviews of existing EEG systems, of sound synthesis strategies using FAUST and the OWL framework, and industrial design.

The second workshop took place on 23 March. It was presented by BBDMI team members, with an audience of university students from Paris and London. Invited experts included: a computational artist/performer, and a medical doctor working in motor rehabilitation. The program was organized in the form of three ateliers: 1) Introduction to machine learning, 2) Comparison of the commercial Myo system and EAVI for EMG, and 3) Live demonstration of EEG sound mapping with volunteers from the audience.

The third and fourth workshops were public events. One was as part of the Journées Européen du Patrimoine (European Days of Patrimony) on 17 September, and the other the CNRS<fnref n="3" /> open labs during the Fêtes des Sciences (Festival of Science) on 7 October. In both cases, members of the public visited the sound studio at the MSH Paris Nord, a facility with an 8 channel, 7 order ambisonic sound system. Groups of ~15 people took part in each session. After an introduction to the project, we asked for volunteers to try out our platform. Two volunteers were connected to EMG, and one volunteer a commercial EEG headset (Mentalab Explore<fnref n="4" />). A sequence of exercises of increasing complexity and interactivity then ensued:

* Sonification of raw EMG

* Modulation of pitch by EEG alpha activity

* Direct modulation of subtractive synthesis and sound spatialization

* Mixed modulation using a neural network to combine all volunteers into a single controller

* The addition of a granular synthesizer

* Live audio input in the style of *mixed music*

We learned to give the audience members sufficient time to explore the sonic landscape and their connection between sound and muscle or brain activity.

## User Studies

We carried out three weeklong user studies adopting a common protocol with diverse musicians: one, an autistic guitar student, a virtuoso cellist (Fig. 1), a guitarist who performs electro-acoustic music. Each study consisted of:

* Introduction to EEG and EMG and musical applications

* Presentation of the hardware including electrodes for EMG, and electrode cap for EEG

* Sound synthesis techniques in studio

The empirical work consisted of trying different electrode positions to capture gestures made while playing their instrument. We wanted to study each musician's existing instrumental practice, with pieces they were playing, and musical projects they were working on.

The musical gesture captured by EMG allowed subtle interactions with spatialized granular synthesis. The

affordance of EMG captured the intention of the musician as they prepared a gesture or musical phrase. To our surprise, we observed a focus on the slowness of gesture, where subtle gestures mixed anticipation and listening in the corporeal act. This led to the idea that it could be written or transcribed as a kind of musical choreography. They were all, despite their differing levels of expertise, able to adopt and incorporate this interaction in their playing technique.

While the EMG worked well with gestural articulation and transitory movement, the EEG functioned at another time scale, one that is much slower and dilated in time. This took the musician outside of a control paradigm, but nonetheless kept them in a live musical interaction.

<figure><img src="image_2.png" /><figcaption>Figure 1. Cellist Guilherme Carvalho with the EAVI board by his write, and with EMG electrodes on the upper arms and shoulder.</figcaption></figure>

## Concerts

Tanaka and Whitmarsh gave a network performance during ICLI 2022 (June 20-23) as an EEG/EMG duo. Tanaka performed EMG in London with his audio streamed Whitmarsh in Paris using JackTrip. At the Paris site, the audio was processed through an analogue modular synthesizer. At the same time, EMG features were sent over the UDP as Open Sound Control message and patched to the modular synthesizer. Whitmarsh recorded his EEG in real-time with the Mentalab Explore, using the EEGsynth to extract EEG features (oscillatory band power), sent as control signals to the modular synthesizer. Part of the patch in turn modulated Tanaka's audio stream. The resultant mix was streamed to ICLI in Lisbon over Zoom.

We presented a concert on 8 December at the MSH where BBDMI team member Tanaka performed with the brutpop artist Cicanoise (Robin Dussurget), both using the EAVI

<footnote n="3">³Centre national de la recherche scientifique</footnote>

<footnote n="4">⁴https://mentalab.com/</footnote>
EMG. Dussurget is an autistic musician and has lived with
motor control issues of the lower body since birth and is in
a wheelchair. He performs noise music with modular
synthesizers. Cicanoise and Tanaka performed a duo using
software modules developed (above) by Di Maggio and
Fierro. Cicanoise controlled his analogue synth oscillators
using concentrated tension, making fists and jabs, while
Tanaka played an EMG homage to the Theremin followed
by a spatial-granular deconstruction of rock tracks sampled
from vinyl. The concert took in the context of the Soigner
en Temps de Crise (Healing in Times of Crisis) colloquium
at the MSH Paris Nord. Insights drawn from the rehearsals
and the performance enter the user study cycle by informing
continuing collaboration and subsequent system design.

JIM workshop/concert

We will present BBDMI next on 24 May at the ISEA2023 affiliated event, *the Jounées d'Informatique Musicale* (JIM) conference. The workshop will include:

* Presentation of the EAVI ExG prototype

* Presentation of the BBDMI software suite

* Experimentation with modular electronic music and mixed music with acoustic instrumentalists

We will present the next BBDMI concert, this time
including EEG with partipation from other team members.

Conclusion and Future work

This paper summarizes the first-year activity for the BBDMI
project as work-in-progress and announces the next
workshop and concert in the context of JIM2023. The
project continues for two and a half more years, into 2025.
Planned activities for Year 2 of the project include:

* Release of the software system as open source

* Study of dry electrode technology

* Ergonomic design of electrode harnesses

* Hardware design strategy in view of supply chain issues

* Integration into modular synthesis racks

A technical study of EMG is ongoing, with a review of
surface electrode technologies. This will lead to design
studies with partner Soixante Circuits on ergonomic designs
that will allow electrode placement not just on the arms but
on different parts of the body including the upper arms,
shoulder, legs, and back. Our initial user study results will
guide these design studies. They will enable the project to
accommodate non-normative bodies and uses.

We will build upon the user studies to roll out a series
with students from the St-Denis Conservatory and the
University of Paris 8 during their introductory courses and
ateliers on electronic and mixed music.

Finally, the user base will grow as we roll out the
hardware to a group of beta testers. These include Jose
Corredera, for hospital trials in EMG driven musical
sonification of motor rehabilitation exercises. Goldsmiths
PhD student Lucy Strauss will use the EAVI EMG to
monitor viola bowing position and apply music movement
exercises including the Tuttle technique to the design and
development of an interactive music partner. Finally,

Electronic musician and PhD student Fabi Pereira will present a first experiment in an embodied EMG based electronic music system in a lecture/performance in Lisbon in January 2023. We will continue working with users in pairings of composer and performer, including young music students and in contexts of neurodiversity.

Acknowledgements

The project has been supported by funding from the Agence Nationale de la Recherche (ANR CE38-21). We recognise expertise brought to the project by synthesizer designer Martin Klang of Rebel Technologies for EAVI hardware design (Horizon2020 ERC 789825), and external advisor Robert Oostenveld. We would like to thank participants Laurie Pompidour, Guilherme Carvalho, Amélia Mazarico.

References

1. Baptiste Caramiaux, M. Donnarumma, and A. Tanaka. "Understanding Gesture Expressivity through Muscle Sensing". *ACM TOCHI* (2015) 21, 6: 31.

2. Marco Donnarumma. 2012. Biotechnological Performance Practice. *eContact!* 14, 2. Retrieved May 17, 2017 from http://www.econtact.ca/14_2/index.html

3. Paul Goutmann and A. Bonardi. "Approaching Spatial Audio Processing by Means of Decorrelation and Ring Modulation in Ambisonics-Sound 2". *SMC 2022*.

4. Amelia Mason. "Pamela Z Sculpts Musical Abstraction From The Sounds (And Sights) Of Daily Life". *Arts at MIT*. (2016). Retrieved August 8, 2022 from https://arts.mit.edu/pamela-z-sculpts-musical-abstraction-sounds-sights-daily-life/

5. Eduardo Reck Miranda and J. Castet (eds.). 2014 *Guide to Brain-Computer Music Interfacing*. Springer.

6. Yoichi Nagashima. 2003. "Bio-sensing systems and bio-feedback systems for interactive media arts." *Proc NIME*.

7. Yann Orlarey, D. Fober, and S. Letz. 2009. "FAUST: an Efficient Functional Approach to DSP Programming". In *New Computational Paradigms For Computer Music*.

8. Gert Pfurtscheller and C. Neuper. 2006. "Future prospects of ERD/ERS in the context of brain–computer interface (BCI) developments." *Progress in Brain Research* 159: 433–437.

9. Miller S. Puckette. 1997. Pure Data. In Proc ICMC.

10. David Rosenboom. 1977. *Biofeedback and the arts: results of early experiments*. Aesthetic Research Center.

11. T. Scott Saponas, D. S. Tan, D. Morris, J. Turner, and J. A. Landay. 2010. "Making Muscle-computer Interfaces More Practical". In Proc ACM-CHI.

12. Anne Sèdes. 2015. "Approche musicale de la décorrélation microtemporelle dans la bibliothèque HOA". In Proc JIM.

13. Anne Sèdes, P. Guillot, and E. Paris. 2014. "The HOA library, review and prospects". In Proc ICMC.

14. Thomas Webster, G. LeNost, and M. Klang. 2014. "The OWL Programmable Stage Effects Pedal: Revising the Concept of the Onstage Computer for Live Music Performance." In NIME.

15. Stephen Whitmarsh, H. Barendregt, J-M. Schoffelen, and O. Jensen. "Metacognitive awareness of covert somatosensory attention corresponds to contralateral alpha power." *NeuroImage* (2014)85: 803–809.