# WAM-Studio: A Web-Based Digital Audio Workstation to Empower Cochlear Implant Users

Michel Buffa, Antoine Vidal-Mazuy, Lloyd May, Marco Winckler

► To cite this version:

Michel Buffa, Antoine Vidal-Mazuy, Lloyd May, Marco Winckler. WAM-Studio: A Web-Based Digital Audio Workstation to Empower Cochlear Implant Users. INTERACT 2023 - 19th IFIP TC13 International Conference, IFIP TC13, University of York, UK, Aug 2023, York, United Kingdom. pp.101-110, 10.1007/978-3-031-42280-5_6. hal-04233343

HAL Id: hal-04233343

https://inria.hal.science/hal-04233343v1

Submitted on 9 Oct 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# WAM-Studio: A Web-based Digital Audio Workstation To Empower Cochlear Implant Users

Michel Buffa¹[0000-0003-1900-0871], Antoine Vidal-Mazuy¹, Lloyd May²[0000-0003-4692-8261], and Marco Winckler¹[0000-0002-0756-6934]

¹ University Côte d'Azur, CNRS, INRIA

michel.buffa, antoine.vidal-mazuy, marco.winckler@univ-cotedazur.fr

² Stanford University

lloydmay@stanford.edu

**Abstract.** This paper introduces WAM-Studio, an online Digital Audio Workstation (DAW) for recording, mixing, producing, and playing multitrack music. WAM-Studio advances music development by proposing a web-based environment based on a visual programming paradigm of end-user programming (EUP). In this paper, we describe how users can associate individual tracks with real-time audio processing plugins that can then be customized to produce a desired audio effect. Moreover, we describe how users can visually create macros to control multiple plugin parameters at once. While programming macro controls and customizing track parameters might have many applications in the music industry, they also present an opportunity to afford Hard-of-Hearing users greater control over their music listening. To illustrate the potential of WAM-Studio, we present a case study illustrating how this tool could be used by Hard-of-Hearing users to modify individual musical elements in a multi-track listening context to create a more enjoyable listening experience.

**Keywords:** Web Audio, DAWs, plugin architecture, Web standards

## 1 Introduction

The advent of synthesizers, samplers, and sequencers completely changed the paradigm in the music creation process [4]. Computer-assisted music production is a rapidly evolving field that utilizes computers to record, edit, and produce music. Many musicians embrace the use of Digital Audio Workstations (DAWs) for creating and manipulating digital audio and MIDI content to create music.

A DAW is a feature-rich software, resulting in a notably high complexity of use. It allows musicians to create multi-track songs by: (1) using audio samples directly (e.g., by incorporating an audio file into a track or recording from a microphone or sound card input), (2) synthesizing audio using virtual instruments (e.g., a software recreation of a piano), (3) mixing various audio tracks together, and (4) applying sound effects to each track (e.g., reverb, frequency equalization, or auto-tune on vocals).
The four DAWs with the lion-share of the market (Logic Audio, Ableton, Pro Tools, Cubase)<fnref n="3" />, are all large standalone software applications that must be installed. The first online DAWs appeared in 2008, using Flash technology, while the first DAWs using HTML5 and the Web Audio API for audio processing only appeared between 2015 and 2016 [3]. Online DAWs present distinct advantages over conventional DAWs such as lower barriers to entry and the ability to easily share and access projects from any device with an internet connection. A large variety of audio plugins (a kind of software module) extends the functionality of DAWs, offering users greater flexibility and control over their music production [2]. Since 1997, a significant market has developed for third-party plugin developers offering thousands of plugins that are compatible with all major DAWs.

A DAW is a complex application. A variety of individually adjustable parameters are available on each track, including volume, stereo panning, plugins, and their associated parameters, as well as the plugin order. Finally the “master track” sums all individual tracks together and presents an additional opportunity for plugin use and parameter tweaking. Given the amount of customization and complexity present in DAWs, they can be daunting applications for many users to engage with [4]. However, previous work has shown that even basic DAW controls can effectively be used to empower cochlear implant (CI) users to customize their music listening experience [7]. This is largely due to the reduced frequency resolution and speech-focused nature of CI processing.

Previous work has illustrated the similarities between end-user programming (EUP) tools and music composition tools such as DAWs [6, 11, 15]. EUP aims to solve the mismatch between end users’ high expectations and specific domain knowledge but limited programming expertise [14]. Current practice shows that computer musicians become a kind of end-user programmer who face challenges that are similar to their professional counterparts in software engineering [8, 12]. The difference is that DAWs aim to enable end users (musicians) to design, tailor, and customize audio. We suggest that some of the difficulties might be a result of the choice of programming paradigm. The user’s ability to negotiate the constraints of the tool and assimilate its particular language is crucial in either case, whether engaging with visual metaphors or learning system-specific languages to build highly determined musical processes.

In this paper, we present a new Web-based digital audio workstation, WAM-Studio, that employs the visual programming paradigm for creating and editing multitrack audio. Section 2 introduces the main features of WAM-Studio and highlights how WAM-Studio enables users to create macro controls for adjusting many plugin parameters simultaneously (section 2.4). Section 3 describes the overall tasks orchestration, demonstrating how the macro manager of WAM-Studio helps to simplify the process of adjusting various parameters at the same time. Apart from its conventional application in the music industry, we have discovered a potential utilization of the macro control feature to empower Hard-of-Hearing users to customize their multi-track listening experience, as detailed

<footnote n="3">³ https://tinyurl.com/s4tbjzew</footnote>
in section 4. The remaining sections compare WAM-Studio (section 5) with
similar tools before presenting conclusions and future work.

## 2 WAM-Studio's Main Features

Figure 1 shows WAM-Studio's user interface. This includes multiple tracks, their associated audio waveform, and a selection of available effect plugins (located inside the window titled *WAM2 Pedalboard* on the right-hand side) that can also be uniquely assigned and configured to each track.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Overview of WAM-Studio, featuring a set of tracks (on the left-side) and the associated effect plugins (the WAM2 Pedalboard window to the right-side).</figcaption></figure>

### 2.1 Overview of tracks

A track is a container for audio-related data that comes with an interactive display of these data, editing, and processing facilities as well as basic control parameters such as volume and left/right panning. WAM-Studio supports two types of tracks: **audio tracks** and **MIDI tracks**. Audio tracks contain recorded audio, such as a vocal take, a guitar recording, or any other type of audio signal, that is generally rendered graphically as waveforms. These audio tracks can be edited, processed, and mixed by copying, cutting, and pasting audio samples in the audio buffer associated with the track. The audio track's output can be further processed by a chain of audio effects, or plugins. As the name suggests, MIDI tracks do not contain audio information but rather MIDI data (the pitch of notes, velocity, and duration), which is used to control virtual instruments such as synthesizers. Figure 2 shows an isolated audio track with the waveform display of the associated audio buffer, and the default track controls/parameters on the left side (mute/solo, volume, stereo panning).
<figure><img src="image_3.png" /><figcaption>Fig. 2. A track in WAW-Studio.</figcaption></figure>

Tracks can be added or removed from WAM-Studio, played in isolation, or with other tracks. They can also be armed for recording, allowing the armed track to record selected incoming audio while all other tracks can play along. Each track output is connected to a single “master track” where the global volume and panning of the final mix can be adjusted. It is also possible to apply audio effect plugins to the master track, allowing for a final adjustment of dynamics, frequency balancing, etc. A plugin, or multiple, can be applied to individual tracks, as shown in Figure 3, to apply audio effects or synthesize audio (in the case of virtual instruments).

All audio effects and virtual instruments are plugins in the WAM-Studio. This design gives an extensive degree of control and adaptability and enables users to blend and manipulate the sound of each track with high precision and sophistication, thus making it easier to create intricate audio productions. So that, when one presses the play button of the DAW, all the tracks are rendered simultaneously, resulting in the final output signal, i.e. “the mix”.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Selecting a track and associating it to a chain of plugins.</figcaption></figure>
## 2.2 Managing plugin chains

To apply a chain of effect plugins, WAM-Studio provides a special plugin, called *Pedalboard* [2], that acts as a central host for all other plugins, as illustrated by Figure 3. The Pedalboard connects to a plugin server that sends back the list of plugins available as a JSON array of URIs. From this list of URIs, the Pedalboard plugin retrieves the descriptors and initializes the plugins to be displayed (upper part of Fig. 3). To create a chain of effects, plugins can be moved to the bottom part of the window in Figure 3, re-ordered, and have their parameters set to create special effects.

Any configuration can be saved as a named preset (e.g. "guitar crunch 1"). Presets can be organized into banks ("rock", "funk", etc). Naming and management of banks of presets is a task of the Pedalboard plugin. The parameters exposed correspond to the entire set of parameters of the active preset, with this whole process able to be automatad [3]. When a project is saved, the state of each track is saved, along with the state of the plugin configurations.

## 2.3 Recording, dealing with latency, other features

Recording in a DAW is one of the most delicate features to implement correctly. For instance, when recording a guitar track with real-time effects plugins, and playing over drum and bass tracks, it is crucial to ensure that the latency during recording is not noticeable as the musician must be able to play comfortably. Additionally, input latency is introduced as the time between a signal being emitted from a physical instrument and the time it is actually digitally recorded is non-trivial. This time depends on the sound card, operating system, drivers, audio buffer size, etc. The recorded audio must then be shifted back in time so that during playback, it is perfectly synchronized with the other tracks. There are many strategies to deal with latency issues [3] and WAM-Studio is able to automatically select the best strategy according to the context of use.

## 2.4 Macro control for synchronous multi-track management

A macro is a customized control that allows users to adjust a variety of settings at once, making it easier to manipulate sounds and effects in real-time, as illustrated in Figure 4. A macro control refers to a way of controlling multiple parameters belonging to a plugin (effect or instrument) using a single knob, slider, or button. Therefore, when a parameter is assigned to a macro, a mapping is created that associates it to a specific range of values. When the macro is adjusted, the associated parameters change accordingly. For example, one might assign the filter cutoff, resonance, and envelope amount of a synth to a macro, so that it is possible to adjust all of these settings at once using a single knob called "timbre". Overall, macros are a powerful tool in WAM-Studio that can help streamline the workflow of sound production. WAM-Studio's macros are inspired by the system available in the *Ableton Live* DAW<fnref n="4" />.

<footnote n="4">⁴ https://www.youtube.com/watch?v=NOufylMAEA&t = 177s</footnote>
<figure><img src="image_5.png" /><figcaption>Fig. 4. Editing Macros in WAM-Studio: macro editor above the plugin chain.</figcaption></figure>

Macros can be created, loaded, and saved dynamically within the WAM-
Studio macro manager. They can be saved as presets, allowing custom settings
to be quickly recalled and reused in future projects. When a project is saved,
the current state and configuration of macros are saved as well.

3 Tasks for creating a multitrack song in a nutshell

The process of recording and mixing a new song is often iterative. Here are the
different main tasks involved:

- **Hardware and instrument set-up:** Connect the computer, sound card, instruments, MIDI controllers, etc. and calibrate latency<fnref n="5" />.

- **Create a new track:** For each audio element being recorded or played back;

- **Add plugins:** As needed on a track-by-track basis, such as audio effects for voice or virtual instrument to synthesise MIDI data, and adjust parameters;

- **Arm the track and record:** Record the desired section of audio and, if other tracks are present, recording is made while playing back other tracks in time.

- **Mix:** Adjust the volume and stereo panning of each track, and refine the sound by adjusting the plugin parameters of each track, including the master track.

All these steps are time-consuming, and require a certain amount of expertise
and knowledge about each plugin. Macros play an important role in simplifying

<footnote n="5">⁵ See at [3] details about why gear setup is required for every new hardware connected and security constraints prevent hardware discovery.</footnote>
the often complicated process of plugin parameter adjustment, especially for
users unfamiliar with audio processing. Unlike most existing DAWs, the WAM-
Studio is a web application so that projects can be shared via simple http links,
allowing remote collaborators to record and tweak settings. Therefore, macros
play an additional role as they allow collaborators to create and share macros
easily among collaborators, or even to the entire community of users.

4 Using Macro Controls To Customize Multi-track
Listening for Cochlear Implant Users

More than 1 in 5 people worldwide are D/deaf or Heard-of-Hearing (DHH) <fnref n="6" />,
with many of them experiencing a large quality of life improvements through
the use of hearing assistive technologies, such as hearing aids or cochlear im-
plants (CIs) [18]. CIs are electronic devices that convert acoustic sound signals
into electrical ones that are used to stimulate the cochlea. Currently, the inter-
nal audio processing on CIs is optimized for speech, resulting in the perceptual
experiences of music and other complex auditory stimuli varying greatly among
CI users [16, 10]. CI users experience a maximum dynamic range that is reduced
from 40-80 dB and a frequency resolution approximately 10-20 times lower than
people with traditional hearing [5, 19]. Therefore, CI users’ perceptions of cer-
tain musical features, such as rhythm and tempo, are comparable to those with
traditional hearing; however, the perception of timbral, harmonic, and melodic
information differs greatly [9].

<figure><img src="image_6.png" /><figcaption>Fig. 5. Simplified view of WAM-Studio allowing DHH users to adjust sound properties.</figcaption></figure>

Whilst the target users of WAM-Studio are musicians, we have found that
macro controls could be useful for Hard-of-Hearing (HH) users. 10 CI users pro-

<footnote n="6">⁶ DHH is an expansive term for people with hearing loss or are otherwise aurally diverse, including those who identify as culturally Deaf and may use a signed language as their primary language.</footnote>
vided feedback and rated various mixes using a mix of qualitative and quantita-
tive metrics, allowing us to determine mixing strategies and plug-in combinations
that were often used in highly-rated mixes. Using WAM-Studio's macro controls
to adjust certain sound parameters may increase CI user's enjoyment of listening
to recorded music. In order to make the use of controls more easier to users, we
have created a simplified view of macro controllers for HH users shown in as
shown in Figure 5. The first simplification is the replacement of the audio buffer
waveform associated with the track by relevant macro sliders.

As we shall see in Figure 5, the macro controllers are generated through
previous research and given non-technical labels, such as “clarity”, “punch”, or
“weight”, allowing users to granularly adjust multiple settings of a plug-in chain
using a single slider. A series of macros have been developed to be used on
multiple instruments of varying genres. This includes macros tailored to both
genre and instrumentation, for example:

- **“Clarity (Pop Vocal)”**: Increasing this macro increases the amount of 2kHz and 5kHz present in the signal, increases the wet/dry mix of a de-esser, and adds a medium-attack, slow-release compressor.

- **“Punch (Rock Drums)”**: Increases the compression ratio and wet/dry mix of a compressor with medium-fast attack and fast release, increasing the wet/dry mix on a sub-octave pitch-shifter, as well as the wet/dry mix on a saturation plug-in.

- **“Shine (Country Guitar)”**: This reduces the amount of sub-250 Hz information, increases the wet/dry mix on a one-octave-above pitch-shifter, and increases the ratio and wet/dry on a slow-attack slow-release compressor.

Several plugins have been developed specifically for HH users, such as an
octaver and a tracking-EQ which boosts the fundamental frequency of the sig-
nal. The macros controls for HH users were created by the research team and
advanced users using the WAM-plugin macro creator. The macro created can be
exported for use in the simplified view that is aimed to reach a large audience
of CI's users.

**5 Related work**

In [17] a comparison of an online DAW (*Soundtrap*) with two native-based so-
lutions (*Avid Cloud Collaboration* and *VST Transit*) showed that web-based
DAWs “have the potential for widespread adoption and may even surpass the
usage of the existing paradigms in professional audio mixing practice in future”.
Additionally, the use of a custom DAW for empowering HH users to customize
multitrack listening is an original contribution to the field. Current approaches
to enhance music enjoyment for CI users include adjustments to the internal
signal processing on the CI itself, the creation of music composed specifically for
CI users, and algorithmic pre-processing [13]. These approaches certainly have
merits, but they fail to recognize the enormous diversity and variance in audi-
tory perception and aesthetic preferences among CI users. Additionally, these
processes assume a passive CI listener with a limited desire to play an active
role in their listening experience.

WAM-Studio is an ongoing work, nevertheless, it is the only web-based DAW
that features a macro system, that is open source, and that supports third-party-
plugins. A survey of online DAWs can be found in [3] and a comparison with
collaborative solutions based on native DAWs in [17].

6 Discussion and future work

In this paper, we have presented the design details and various features of WAM-
Studio. Whilst the target users of WAM-Studio are musicians, we have found
that some of its features (in particular the macro controls for multi-track man-
agement) could be useful for HH users. We illustrated how some macros can be
created to help to adjust multitrack music using a simplified view of the WAM-
Studio. This unexpected use of WAM-Studio presents an exciting opportunity to
research questions of practical and theoretical importance about the uses of tools
for creating sounds and customizing listening experiences. Of particular interest
are CI users, as this user population is largely under-designed for, despite there
being over 736K registered CI users (as of December 2019) [1]. CI users are
additionally extremely heterogenous, potentially being well served by the level
of customization and personalization offered by WAM-Studio.

It is worthy of notice that all tools described in this paper use recent Web
technologies: W3C APIs WebAudio, WebMidi, Web Components, WebAssembly
and have been developed as an open source demonstrator of what can be done
on the web today in real time audio processing. WAM-Studio is readily available
at http://annonymousURL.

Future work should include user testing with both versions of the tools in-
cluding musicians and Hard-of-Hearing users, as well as additional focused user-
testing and co-design sessions with CI users to adjust the mapping of various
macros and further tailor the UI to increase the ease of use of the customized
multitrack music player.

References

1. Nidcd fact sheet: Cochlear implants. National Institute of Deafness and Other Communication Disorders (Mar 2021)

2. Buffa, M., Kouyoumdjian, P., Beauchet, Q., Forner, Y., Marynowic, M.: Making a guitar rack plugin -WebAudio Modules 2.0. In: Web Audio Conference 2022. Cannes, France (Jul 2022), https://hal.inria.fr/hal-03812948

3. Buffa, M., Vidal-Mazuy, A.: Wam-studio, a digital audio workstation (daw) for the web. In: In The Web Conference 2023 - DevTrack. Austin, Texas, USA (2023)

4. Burlet, G., Hindle, A.: An empirical study of end-user programmers in the computer music community. In: 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. pp. 292–302 (2015). https://doi.org/10.1109/MSR.2015.34

5. Hartmann, R., Kral, A.: Central responses to electrical stimulation. In: Cochlear implants: Auditory prostheses and electric hearing, pp. 213–285. Springer (2004)
6. Hillerson, T.: In: Programming Sound with Pure Data. p. 196. The Pragmatic Bookshelf (2014)

7. Hwa, T.P., Tian, L.L., Caruana, F., Chun, M., Mancuso, D., Cellum, I.P., Lalwani, A.K.: Novel web-based music re-engineering software for enhancement of music enjoyment among cochlear implantees. Otology & Neurotology 42(9), 1347–1354 (2021)

8. Ko, A.J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaf-fidi, C., Lawrance, J., Lieberman, H., Myers, B., Rosson, M.B., Rothermel, G., Shaw, M., Wiedenbeck, S.: The state of the art in end-user software engineering. ACM Comput. Surv. 43(3) (apr 2011). https://doi.org/10.1145/1922649.1922658, https://doi.org/10.1145/1922649.1922658

9. Limb, C.J., Roy, A.T.: Technological, biological, and acoustical constraints to music perception in cochlear implant users. Hearing research 308, 13–26 (2014)

10. Maarefvand, M., Marozeau, J., Blamey, P.J.: A cochlear implant user with exceptional musical hearing ability. International journal of audiology 52(6), 424–432 (2013)

11. Nash, C.: Manhattan: End-user programming for music. In: Proceedings of the International Conference on New Interfaces for Musical Expression. pp. 221–226. Zenodo (Jun 2014). https://doi.org/10.5281/zenodo.1178891, https://doi.org/10.5281/zenodo.1178891

12. Nishino, H.: Cognitive issues in computer music programming. In: Proceedings of the International Conference on New Interfaces for Musical Expression. pp. 499–502. Zenodo (Jun 2011). https://doi.org/10.5281/zenodo.1178123, https://doi.org/10.5281/zenodo.1178123

13. Nogueira, W., Nagathil, A., Martin, R.: Making music more accessible for cochlear implant listeners: recent developments. IEEE Signal Processing Magazine 36(1), 115–127 (2018)

14. Paternò, F., Wulf, V. (eds.): New Perspectives in End-User Development. Springer International Publishing (2017). https://doi.org/10.1007/978-3-319-60291-2, https://doi.org/10.1007/978-3-319-60291-2

15. Sarkar, A.: In: Proceedings of the 27th Annual Conference of the Psychology of Programming Interest Group (PPIG 2016)

16. Spangmose, S., Hjortkjær, J., Marozeau, J.: Perception of musical tension in cochlear implant listeners. Frontiers in neuroscience 13, 987 (2019)

17. Stickland, S., Athauda, R., Scott, N.: A new audio mixing paradigm: evaluation from professional practitioners’ perspectives. Creative Industries Journal pp. 1–49 (2022)

18. WorldHealthOrganization: Deafness and hearing loss fact sheet (Feb 2023), https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss

19. Zeng, F.G., Tang, Q., Lu, T.: Abnormal pitch perception produced by cochlear implant stimulation. PloS one 9(2), e88662 (2014)