# WAM Jam Party: Using Web Audio Modules in the Musical Metaverse

Michel Buffa, Ayoub Hofr, Dorian Girard

<figure><img src="image_1.png" /></figure>

## To cite this version:

Michel Buffa, Ayoub Hofr, Dorian Girard. WAM Jam Party: Using Web Audio Modules in the Musical Metaverse. IS2 2024 - IEEE International Symposium on the Internet of Sounds 2024 / 1st IEEE International Workshop on the Musical Metaverse (IEEE IWMM), Sep 2024, Erlangen, Germany. hal-04695252

HAL Id: hal-04695252

https://inria.hal.science/hal-04695252v1

Submitted on 12 Sep 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# WAM Jam Party: Using Web Audio Modules in the Musical Metaverse

Michel Buffa

Université Côte d'Azur,
I3S, CNRS, INRIA

Sophia-Antipolis, France

michel.buffa@univ-cotedazur.fr

Dorian Girard

Université Côte d'Azur,
Sophia-Antipolis, France

dorian.girard@etu.unice.fr

Ayoub Hofr

Université Côte d'Azur,
I3S, CNRS, INRIA, EMSI

Sophia-Antipolis, France
ayoub.hofr@inria.fr

**Abstract**—In 2020, the Web Audio plug-in standard, known as "Web Audio Modules V2" or "WAM," made a significant impact on the world of computer music on the web. These modules facilitate the creation of host applications that leverage reusable web components, such as note generators, virtual instruments, and audio effects. The WAM ecosystem now boasts over a hundred plugins of various types, supported by the WAM Community initiative, online tutorials, and multiple host applications. Sequencer Party, a real-time collaborative audio/visual platform built entirely from WAMs, designed for creating collaboratively music installations, exemplifies this ecosystem. We propose to demonstrate the first prototype of a new multiparticipant WebXR application that transposes the concepts of Sequencer Party into a persistent immersive world. It demonstrates the reuse of existing WAMs without modification by automatically generating 3D graphical interfaces suitable for VR/XR manipulation or by manually recreating dedicated interfaces. This highlights the potential of the WAM standard beyond traditional 2D web applications. For the 3D rendering of the virtual world, the BabylonJS framework was utilized, offering robust support for WebXR. A CRDT approach was chosen for state synchronization between clients. The software is open source and hosted online<fnref n="1" />.

**Keywords**— Musical Metaverse, WebXR, WebAudio, WebMidi, Web Audio Plugins, Web standards, Web Audio Modules, CRDT.

## I. INTRODUCTION

The Musical Metaverse (MM) [1] is an immersive virtual space dedicated to musical activities, expanding on the broader concept of the Metaverse. It has recently gained popularity, reigniting interest in shared virtual environments within the realm of computer music. Applications include virtual concerts, educational tools, and collaborative musical performances. Transposing user experiences into conventional 2D applications generally does not work in an immersive environment, and new ergonomic and sensory approaches need to be employed [2].

The WAM application presented in this paper focuses on a persistent, real-time multi-participant immersive world for shared music creation, exploiting recent W3C web standards such as Web Audio, Web Midi, WebXR, WebGL, WebGPU, WebAssembly, WebSockets, now implemented in the web browsers of the most common VR/XR headsets available on the market.

## II. WEB AUDIO, WEB MIDI AND WEB AUDIO

### The Web Audio API and the Web Audio Modules (WAM) plugin standard

With the Web Audio API, developers can assemble nodes into a graph to create complex audio effects and instruments like delay effects, chorus, and synthesizers. Over the years, many high-level audio effects and instruments have been developed this way. However, chaining effects and instruments, as needed in music production, requires a higher-level standard. In 2015, Jari Kleimola and Olivier Larkin proposed the "Web Audio Modules" (WAM) standard for Web Audio plugins [3]. This standard was expanded in 2018 [4] and further evolved into Web Audio Modules version 2.0 in 2021 [5]. Dozens of WAM plugins are now available through the WAM Community Endpoint, comparable in quality and complexity to native applications [6]. Available WAMs (more than one hundred) include *note generators* such as: a piano roll, a programmable step, random note generators, chord generators, *virtual instruments*: synthesizers samplers, physical modeling of instruments (flute, clarinet, brass, djembe), audio effects (including all classics: flanger, chorus, reverb, distortion, fuzz, overdrive, etc.).

## III. WAM JAM PARTY

Soon after the first publications on the musical metaverse and its eponymous research project [1], the idea of WAM Jam Party was born: a multi participant immersive application in which users could assemble collaboratively 3D WAM plugins and build music installations altogether.

The first prototype of WAM Jam Party uses WAM introspection to build 3D interactive GUIs from existing 2D WAM plugins, BabylonJS for the 3D rendering of the immersive world, and a CRDT algorithm is used synchronizing incrementally the states of all the elements between clients. While client states are updated 3 times/s, the server saves seamlessly the ongoing session into a file every second, making the world persistent. Special care has been taken for generating the 3D representation of WAM plugins, using an optional JSON file for the mapping of the internal parameters into the 3D world. Each WAM is represented by default as a box with牵 drogge cylinders, boxes or push buttons for its parameters (Figure 1).

<footnote n="1">¹ https://musical-metaverse.onrender.com/ source code
https://github.com/Doorii4N/musical-multiverse-vr</footnote>
<figure><img src="image_3.png" /><figcaption>Figure 1: 3D representation of the Big Muff WAM.</figcaption></figure>

A menu proposes a large set of WAMs enabling users to add elements in the 3D world (Figure 2). In addition, a green and a right sphere are located on the sides of the box, enabling users to connect WAMs together to form an audio graph (Figure 3). Connect a step sequencer to a synthesizer to some audio effects and you have a first basic music installation. Any element can be moved, oriented, or have its parameters adjusted using different controllers (see Figure 4).

<figure><img src="image_4.png" /><figcaption>Figure 2: main menu for adding elements in the 3D scene.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 3: Example of a graph made of two note generators (step sequencer on the left), two instruments and audio effects.</figcaption></figure>

In a multi participant session, each user can perform the same interactions. The sound produced by each installation is spatialized and changes as you move your avatar in the 3D world.

Each user in the virtual environment is represented by a simple avatar consisting of a body, head, and two hands. The avatar's head orientation is directly mapped to the user's VR headset orientation, ensuring that the avatar's gaze direction aligns with the user's actual view within the virtual environment. The avatar's hands are positioned and oriented to match the user's controller movements. This visual

representation provides a clear indication of the user's current actions, such as selecting WAMs or manipulating parameters.

<figure><img src="image_6.png" /><figcaption>Figure 4: Example of highlighting a parameter.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 5: A player's avatar.</figcaption></figure>

## ACKNOWLEDGMENTS

We would like to thank Antoine Vidal-Mazuy for his investment in WAM-Studio over the course of 2023 (he was its main coder and designer) and the team behind the Web Audio Modules, without which this DAW would never have existed.

This work has been supported by the French government, through the France 2030 investment plan managed by the *Agence Nationale de la Recherche*, as part of the "UCA DS4H" project, reference ANR-17-EURE-0004

## REFERENCES

[1] L. Turchet. "Musical Metaverse: vision, opportunities, and challenges." Personal and Ubiquitous Computing 27: pp 811-1827. 2023.

[2] A. Boem, and L. Turchet. "Musical Metaverse Playgrounds: exploring the design of shared virtual sonic experiences on web browsers.", 4th International Symposium on the Internet of Sounds. IEEE, 2023.

[3] J. Kleimola and O. Larkin. Web Audio Modules. In Proceedings of the Sound and Music Computing Conference 2015.

[4] M. Buffa, J. Lebrun, J. Kleimola, O. Larkin, and S. Letz. 2018. Towards an open Web Audio plugin standard. In WWW2018 – TheWebConf 2018 : The Web Conference, 27th International World Wide Web Conference. Lyon, France. 2018.

[5] M. Buffa, S. Ren, O. Campbell, J. Kleimola, O. Larkin, and T. Burns. 2022. Web Audio Modules 2.0 : An Open Web Audio Plugin Standard. In WWW '22 : The ACM Web Conference. 2022. ACM, France, pp 364–369. 2022 https://doi.org/10.1145/3487553.3524225

[6] M. Buffa, S. Ren, T. Burns, A. Vidal-Mazuy & S. Letz. Evolution of the Web Audio Modules Ecosystem. In proceedings of the Web Audio Conference 2024. https://zenodo.org/doi/10.5281/zenodo.10825646