# Somax2 – A Distributed Co-Creative System for Human-Machine Co-Improvisation

Marco Fiorini, Mikhail Malt

► To cite this version:

Marco Fiorini, Mikhail Malt. Somax2 – A Distributed Co-Creative System for Human-Machine Co-Improvisation. Proceedings of the Second International Conference on Hybrid Human-Artificial Intelligence, Jun 2023, Munich, Germany. 10.3233/FAIA230106. hal-04444997

HAL Id: hal-04444997

https://hal.science/hal-04444997v1

Submitted on 7 Feb 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
Somax2 – A Distributed Co-Creative
System for Human-Machine Co-
Improvisation

Marco FIORINI, Mikhail MALT

STMS Lab:IRCAM - CNRS - Sorbonne University

Marco Fiorini https://orcid.org/0009-0004-0523-2481,

Mikhail Malt https://orcid.org/0000-0003-2859-3445

**Abstract.** Somax2 is a multi-agent interactive system, based on machine-listening, machine learning and generative units, performing live machine co-improvisation with musicians. The system is trained on any musical materials chosen by the user (Audio or MIDI), effectively constructing a generative model, a corpus, from which it draws its musical “knowledge”. Listening to and adapting to a musician, that activates dynamic profiles, Somax2 provides stylistically coherent improvisation in real-time. As part of the REACH project, defining co-creativity as distributed agency between human and machine, Somax2 is indeed a good example of a distributed co-creative system, where multimodal cross-feedback affects collective behavior and increases player’s engagement through emotion and motivation.

**Keywords.** Machine learning, Co-improvisation, Audio Features, Co-creativity, A.I., Interaction Paradigms, Multi-agent systems

1. Introduction

Somax2 [1] is a multi-agent interactive system performing live machine co-
improvisation with musicians, based on machine-listening, machine learning and
generative units. The current version [2] is a recent development and algorithms
improvement from the former Somax version and previous work in RepMus team [3, 4,
5, 7, 8]. It has been totally rewritten from (never released) Somax, one of the multiple
descendants of the well-known Omax [3] developed in the Music Representation team
over the years and offers now a powerful and reliable environment for co-improvisation,
composition, installations, etc.

2. Somax2 Model and Implementation

Somax2 is one of the improvisation systems (Omax, DICY2, Improtech, Djazz) descending from the well-known Omax paradigm (learn/model/listen/generate) [3], presented here in a totally new implementation. Somax2, as its siblings, is conceived to be a co-creative partner in the improvisational process, by providing distributed agency between human and machine [1].
The system is initially trained on any musical materials chosen by the user (Audio or MIDI), effectively constructing a generative model, called a corpus.

After the first “learn” and “model” stages, the “listen” and “generate” phases are based on a navigation model, consisting of a dynamic multilayer n-gram matching and selection scheme, applying a short-term harmonic and melodic memory model [1].

When a musician interacts with the system, a similar process of segmentation and multilayer analysis is computed in real-time on the input stream. At each point in time, this is matched to the information in the navigation model, generating activations, or peaks, at certain points in the sequential memory, where the input corresponds to the model [1]. Memory and live musical data are matched through a representation space (latent space) resulting from one-off training over a large set of music pieces spanning four centuries and capturing textural relationships based on chroma analysis. Original latent space was a SOM (Self Organizing Map) and experiments have been carried with a VAE (Variational Autoencoder) showing a slightly better performance when compared with a Tonnetz as ground truth [6]. Thus, agents provide stylistically coherent improvisations based on learned musical knowledge while continuously listening to and adapting to input from musicians or other agents in real-time.

Somax2 adds a totally new versatility by being highly reactive to the musician’s decisions, and by putting its creative agents to communicate and coordinate in the same way. Thanks to cognitively inspired memory model/ interaction strategies, and finely optimized concurrent architecture, that make all its units smoothly cooperate together and participating in a diversity of live co-creative interaction with musicians in improvisation, composition or installation scenarios.

Defining co-creativity as distributed agency between human and machine [1], Somax2 is indeed a good example of a distributed co-creative system, where multimodal cross feedback affects collective behavior and increases player’s engagement through emotion and motivation.

Written in Max and Python [2], Somax2 features a modular multithreaded implementation, new UI design with tutorials and documentation [1]. The new 2.5 version, it is presented, in Max (©cycling74), both as an application (plug and play abstractions) and a library, allowing the user to program custom Somax2 patches in order to design one’s own environment and processing, involving as many modules are needed.

Somax2 allows detailed parametric controls of its agents and can even be played
alone as an instrument in its own right, or even used in composition workflow.

3. Somax2 Demo Proposal

This demo presentation will introduce the software environment, demonstrate its
learn/model/listen/generate phases, explain the basic and advanced controls in the user
interface, and allow people to interact and play with it. See video resources, tutorials and
demos at [1].

4. Project Context and Credits

Somax2 development by Joakim Borg, Somax created by Gérard Assayag and
Laurent Bonnasse Gahot.
Somax2 is part of the research projects ANR MERCI (Mixed Musical Reality with Creative Instruments) and ERC REACH (Raising Co-creativity in Cyber-Human Musicianship) of the European Union's Horizon 2020 framework. PI: Gérard Assayag.

## References

[1] Somax 2 Project Page: docs, tutorials, research reports, videos, resources : http://repmus.ircam.fr/somax2

[2] Somax2 Code Repository: https://github.com/DYCI2/Somax2/releases

[3] Gerard Assayag, George Bloch, Marc Chemillier, Arshia Cont, Shlomo Dubnov. Omax brothers: a dynamic topology of agents for improvisation learning. In: Proceedings of the 1st ACM workshop on Audio and music computing multimedia. Santa Barbara, United States. 2006. p. 125–132.

[4] Gérard Assayag, Improvising in Creative Symbolic Interaction, in Jordan B. L. Smith; Elaine Chew; Gérard Assayag (Eds) Mathematical Conversations: Mathematics and Computation in Music Performance and Composition, World Scientific Imperial College Press, pp.61 - 74, 2016

[5] J. Nika, K. Déguernel, A. Chemla-Romeu-Santos, E. Vincent, and G. Assayag. Dyci2 agents: merging the "free", "reactive", and "scenario-based" music generation paradigms. In International Computer Music Conference, Shangai, China, Oct 2017.

[6] Benjamin Feldman, Improving the latent harmonic space of SOMax with Variational Autoencoders, Master's report, Ircam, 2021; G. Assayag and T. Carsault supervisers.

[7] Gérard Assayag. Human-Machine Co-Creativity. in Bernard Lubat, Gérard Assayag, Marc Chemillier. Artisticiel / Cyber-Improvisations. Phonofaune, 2021, Dialogiques d'Uzeste, 2021

[8] Joakim Borg, Gérard Assayag, Mikhail Malt. Somax 2 A Reactive Multi-Agent Environment for Co-Improvisation. Proc. of the Sound Music & Computing SMC'22, Jun 2022, Saint-Étienne, France.