# Somax 2 a Reactive Multi-Agent Environment for Co-Improvisation

Joakim Borg, Gérard Assayag, Mikhail Malt

► To cite this version:

Joakim Borg, Gérard Assayag, Mikhail Malt. Somax 2 a Reactive Multi-Agent Environment for Co-Improvisation. SMC 2022 - Sound Music & Computing, Jun 2022, Saint-Étienne, France. Proceedings of the 19th Sound and Music Computing Conference, 2022. hal-04001271

HAL Id: hal-04001271

https://hal.science/hal-04001271

Submitted on 22 Feb 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Somax 2 a Reactive Multi-Agent Environment for Co-Improvisation

Joakim Borg, Gérard Assayag, Mikhail Malt

► To cite this version:

Joakim Borg, Gérard Assayag, Mikhail Malt. Somax 2 a Reactive Multi-Agent Environment for Co-Improvisation. SMC 2022 - Sound Music & Computing, Jun 2022, Saint-Étienne, France. Proceedings of the 19th Sound and Music Computing Conference, 2022. hal-04001271v2

HAL Id: hal-04001271

https://hal.science/hal-04001271v2

Submitted on 29 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

# Somax 2
a Reactive Multi-Agent Environment for Co-Improvisation

Joakim Borg, Gérard Assayag, Mikhail Malt

joakim.borg@ircam.fr - gerard.assayag@ircam.fr - mikhail.malt@ircam.fr

<figure><img src="image_3.png" /></figure>

## Introduction

Somax 2 [1] is a multi-agent interactive system performing live machine co-improvisation with musicians, based on machine-listening, machine-learning, and generative units. The actual version [6, 7, 8] is a recent development and algorithms improvement from the former Somax version and previous work in RepMus team [3,4, 5, 9, 11, 12].

Agents provide stylistically coherent improvisations based on learned musical knowledge while continuously listening to and adapting to input from musicians or other agents in real time. The system is trained on any musical materials chosen by the user, effectively constructing a generative model (called a corpus), from which it draws its musical knowledge and improvisation skills. Corpora, inputs and outputs can be MIDI as well as audio, and inputs can be live or streamed from Midi or audio files. Somax 2 is one of the improvisation systems descending from the well-known Omax software, presented here in a totally new implementation. As such it shares with its siblings, the general loop [listen/learn/model/generate], using some form of statistical modeling that ends up in creating a highly organized memory structure from which it can navigate into new musical organizations, while keeping style coherence, rather than generating unheard sounds as other ML systems do. However Somax 2 adds a totally new versatility by being incredibly reactive to the musician decisions, and by putting its creative agents to communicate and work together in the same way, thanks to cognitively inspired interaction strategies and finely optimized concurrent architecture that make all its units smoothly cooperate together.

Somax 2 allows detailed parametric controls of its players and can even be played alone as an instrument in its own right, or even used in composition workflow. It is possible to listen to multiple sources and to create entire ensembles of agents where the user can control in detail how these agents interconnect and "influence" on each other. Somax 2 is conceived to be a co-creative partner in the improvisational process, where the system after some minimal tuning is able to behave in a self-sufficient manner and participate to a diversity of improvisation set-ups and even installations.

## RepMus Omax Co-Improvisation Paradigm

**Listen** : cognition aware discovery of a symbolic vocabulary from any audio stream

**Learn** : create incrementally a sequence model, a structured memory that becomes a generative source of new music

**Generate** : Navigate all the relations inside the memory/model using smart heuristics / strategies and create new sequences on the fly

**Omax** has then refined into several co-creative softwares :
**ImproteK / Djazz** : adds a global guidance by a formal scenario (temporal composition, idiomatic frames)

**Somax** : adds a reactive guidance through a continuous real-time influence from the live context

**Dyci2** : adds the concept of micro-scenarios triggered incrementally

The actual version, developed by **Joakim Borg** under the supervision of **Gerard Assayag** ([Borg 2020], [Borg 2021a], [Borg 2021b]), is a recent development and algorithms improvement from the... Former Somax version and previous work in RepMus team ([Bonasse-Gahot 2012], [Bonasse-Gahot 2014], [Carsault 2017], [Carsault 2019], [Carsault 2020], [Carsault 2021])

**Actual Somax2 Version** : https://github.com/DYCI2/Somax2/releases/

<figure><img src="image_4.png" /></figure>

## Navigation Model

<figure><img src="image_5.png" /></figure>

User interface to control the balance between the dimensions, length of matching sequences for each dimension as well as decay time of peaks.

Three layers of peaks corresponding to different musical dimensions such as pitch, harmony, feedback, etc., being merged into one set of peaks before the final scaling and peak selection. Here, all three layers are weighted equally, but it is possible to balance the contribution from each of the layers.

<figure><img src="image_6.png" /></figure>

## Influences

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

Interaction model for the wireless architecture. Dotted arrow lines denote some sort of "wireless" communication between objects while filled arrow lines denote their traditional UML relations (composition, generalization) and cor- responding cardinality.

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

## Somax2 Front End Elements

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

## References

[1] Somax 2 General Information, demos, installation, internal reports: https://www.stms-lab.fr/projects/pages/somax2

[2] Gerard Assayag, George Bloch, Marc Chemillier, Arshia Cont, Shlomo Dubnov. Omax brothers: a dynamic topology of agents for improvisation learning. In: Proceedings of the 1st ACM workshop on Audio and music computing multimedia. Santa Barbara, United States.2006. p. 125-132.

[3] Gérard Assayag. Human-Machine Co-Creativity. in Bernard Lubat, Gérard Assayag, Marc Chemillier. Artistiel / Cyber-Improvisations. Phonofaune, 2021, Dialogiques d’Uzeste, 2021

[4] Laurent Bonnasse-Gahot. Prototype de logiciel d'harmonisation/arrangement à la volée: Somax v0, (2012).

[5] Laurent Bonnasse-Gahot. An update on the SOMax project. Ircam-STMS. Internal Report ANR Project Sample Orchestrator. 2014. Available online: http://repmus.ircam.fr/_media/dyci2/somax_project_lbg_2014.pdf (accessed on 22 October 2021), 2014.

[6] Joakim Borg. Dynamic Classification Models for Human-Machine Improvisation and Composition. Master's report, Ircam and Aalborg University Copenhagen, 2020.

[7] Joakim Borg. The Somax 2 Software Architecture, internal technical report. Ircam, April 2021a

[8] Joakim Borg. The Somax 2 Theoretical Model, internal technical report. Ircam, April 2021b

[9] Tristan Carsault. Automatic chord extraction and musical structure prediction through semi-supervised learning, application to human-computer improvisation. Diss. Ircam UMR STMS 9912, 2017.

[10] Tristan Carsault, Jérôme Nika, and Philippe Esling. Using musical relationships between chord labels in automatic chord extraction tasks. arXiv preprint arXiv:1911. 04973 (2019).

[11] Tristan Carsault. Introduction of musical knowledge and qualitative analysis in chord extraction and prediction tasks with machine learning. Diss. Pierre and Marie Curie University, Paris, France, 2020.

[12] Tristan Carsault, Jérôme Nika, Philippe Esling, Gérard Assayag. Combining Real-Time Extraction and Prediction of Musical Chord Progressions for Creative Applications. Electronics, MDPI, 2021, 10 (21), pp.2634.

[13] Benjamin Feldman. Improving the latent harmonic space of Somax with Variational Autoencoders. Master's Thesis Centrale Supelec & STMS IRCAM. 2021. http://repmus.ircam.fr/merci/publications

[14] Jérôme Nika, Ken Déguernel, Axel Chemla—Romeu-Santos, Emmanuel Vincent, Gérard Assayag. DYCI2 agents: merging the “free”, “reactive”, and “scenario-based” music generation paradigms. International Computer Music Conference, Oct 2017, Shangai, China.
# Somax 2 a Reactive Multi-Agent Environment for Co-Improvisation

Joakim Borg, Gérard Assayag, Mikhail Malt

<figure><img src="image_2.png" /></figure>

## To cite this version:

Joakim Borg, Gérard Assayag, Mikhail Malt. Somax 2 a Reactive Multi-Agent Environment for Co-Improvisation. SMC 2022 - Sound Music & Computing, Jun 2022, Saint-Étienne, France. Proceedings of the 19th Sound and Music Computing Conference, 2022. hal-04001271

HAL Id: hal-04001271

https://hal.science/hal-04001271v1

Submitted on 22 Feb 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>
<figure><img src="image_4.png" /></figure>

# Somax 2
a Reactive Multi-Agent Environment for Co-Improvisation

Joakim Borg, Gérard Assayag, Mikhail Malt

joakim.borg@ircam.fr - gerard.assayag@ircam.fr - mikhail.malt@ircam.fr

<figure><img src="image_5.png" /></figure>

## Introduction

**Somax 2** [1] is a multi-agent interactive system performing live machine co-improvisation with musicians, based on machine-listening, machine-learning, and generative units. The actual version [6, 7, 8] is a recent development and algorithms improvement from the former Somax version and previous work in RepMus team [3,4, 5, 9, 11, 12].

Agents provide stylistically coherent improvisations based on learned musical knowledge while continuously listening to and adapting to input from musicians or other agents in real time. The system is trained on any musical materials chosen by the user, effectively constructing a generative model (called a corpus), from which it draws its musical knowledge and improvisation skills. Corpora, inputs and outputs can be MIDI as well as audio, and inputs can be live or streamed from Midi or audio files. Somax 2 is one of the improvisation systems descending from the well-known Omax software, presented here in a totally new implementation. As such it shares with its siblings, the general loop [listen/learn/model/generate], using some form of statistical modeling that ends up in creating a highly organized memory structure from which it can navigate into new musical organizations, while keeping style coherence, rather than generating unheard sounds as other ML systems do. However Somax 2 adds a totally new versatility by being incredibly reactive to the musician decisions, and by putting its creative agents to communicate and work together in the same way, thanks to cognitively inspired interaction strategies and finely optimized concurrent architecture that make all its units smoothly cooperate together.

Somax 2 allows detailed parametric controls of its players and can even be played alone as an instrument in its own right, or even used in composition workflow. It is possible to listen to multiple sources and to create entire ensembles of agents where the user can control in detail how these agents interconnect and "influence" on each other. Somax 2 is conceived to be a co-creative partner in the improvisational process, where the system after some minimal tuning is able to behave in a self-sufficient manner and participate to a diversity of improvisation set-ups and even installations.

## RepMus Omax Co-Improvisation Paradigm

**Listen** : cognition aware discovery of a symbolic vocabulary from any audio stream

**Learn** : create incrementally a sequence model, a structured memory that becomes a generative source of new music

**Generate** : Navigate all the relations inside the memory/model using smart heuristics / strategies and create new sequences on the fly

**Omax** has then refined into several co-creative softwares :
**ImproteK / Djazz** : adds a global guidance by a formal scenario (temporal composition, idiomatic frames)

**Somax** : adds a reactive guidance through a continuous real-time influence from the live context

**Dyci2** : adds the concept of micro-scenarios triggered incrementally

The actual version, developed by **Joakim Borg** under the supervision of **Gerard Assayag** ([Borg 2020], [Borg 2021a], [Borg 2021b]), is a recent development and algorithms improvement from the... Former Somax version and previous work in RepMus team ([Bonasse-Gahot 2012], [Bonasse-Gahot 2014], [Carsault 2017], [Carsault 2019], [Carsault 2020], [Carsault 2021])

**Actual Somax2 Version** : https://github.com/DYCI2/Somax2/releases/

## Learning

<figure><img src="image_6.png" /></figure>

## Corpus

<figure><img src="image_7.png" /></figure>

## Navigation Model

<figure><img src="image_8.png" /></figure>

User interface to control the balance between the dimensions, length of matching sequences for each dimension as well as decay time of peaks.

Three layers of peaks corresponding to different musical dimensions such as pitch, harmony, feedback, etc., being merged into one set of peaks before the final scaling and peak selection. Here, all three layers are weighted equally, but it is possible to balance the contribution from each of the layers.

## Generate

<figure><img src="image_9.png" /></figure>

## Influences

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

## Somax2 Front & Back End

<figure><img src="image_14.png" /></figure>

Module diagram over the main modules in the system and the relationship between them.

<figure><img src="image_15.png" /></figure>

This action is part of the project REACH supported by the European Research Council under Horizon 2020 programme (Grant ERC-2019-ADG #883313) and project MERCI supported by Agence nationale de la Recherche (Grant ANR-19-CE33-0010)

## References

[1] Somax 2 General Information, demos, installation, internal reports: https://www.stms-lab.fr/projects/pages/somax2

[2] Gerard Assayag, George Bloch, Marc Chemillier, Arshia Cont, Shlomo Dubnov. Omax brothers: a dynamic topology of agents for improvisation learning. In: Proceedings of the 1st ACM workshop on Audio and music computing multimedia. Santa Barbara, United States.2006. p. 125-132.

[3] Gérard Assayag. Human-Machine Co-Creativity. in Bernard Lubat, Gérard Assayag, Marc Chemillier. Artistiel / Cyber-Improvisations. Phonofaune, 2021, Dialogiques d'Uzeste, 2021

[4] Laurent Bonnasse-Gahot. Prototype de logiciel d'harmonisation/arrangement à la volée: Somax v0, (2012).

[5] Laurent Bonnasse-Gahot. An update on the SOMax project. Ircam-STMS. Internal Report ANR Project Sample Orchestrator. 2014. Available online: http://repmus.ircam.fr/_media/dyci2/somax_project_lbg_2014.pdf (accessed on 22 October 2021), 2014.

[6] Joakim Borg. Dynamic Classification Models for Human-Machine Improvisation and Composition. Master's report, Ircam and Aalborg University Copenhagen, 2020.

[7] Joakim Borg. The Somax 2 Software Architecture, internal technical report. Ircam, April 2021a

[8] Joakim Borg. The Somax 2 Theoretical Model, internal technical report. Ircam, April 2021b

[9] Tristan Carsault. Automatic chord extraction and musical structure prediction through semi-supervised learning, application to human-computer improvisation. Diss. Ircam UMR STMS 9912, 2017.

[10] Tristan Carsault, Jérôme Nika, and Philippe Esling. Using musical relationships between chord labels in automatic chord extraction tasks. arXiv preprint arXiv:1911. 04973 (2019).

[11] Tristan Carsault. Introduction of musical knowledge and qualitative analysis in chord extraction and prediction tasks with machine learning. Diss. Pierre and Marie Curie University, Paris, France, 2020.

[12] Tristan Carsault, Jérôme Nika, Philippe Esling, Gérard Assayag. Combining Real-Time Extraction and Prediction of Musical Chord Progressions for Creative Applications. Electronics, MDPI, 2021, 10 (21), pp.2634.

[13] Benjamin Feldman. Improving the latent harmonic space of Somax with Variational Autoencoders. Master's Thesis Centrale Supelec & STMS IRCAM. 2021. http://repmus.ircam.fr/merci/publications

[14] Jérôme Nika, Ken Déguernel, Axel Chemla—Romeu-Santos, Emmanuel Vincent, Gérard Assayag. DYCI2 agents: merging the “free”, “reactive”, and “scenario-based” music generation paradigms. International Computer Music Conference, Oct 2017, Shangai, China.

## Somax2 Front End Elements

<figure><img src="image_16.png" /></figure>