# Crafting Musical Agents: Interactive Generation as a Catalyst for Artistic Formalization

Jérôme Nika, Diemo Schwarz, Augustin Muller

► To cite this version:

Jérôme Nika, Diemo Schwarz, Augustin Muller. Crafting Musical Agents: Interactive Generation as a Catalyst for Artistic Formalization. 6th Conference on AI Music Creativity (AIMC 2025), Sep 2025, Brussels, Belgium. hal-05314380

HAL Id: hal-05314380

https://hal.science/hal-05314380v1

Submitted on 14 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Crafting Musical Agents: Interactive Generation as a Catalyst for Artistic Formalization

**Jérôme Nika**

ISMM Team

STMS Lab Ircam - CNRS - Sorbonne Université

**Diemo Schwarz**

ISMM Team

STMS Lab Ircam - CNRS - Sorbonne Université

**Augustin Muller**

Computer Music Designer
Ircam Production Department

1 Place Igor Stravinsky, 75004 Paris, France
firstname.lastname@ircam.fr

## Abstract

This paper presents an overview of a research-creation project aimed at designing libraries for the development of user-defined musical agents, conducted at Ircam over the past decade within a performance-led and practice-based framework. It reflects on a set of research directions in generative music systems oriented toward co-specification, tunability, and the formalization of artistic reasoning. These insights draw from the experiences accumulated through the incremental design of Dicy2 and OpenTuning in collaboration with expert artists to outline possible orientations for future research in human-system co-individuation within musical practice. These systems are approached not as autonomous generators, but as epistemic instruments—interfaces through which artistic thinking can be shaped, tested, and iteratively refined. Rather than focusing on creativity as surprise or stylistic reproduction, they emphasize the role of generative tools as catalysts for the externalization and evolution of compositional ideas.

## 1 Looking for Surprises in Rehearsals, for Reproducibility on Tour

This paper presents an overview of a research-creation project aimed at designing libraries for the development of user-defined musical agents (Tatar and Pasquier, 2018), conducted at Ircam over the past decade within a performance-led and practice-based framework. It has been punctuated by over 60 artistic creations (Roulette Brooklyn New York; Onassis Center Athens; Ars Electronica Festival Linz; Frankfurter Positionen festival; Annenberg Center Philadelphia; Bimhuis Amsterdam; Maison de la Radio, Centre Pompidou, Le Centquatre Paris; Le Fresnoy Tourcoing; Montreux Jazz festival; Montreal Jazz Festival...)<fnref n="1" /> born from artistic collaborations in jazz and improvised music (Steve Lehman, Rémi Fox, Orchestre National de Jazz, Hervé Sellin, Bernard Lubat, Benoît Delbecq, Jozef Dumoulin, Ashley Slater), contemporary music (Pascal Dusapin, Alexandros Markeas, Ensemble Modern, Marta Gentilucci), and contemporary art (Le Fresnoy, Vir Andres Hera, Gaëtan Robillard). These collaborations served as case studies in which scientific and technological research was used to deepen compositional inquiry rather than to introduce machinic surprise or otherness: all of these

<footnote n="1">¹See tinyurl.com/dicy2-prod for more resources.</footnote>
projects involved generative systems that exhibit some degree of autonomy, yet the focus was not on unpredictability, but on composing the behavior of the software instrument itself.

<figure><img src="image_2.png" /><figcaption>Figure 1: “Music of choices” by Alexandros Markeas at Centre Pompidou, Paris, 2021.</figcaption></figure>

Among them, the musical happening “Music of choices” created at Centre Pompidou in 2021,<fnref n="2" /> implemented an early prototype representing a foundational step in the incremental development process that led to the creation of Dicy2 (Nika et al., 2022a,b). In “Music of choices”, composer and improviser Alexandros Markeas’s piano improvisation was analyzed in real time by a system that generated a response on a second mechanical piano (Disklavier) (Figure 1). The project integrated real-time symbolic music generation through the activation of custom generative agents embedding models learned on a selected “memory” corpus by “scenario” queries derived from the analysis of the pianist’s live audio. The preparatory studio phase with the performer involved curating and categorizing sound materials (such as “chimes” or “fluid textures”) to train the system. Through iterative sessions, the generative responses were refined to establish a complex yet reproducible interaction model. Although each performance remained improvised, the relationship between human and virtual pianists was no longer exploratory: it had become a fully integrated component of the compositional framework. In this creation the second piano was not conceived as a source of fertile otherness or as a provocateur of improvisational deviation, as confirmed by Alexandros Markeas during a semi-structured interview when asked: “Were you ever surprised by what the machine could do on its own?”—even though the project was built around generative improvisation:

> Oh no, no, no, we weren't playing that game. We had it all worked out. I personalize things, I interact with Jérôme, not with "an agent". Sometimes I'd visualize his face, and he'd say to me, "Careful, this agent is analysing chromas! [...] We fixed things and I play with him. It could have been a violinist with whom I recorded, and then I have the violin recording, and I play with it.

This pattern reappeared in most of the artistic collaborations mentioned in this paper: the broader motivation falls within what could be called “software luthiery” or “meta-composition”. One recurrent insight was that while the early stages of creation often embraced surprise, once the instrument had

<footnote n="2">²Created at Centre Pompidou in 2021, the piece was then performed in Bourges, Besançon, Nanterre between 2023 and 2025. More resources and video extract: www.tinyurl.com/music-of-choices.</footnote>
taken shape, surprise was no longer the objective. Contrary to what one might expect when addressing interaction with generative agents for live music performance, the goal was not to be assisted in the creation process, but rather to increase its complexity, extending the act of composition to the design of the instrument itself.

After providing some background on related work and methodological grounding (Section 2), this paper presents an overview of the Dicy2 agents<fnref n="3" /> that emerged as a first milestone from this research (Section 3) whose aim is to produce finished artworks while supporting long-term development frameworks, and to foster critical reflection on creative processes by embedding generative technologies under development within artistic productions (Section 4). We then discuss new research directions entangling the design of interactive systems by modeling relational behaviors between paired tracks through the OpenTuning project<fnref n="4" /> (Section 5), to conclude by addressing the question of idiomatic acceptability (Section 6).

## 2 Situated Research for Designing Musical Agents

This project is situated at the intersection of creative AI systems, human-machine improvisation, and compositional toolmaking. The brief overview below outlines the main bodies of work that inform our approach. Ethnographic methodologies and practice-based research have emphasized the importance of studying creative AI systems within real-world artistic settings. Practice-based research involves creating new knowledge through the process of artistic making, while participant observation implies deep involvement of the researcher in the artistic act. These orientations align with performative epistemologies, which understand knowledge production as emerging through action and embodied interaction. Caramiaux and Donnarumma (Caramiaux and Donnarumma, 2020), for instance, explore the affective and situated dimensions of AI integration in live performance contexts. Fiebrink and Caramiaux (Fiebrink and Caramiaux, 2018) show how example-based training allows artists to iteratively shape system behavior. Bown (2015) further explores how autonomous agents operate in real-time co-creative contexts, highlighting the tension between emergent behaviors and pre-scripted structures.

Generative systems and musical agents frame generative behavior as a compositional material. Within the field of musical metacreation (Pasquier et al., 2017), systems are designed to encode stylistic constraints and intentional structures rather than produce sound alone. Our approach relates closely to other agents generate audio through corpus-based concatenative synthesis (Schwarz, 2007) such as Somax (Borg et al., 2022), MASOM (Tatar and Pasquier, 2017) and SpireMuse (Thelle and Pasquier, 2021), which integrate affect analysis and real-time listening, and tINNbre (Bolzoni et al., 2021), which embeds neural decision-making and uses Griffin-Lim synthesis (Griffin and Lim, 1984). Expanding the range of methods, recent creative AI systems have explored ways to condition generation through multimodal inputs: audio (Donahue et al., 2023; Caillon and Esling, 2021), text (Agostinelli et al., 2023), scores (Rouard and Hadjeres, 2021), or controllable audio parameters (Nistal et al., 2022). These systems integrate specification mechanisms directly into the generative process. Other approaches avoid explicit output specification, instead generating sound by interpolating or extrapolating between multiple timbres using latent audio representations (Tatar et al., 2021). While the Dicy2 environment presented in this paper (Section 3) is not centered on affective computing in the classical sense (i.e., emotion detection and response), it shares with systems such as MASOM and SpireMuse a meta-design perspective in which the generative system is co-shaped through human interaction. Here, artistic collaborators are not merely providing materials or stylistic constraints, but are actively involved in crafting how the system listens and responds, a process that acknowledges the situated, subjective, and dynamic nature of musical interaction. Relational reasoning, as pursued in the OpenTuning project (Section 5), shifts the focus from similarity-based generation to the modeling of functional and stylistic relationships between two streams of material (e.g., a memory and a reference). This allows generative behaviors to express not imitation, but complementarity. The layering of Dicy2 and OpenTuning environments thus reflects a conceptual progression: from agents trained on example sequences toward agents trained on example relations.

<footnote n="3">³Dicy2 for Max and Ableton Live are available at the following address: https://forum.ircam.fr/projects/detail/dicy2/.</footnote>

<footnote n="4">⁴OpenTuning is an ANR-funded project supported by the French National Research Agency (https://anr.fr/en/) for the 2025-2029 period.</footnote>
# 3 From Custom-built Instruments to the Dicy2 Library

This research-creation process led to the final release of Dicy2 in late 2022.. In this section, we briefly outline its features from a user point of view, in order to provide the minimal interpretive framework necessary to understand the musical issues that gave rise to it, which are discussed in the following sections. Dicy2 is both a library for Max<fnref n="5" /> and a plugin for Ableton Live<fnref n="6" /> implementing interactive agents using machine learning to generate musical sequences that can be integrated into musical situations ranging from the production of structured material within a compositional process to the design of autonomous agents for improvised interaction. It enables the creation of agents that learn from example sequences and later respond to scenario-like queries, generating coherent outputs under specific constraints.

## 3.1 A Platform for Scenario-Based Symbolic and Audio-Driven Generation Workflows

<figure><img src="image_3.png" /><figcaption>Figure 2: Overview of the different elements involved in the design workflow of an interactive audio device with Dicy2 and the interaction with it.</figcaption></figure>

The operational process within a complete Dicy2 setup (consisting of an agent, perception, and rendering modules, as shown in Figure 2) can be outlined as follows:

<footnote n="5">⁵http://cycling74.com/max</footnote>

<footnote n="6">⁶https://www.ableton.com/</footnote>
1. A Memory Model, embedded in an Agent, is trained on a corpus memory composed of (content, label) pairs.

2. The Agent receives scenarios in the form of label, label, ... sequences to query the Memory Model.

3. The Agent outputs optimized content, content, ... sequences.

4. These sequences are played back or used to control a user-defined process that accepts content as input.

The generative engine combines probabilistic learning mechanisms with customized pattern-matching algorithms. Readers interested in algorithmic and computational details are referred to publications dedicated to this topic (Nika et al., 2017). We only mention here that the generative model at the heart of Dicy2 produces optimal sequences by selecting sub-sequences from the memory, maintaining at each step a continuity between the scenario's projected structure and the memory's learned material, as well as between concurrent or successive queries. It learns associations between sequences of content symbols (e.g. time markers in an audio buffer, midi contents, etc.) and label symbols (e.g. user-defined symbols, automatically extracted audio features, harmonic functions, etc.), enabling the generation of new sequences in response to novel scenario inputs.

A Python library defines the core generative models. A generative engine built from it is then embedded within a Max library that provides utilities for the automatic construction of label sequences. The agent learns sequences of labeled content and responds to scenario-like inputs by generating new sequences that reflect temporal structure. These scenarios may be used for querying generative behavior in either compositional or real-time contexts across different modalities (e.g., audio, MIDI). Learning and generation rely on content/label pairs, supporting continuity, coherence, and user-defined optimality in sequence construction.

Figure 3 shows a minimal setup instantiating one agent: the training material in this case is symbolic: a short sentence annotated with grammatical labels such as pronoun, verb, and adjective. The agent learns the sequence and can then be queried with a new sequence of labels.<fnref n="7" />

A canonical Dicy2 audio setup, as explored throughout the research and creation process leading to the finalized models and environment, is that of a corpus-based generative agent for interactive audio. Designed for real-time generation, the system listens dynamically to live audio streams and supports responsive behaviors that range from deterministic mapping to autonomous variation, while maintaining a continuous yet flexible coordination with the input. In real-time audio use, the workflow consists of:

* **An audio memory segmented and analyzed offline:** consisting of an audio buffer containing the source sound; a MuBu container (Schnell et al., 2009) with segmentation and analysis data; and a file encoding the clustering model;

* **A symbolic layer built via clustering of audio descriptors (e.g., chromas + spectral centroid):** audio is segmented into discrete events—typically based on loudness—and a set of descriptors is extracted. The clustering process then assigns each segment to a symbolic class, establishing the label space on which generative operations are based;

* **A real-time input stream analyzed live and mapped to labels:** labels from the memory are obtained through the offline analysis and clustering pipeline, while labels from the guiding input are inferred in real time by matching the live audio stream to the latent structure created during memory construction;

* **A generative agent that selects matching audio segments:** temporal scanario to query the agent are then derived from the analysis according to user-defined mechanisms;⁸

* **A sequencer-renderer module:** the system includes a sequencer-renderer abstraction based on concatenative synthesis (Schwarz et al., 2006). This module maps symbolic output to

<footnote n="7">⁷learn {label} {content} = {gram_label (str)} {word (str)}
generate {label} {label}... = generate {gram_label (str)} {gram_label (str)}...
→ {content} {content}... = {word (str)} {word (str)}...

⁸learn {label} {content} = {cluster_id (int)} {time标志 audio (int)}
generate {label} {label}... = generate {cluster_id (int)} {cluster_id (int)}...
→ {content} {content}... = {time标志 audio (int)} {time标志 audio (int)}...</footnote>
audio segments and schedules their playback. Strategies include direct concatenation or
alignment with rhythmic structures, depending on the context.

<figure><img src="image_4.png" /><figcaption>Figure 3: Toy example of sequence generation from a memory model and an input scenario embedded in Max.</figcaption></figure>

## 3.2 From Concert Patchers to Tutorials: Designing Through Artistic Immersion

### 3.2.1 Distributed Experimentation and Artistic Evaluation

The research activities were initially conducted within the framework of documented artistic pro-
ductions (Szpirglas, 2022e,f) and artist residencies (Szpirglas, 2021, 2022b,a), and later extended to
include autonomous projects led by partner computer music designers (Szpirglas, 2022d,c). Follow-
ing the consolidation of core prototypes by extracting reusable features and generalized modes of
interaction from these projects, a second phase of development involved autonomous experimentation
by early users, musicians, computer music designers, and sound artists. Their feedback, gathered
through interviews and critical listening sessions, contributed to refining the system. These users
explored diverse applications, among them for instance: Franziska Baumann focused on live memory construction through voice and gesture; Kalikay worked with field recordings and non-musical corpora; Jonathan Bell integrated the system into a VR environment where spatial navigation triggered both audio and animation; Federico Placidi developed live coding practices involving subjective annotation and scenario control; Hans Tutschku implemented a dual-piano system for real-time MIDI generation, in collaboration with Vijay Iyer. Each singular project introduced new evaluation criteria: gesture responsiveness, idiomatic coherence, or behavioral plasticity.<fnref n="9" />

Some of these endeavors have furthered the research and creation process by implementing creative projects in an analytical and reflective approach, aiming to characterize the creative appropriation of successive prototypes of these generative agents by artists (Tsiros and Palladini, 2020; Hoy and Van Nort, 2024). In certain instances as in the case of the project “eTu{d,b}e”, Dicy2 has also been examined in relation to the typologies of interaction supported by related technologies (Davis et al., 2023; Davis, 2024; Davis et al., 2024). These examples highlight how users not only deploy but also reinterpret system behavior, acting as co-designers.

### 3.3 Performance-Led Methodologies

This research and creation project was developed over a ten-year period through a performance-led approach grounded in participant-observation (Rice, 2013). Rather than observing from a distance, the first author was embedded within the artistic process as electronic musician, shaping the evolving system through direct involvement in its creative use. This situated engagement echoes Mantle Hood's notion of acquiring “musicality” through instrumental immersion (Hood, 1960), though the instrument in question is not acoustic but computational, and above all is still under construction. Here, musical reflexivity emerges not through bodily instrumental practice, but through sustained interaction with code, interface, and generative behavior. In this framework, the project also resonates with George Lewis's call to recognize the symbiotic relationship between human and machine actors in creative processes (Lewis, 2021), and with Pierre Hadot's view of artistic activity as a transformative practice that reshapes both subjectivity and society (Hadot et al., 1997). These perspectives frame technical experimentation and aesthetic inquiry as interdependent modes of thought and action. The ethnographic component of the project included semi-structured interviews and critical listening sessions with artistic collaborators, offering shared moments of reflection and supporting an iterative dialogue between design choices and aesthetic evaluation. These interactions spanned both upstream and downstream phases of production thereby enriching the system's development with diverse artistic perspectives.

Dicy2 implements a complete workflow abstracted from the various artistic projects that served as research environments, whose main characteristics lie in the possibility of involving the artist directly in the construction of the instrument, in particular by specifying what the system listens to, how it interprets this input, and how it generates and sequences material over time. Each generative engine includes tutorial patches based on real productions, repackaged as educational resources. These examples allow users to interact with symbolic scenarios, train new memories, and manipulate temporal structures in performance or compositional contexts.

Across these research grounds, the project explored two distinct temporalities of composition: offline structural composition, supporting the organization of musical form outside real time; and online, interaction-based composition, enabling improvisation and responsiveness in live performance. These dual axes informed both the system's functional design and its aesthetic affordances as developed in the following section.

## 4 Formalizing and Composing Generative Behaviors

At the core of the design lies the question: what behavior are we composing? What kind of rule, logic, or process will articulate a memory model learned on a selected corpus and a guiding input to create new musical output? In composition or composed improvisation contexts, a predefined scenario may guide memory traversal independently of live input (4.1), whereas in generative improvisation, the system may respond directly to a musician's real-time audio inputs (4.2).

<footnote n="9">⁹More resources about these projects: https://discussion.forum.ircam.fr/t/thread-projects-by-forum-members-using-dicy2/</footnote>
## 4.1 Composing or Live-coding at the "Scenario" Level

Early productions explored the use of composed temporal structures to guide the traversal of musical memory. In these configurations, generation was driven by predefined structural scenarios. A particularly illustrative example is “Lullaby Experience”, developed in collaboration with composer Pascal Dusapin. <fnref n="10" /> The objective was to compose a sonic scene inside a dome of loudspeakers using generative agents guided at a high level of abstraction. The process followed an iterative loop of listening, evaluation, and refinement. The generative engine worked at a narrative level, allowing users to target either specific combinations of audio features or more subjective expressive targets. “Lullaby Experience” demonstrated the capacity of scenario-based generation to structure long-duration audio behaviors and support the layering of evolving textures.

Another example, “Misurgia Sisitlallan”, created with video artist Vir Andres Hera, involved the design of an installation film’s soundscape through the hybridization of spoken legends told in multiple languages and traditional Mexican melodies. <fnref n="11" /> The generative agent here functioned as a dramaturgical connector, producing composite sequences that transitioned fluidly across stylistic anchors (Figure 4).

<figure><img src="image_5.png" /><figcaption>Figure 4: “Misurgia Sisitlallan” by Vir Andres Hera, OVNi video festival, Nice, 2023.</figcaption></figure>

The fixed scenario approach opened the way to the next step: how can such structural guidance be produced in real time? One first option was to use the agent as a live coding instrument for “meta-DJing”. Instead of triggering pre-stored sequences, it would receive real-time specifications such as: “now generate something matching this short sequence of labels.” This transformed our

<footnote n="10">¹⁰February 2019, festival Frankfurter Positionen; June 2019, Le Centquatre, Paris, IRCAM Manifeste festival
Example: https://www.youtube.com/ playlist?list=PL-C_JLZNFA GddZ YJ4hwQSo17BcNJq3roU
See www.tinyurl.com/lullaby-experience, www.tinyurl.com/lullaby-docu, tinyurl.com/lullaby-itw, for more resources about “Lullaby Experience”.</footnote>

<footnote n="11">¹¹Extract: https://www.youtube.com/ playlist?list=PL-C_JLZNFA GddZ YJ4hwQSo17BcNJq3roU
Multimedia installation created at Le Fresnoy - Studio National des Arts Contemporains in 2020, presented at La Gaîté Lyrique (Paris) in 2022 and at the OVNi video festival in Nice in 2023. See www.tinyurl.com/misurgia for more resources.</footnote>
generative engines into software instruments for improvisation, enabling real-time response to human intentions specified on the fly through short-term queries.

The motivation for working on such scheduling mechanisms capable of handling dynamic, concurrent scenarios and overlapping time scopes (Nika et al., 2015) originated in a project by Marta Gentilucci during her residency at Ircam in 2017. <fnref n="12" /> Marta Gentilucci sought to perform live with a real soprano while simultaneously controlling a virtual soprano. This configuration enabled real-time control over articulation-level features, producing behaviorally convincing yet physically impossible responses. Marta Gentilucci’s “keyboard” consisted therefore of symbolic combinations such as tenuto–staccato–staccato (Figure 5).

<figure><img src="image_6.png" /><figcaption>Figure 5: Controlling a virtual soprano with dynamic short-term queries</figcaption></figure>

These commands were here entered manually, but the same mechanism could be applied differently: other project required that the system could also derive such queries by listening to a live musician’s performance.

## 4.2 From Manual Control to Generation Driven by Machine Listening

A typical use case that has become established over time involves designing a device capable of generating structured audio in real time in interaction with a guiding audio stream for example, as a responsive virtual soloist or dynamic texture. In this setup, the memory consists of audio files or buffers recorded on the fly that are automatically segmented and labeled along user-defined audio-musical dimensions. The guiding input is processed in real time using the same descriptors, and the resulting analysis is used to generate optimized symbolic responses from the Dicy2 agent.

This configuration was initially motivated by duo improvisation projects, to enable an acoustic musician to transmit scenario queries through live analysis of a live audio stream. To support this interaction, a live listening module was introduced. Its purpose was to analyze the incoming audio stream in real time, identify a symbolic label for each new audio segment by projecting it in the label space of the learned memory, and use this label to create a new scenario query from user-defined rules. Thus, setting the parameters for machine listening became compositional: composing how listening itself is translated into musical intent. Notable examples include “Silver Lake Studies”, developed in collaboration with Steve Lehman, and “C’est pour ça” with Rémi Fox (Figure 6).

<footnote n="12">¹²Extract: https://www.youtube.com/ playlist?list=PL-C_JLZNFA GddZYJ4hwQSo17BcNJq3roU</footnote>
<figure><img src="image_7.png" /><figcaption>Figure 6: “C’est pour ça” with Rémi Fox</figcaption></figure>

In the case of “Silver Lake Studies”<fnref n="13" />, the agent transitions from live-coding modes, receiving manually input harmonic and rhythmic density scenarios, to phases where it is guided directly by Steve Lehmans’s playing. Through this configuration, the agent shifts from an externally controlled accompanist to a spectral extension of the instrument, dynamically enriching the saxophone’s output. A central ambition of the project is to design a system that listens to an improviser and produces a new discourse, guided by what it hears on the fly according to behaviors premeditated collaboratively beforehand during rehearsal sessions.

Most of the work with “C’est pour ça” was precisely to create a taxonomy of playable modes by combining different parameters: memory analysis, live audio input listening, and typologies of scenarios that could be inferred from them. For example, “play a response exclusively matching the same label than the one analysed from the live input,” or “starting from this label,” or “moving toward this label”. This label represents a symbol characterizing both the analyzed audio and the memory slice to be retrieved in terms of a previously selected audio descriptor. In these examples, first inharmonicity and spectral centroid, then pitch class and chromas.<fnref n="14" />

To do so, the agent must be composed with specific listening dimensions and reactive strategies. In the example patcher presented in Figure 7, the user loads an audio memory along with segmentation and analysis parameters (descriptor types and time windows). The system listens to a live audio input, analyzes it, and infers the corresponding symbolic label. This label is then used to trigger a scenario processed by the trained agent. In the example shown in Figure 7, the memory contains guitar recordings, while the live input is a saxophone. The system attempts to match the input to known material and plays segments accordingly.

<footnote n="13">¹³Created at Onassis cultural center in Athens, 2019.
More resources and video extract: www.tinyurl.com/silver-lake-studies.</footnote>

<footnote n="14">¹⁴Examples: https://www.youtube.com/ playlist?list=PL-C_JLZNFA GddZ YJ4hwQSo17BcNJq3roU
See www.tinyurl.com/cest-pour-ca for more resources about “C’est pour ça” (performances, sound instal-
lations, etc.).</footnote>
<figure><img src="image_8.png" /><figcaption>Figure 7: Controlling generation through short-term scenarios derived from machine listening</figcaption></figure>

# 5 Rethinking and Teaching Generative Behaviors

## 5.1 Developing a taxonomy of generative behaviors

The artistic collaborations we have conducted suggest that interaction with so-called AI generative technologies<fnref n="15" /> can support artistic reflexivity and reshape the creative process by acting as a catalyst for formalization. In line with Bernard Stiegler’s notion of disautomation (Stiegler, 2015), machine learning in generative aesthetics may then serve as a counterpoint to technological determinism. It reframes the role of AI in creative contexts, emphasizing data curation, interactivity, and compositional authorship over automation (Robillard and Nika, 2024). Each digital instrument, specific to its project, can be understood as a potential piece developed in the studio, and interaction activates this potential during performance. Two artistic concerns guide this approach: designing behaviors that reflect a particular artistic intent, and ensuring those behaviors are reproducible. Reproducibility varies by context but always concerns the balance between control and freedom. Trial and error often reveals that initial assumptions—about which features the system should attend to or how it should respond—must be revised. A corpus may respond more effectively to dynamics than to pitch, even if intuition suggested otherwise.

This iterative confrontation fosters critical insight: it forces both the performer and the designer to formalize what is intended and what is left open. In many projects, early studio sessions began with large memories and highly flexible generative mechanisms. Over time, both the memory and the behavior became more focused, a convergence that paralleled the refinement of artistic objectives. Furthermore, during preparatory work, the tuning process is not purely technical: musicians also adapt their playing to the system’s behavior. This creates a form of mutual calibration, in which both performer and machine converge toward a shared instrumental gesture.

This principle was explored in depth with Rémi Fox. A short video from this collaboration illustrates the concept of interaction taxonomies. We defined specific parameters for the system and tested how

<footnote n="15">¹⁵A category we define from a usage-based perspective, encompassing systems that combine a degree of autonomy with high-level control, whether or not they rely on deep learning.</footnote>
they influenced the musical relationship—ranging from direct following of the performer, to parallel musical discourse, to more decorative or effect-based roles. The central issue and interest of the approach consist in identifying meaningful configurations between these extremes. As indicated in the captions of the video, composing a behavior involves specifying four key dimensions forming the basis of a compositional grammar for behavioral design:

* **Memory:** What source material is provided to the system?

* **Listening:** What feature of the live input should the system track (e.g., chroma)?

* **Temporal scale:** Should the system react to individual attacks, discrete events, or broader trends inferred from sliding temporal windows?

* **Interpretation strategy:** Should the system match precisely, or allow looser navigation between stylistic landmarks? How should it move from a perceived label to the temporal label sequence used as a generation query?

<figure><img src="image_9.png" /><figcaption>Figure 8: Excerpt from the video documenting the taxonomy of Dicy2 playing modes, produced during Rémi Fox’s residency at IRCAM in 2022.</figcaption></figure>

## 5.2 Internalizing and Playing Generative Behaviors

To go further on this topic, we may consider the example of the piece “13 Soleils” composed by Steve Lehman within the “Ex Machina” project by the French National Jazz Orchestra (Figure 9).

A representative sequence features Fanny Ménégoz on flute, performing with a system trained on a drum track. The system responded in real time to her note density and duration, attempting to follow her phrasing while occasionally diverging. <fnref n="18" /> This produced a form of intersubjective tension: the performer learned to shape the system without explicit control, by developing an embodied knowledge of its reactions. Through rehearsal, Fanny Ménégoz developed an instrumental gesture that allowed her to guide the system’s output. Even without direct control, she was able to shape the

<footnote n="16">¹⁶Premiered at Maison de la Radio in Paris and broadcast live on France Musique in 2022, followed by a European tour in 2022, an American and European tour in 2023, and another European tour in 2024. The album was released on Pi Recordings / L'Autre Distribution in September 2023. See https://jeromenika.com/tag/ex-machina/ and www.tinyurl.com/onj-reportage, for more resources.</footnote>

<footnote n="17">¹⁷Video extract: www.tinyurl.com/onj-dingman.</footnote>

<footnote n="18">¹⁸Video extracts: www.tinyurl.com/onj-menegoz.</footnote>
generative response by internalizing a dual musical gesture that merged execution and interpretation. This became particularly clear at the end of the interlude, when she led the system into a conclusive gesture—slowing it down, introducing loops, and finally cueing the rhythm section to begin the piece.

<figure><img src="image_10.png" /><figcaption>Figure 9: “Ex Machina” by Steve Lehman and French National Jazz Orchestra at Maison de la Radio, Paris, 2022 (top) and Roulette Intermedium, Brooklyn, New York City, 2022 (bottom).</figcaption></figure>

5.3 OpenTuning: Towards Modeling Relational Behaviors

The work described above led to the development of Dicy2, which has since been appropriated
by a growing community of users. It also set the stage for a new research phase, now focused
on OpenTuning a new research project about creative individuation through the interaction design
of generative musical systems funded by the French National Research Agency (2026-2029). Its
objective is to move beyond similarity-based interaction design and focus on composing relationships
between inputs and outputs through machine teaching, using a curated dataset of musical pairs.
<figure><img src="image_11.png" /><figcaption>Figure 10: Designing interaction with generative systems through machine teaching using curated small datasets of paired examples.</figcaption></figure>

Rather than feeding a model with a single file, the system now receives two: the standard memory corpus file and a relational reference illustrating a desired complementarity (Figure 10). We adopt a dual experimental strategy to design different decision modules, developing both lightweight probabilistic models and deep learning architectures (Bujard et al., 2026), including a perception module based on Wav2Vec 2.0 (Ragano et al., 2023) and a transformer decoder with top-P sampling (Holtzman et al., 2020)). This comparative approach aims to determine which modeling paradigm is best adapted to each musical and data context.

<figure><img src="image_12.png" /><figcaption>Figure 11: Introducing a decision module internalizing relationships learned on curated pairs of tracks (Bujard et al., 2026).</figcaption></figure>

To evaluate these models, musicians are asked to record relational examples directly over memory material. In testing, they play live inputs to assess whether the system produces outputs aligned with the previously demonstrated relationship. Documentation videos of initial sessions with Rémi Fox
and Steve Lehman are available. In the first case, the goal was to produce an electro solo reacting to pivot notes that Rémi Fox had recorded over the memory he intended to use.<fnref n="19" /> In the second, the aim was to generate a saxophone solo with engaging phrasing and structure, based on the relationship Steve Lehman defined between a saxophone track and a bass track.<fnref n="20" />

# 6 Towards Acceptability as a Means Rather Than an End

This paper has outlined a long-term research project on generative systems as epistemic instruments. We proposed a framework grounded in scenario-based interaction, meta-design, and behavioral composition. From Dicy2 to OpenTuning, we presented a progression from sequence generation to relationship modeling, from symbolic rule-chaining to embodied inference, from surprise to specification. By embedding generative tools into real-world artistic processes, we advocate for a generative lutherie that is not only creative, but also formalizing: capable of revealing, refining, and even challenging compositional thought.

For this new research phade, we chose to use strongly idiomatic repertoires as research grounds. Idiomatic, here, refers to music grounded in a specific tradition or genre, where stylistic conventions guide musical choices—"Idiomatic [music] is mainly concerned with the expression of an idiom—a style or musical language that is already well-defined and established" (Bailey, 1980). Several recent artistic collaboration have therefore engaged with stylistically rich traditions characterized by strong idioms and high thresholds of acceptability. This raises a central question: what value can such tools offer when applied within an idiomatic repertoire? Indeed, if the goal is merely stylistic reproduction, a generative system may be redundant. If the goal is stylistic rupture, why train on the style at all? One ongoing project with Ensemble Sine Cum draws from the practice of *Cantus super Librum*,<fnref n="21" /> a medieval technique of improvised counterpoint in which a melodic line is extemporized above a notated base. This practice, essential to the training of polyphonic musicians, anticipates many principles of structured improvisation. To compare this new approach with earlier systems, experiments are carried out in two phases: first, the system simply followed the performer's voice; in the second, it generated responses based on learned voice-to-voice relationships, allowing for meaningful divergence within stylistic constraints. The aim here is not to generate correct notes, but to demonstrate responsiveness to the structural logic of the style. A complementary line of research involves the group Sélébéyone, and specifically two rappers, Priest/HPrizm (Antipop Consortium) and Gaston Bandimic exploring whether vocal delivery can modulate pre-recorded layers in real time. Initial tests focus on technical validation: whether vocal density can drive rhythmic change, or pitch inflections guide harmonic shifts.<fnref n="22" />

In both cases, the objective is not performance-ready output, but to probe what kinds of relationships a system can internalize, and how such learning reshapes artistic practice. The most interesting question here is not how to replicate a style, but what it means to use such technologies when embedded in a repertoire with a strong identity? What value do they offer musically and methodologically? These experiments are not yet about making music, but about exploring system behavior through constrained, idiom-informed exercises. In this context, idiomatic acceptability is not the end goal, but a tool for testing whether the system can learn and reproduce meaningful relationships demonstrated to it. The artistic goal will follow, just as, a decade ago, we began by generating polite solos over harmonic progressions, not for their musical value but as test cases for evaluating a model's capacity to follow a scenario.²³

<footnote n="19">¹⁹Research-creation residency with Steve Lehman and the group Sélébéyone at the Jazz Gallery in New York in March 2025 and at the Grand Théâtre National in Dakar in June 2025.
Video extract: https://www.youtube.com/ playlist?list=PL-C_JLZNFA GddZYJ4hwQSo17BcNJq3roU</footnote>

<footnote n="20">²⁰Experimentation with Rémi Fox (2025) in preparation for the Electro-Odyssée Memoria concert at IRCAM with Fabrizio Rat and Augustin Muller in February 2026.
Video extract: https://www.youtube.com/ playlist?list=PL-C_JLZNFA GddZYJ4hwQSo17BcNJq3roU</footnote>

<footnote n="21">²¹"Cantus extra librum" concert at the Évry Cathedral with the ensemble Sine Cum (Valérie Lepage, Raphaële Soumagnas), in collaboration with the SCAI (Sorbonne Center for Artificial Intelligence), March 2025.
Video extract at https://www.youtube.com/ playlist?list=PL-C_JLZNFA GddZYJ4hwQSo17BcNJq3roU</footnote>

<footnote n="22">²²Research-creation residency with the group Sélébéyone at the Jazz Gallery in New York in March 2025 and at the Grand Théâtre National in Dakar in June 2025. Video extracts: https://www.youtube.com/ playlist?list=PL-C_JLZNFA GddZYJ4hwQSo17BcNJq3roU
²³Early works developed with Rémi Fox in preparation for a performance at the Montreux Jazz Festival in 2015. Extract: https://www.youtube.com/ playlist?list=PL-C_JLZNFA GddZYJ4hwQSo17BcNJq3roU</footnote>
## Ethical Statement

The datasets used in this work were accessed legally, either under open access or formal agreement of the data copyright owners. The code developed for this project is and will be released as open-source software to ensure reproducibility. The authors report no conflicts of interest.

## Acknowledgments

Dicy2 is a library of the Ircam Musical Representations team, designed and developed by Jérôme Nika, Augustin Muller (Max library), Joakim Borg (Python generative engine), and Matthew Ostrowski (tutorial patchers and videos, abstractions) in the framework of the projects ANR-DYCI2, ANR-MERCI, ERC-REACH directed by Gérard Assayag, and the UPI-CompAI Ircam project. The audio use cases have been designed and developed with Diemo Schwarz and Riccardo Borghesi, and use the MuBu and CataRT environments of the ISMM team of Ircam. Max4Live plugin by Manuel Poletti. Contributions / thanks : Serge Lemouton, Jean Bresson, Thibaut Carpentier, Georges Bloch, Mikhaïl Malt, Axel Chemla-Romeu-Santos, Tristan Carsault, Vincent Cusson, Tommy Davis, Dionysios Papanicolaou, Greg Beller, Markus Noisternig. OpenTuning is a project led by the ISMM team at IRCAM, and the work presented in this article on medieval music and rap was made possible thanks to the support of the SCAI (Sorbonne Center for Artificial Intelligence) and a grant from the Doris Duke Foundation awarded to Steve Lehman respectively.

## References

- Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., Sharifi, M., Zeghidour, N., and Frank, C. (2023). Musiclm: Generating music from text. (arXiv:2301.11325). arXiv:2301.11325 [cs, eess].

- Bailey, D. (1980). *Improvisation: Its Nature and Practice in Music*. Moorland Pub.

- Bolzoni, A., Di Donato, B., and Laney, R. (2021). tinnbre: a timbre-based musical agent. In *Nordic SMC 2021: The 2nd Nordic Sound and Music Conference*, Copenhagen. Nordic SMC, Aalborg University.

- Borg, J., Assayag, G., and Malt, M. (2022). Somax 2 a reactive multi-agent environment for co-improvisation.

- Bown, O. (2015). Player responses to a live algorithm: Conceptualising computational creativity without recourse to human comparisons. In *Proceedings of the Sixth International Conference on Computational Creativity*, pages 126–133.

- Bujard, B., Nika, J., Bevilacqua, F., and Obin, N. (2026). Learning relationships between separate audio tracks for creative applications. In *Proceedings of the Conference on AI Music Creativity (AIMC)*.

- Caillon, A. and Esling, P. (2021). Rave: A variational autoencoder for fast and high-quality neural audio synthesis. (arXiv:2111.05011). arXiv:2111.05011 [cs, eess].

- Caramiaux, B. and Donnarumma, M. (2020). Artificial Intelligence in Music and Performance: A Subjective Art-Research Inquiry.

- Davis, T., Pocius, K., Cusson, V., Gentili-Morin, M., and Pasquier, P. (2024). Embodied etube gestures and agency. In *Proceedings of the 9th International Conference on Movement and Computing*, pages 1–6.

- Davis, T., Pocius, K., Cusson, V., Pasquier, P., and Wanderley, M. M. (2023). etu {d, b} e: Case studies in playing with musical agents. In *Proceedings of the International Conference on New Interfaces for Musical Expression*, pages 268–276.

- Davis, T. R. (2024). The etube project: Researching human-computer interaction through an interdisciplinary collaboration with improvising musical agents.
Donahue, C., Caillon, A., Roberts, A., Manilow, E., Esling, P., Agostinelli, A., Verzetti, M., Simon, I., Pietquin, O., Zeghidour, N., and Engel, J. (2023). Singsong: Generating musical accompaniments from singing. (arXiv:2301.12662). arXiv:2301.12662 [cs, eess].

Fiebrink, R. and Caramiaux, B. (2018). The machine learning algorithm as creative musical tool. pages 181–208.

Griffin, D. and Lim, J. (1984). Signal estimation from modified short-time fourier transform. *IEEE Transactions on Acoustics, Speech, and Signal Processing*, 32(2):236–243.

Hadot, P., Davidson, A. I., and Chase, M. (1997). Philosophy as a way of life: Spiritual exercises from socrates to foucault. *Philosophical Quarterly*, 47(188).

Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2020). The curious case of neural text degeneration. In *8th International Conference on Learning Representations, ICLR 2020*.

Hood, M. (1960). The challenge of "bi-musicality". *Ethnomusicology*, 4(2):55–59.

Hoy, R. D. and Van Nort, D. (2024). Human-machine co-creation in an electroacoustic ensemble performance context. In *Proceedings of the 19th International Audio Mostly Conference: Explorations in Sonic Cultures*, pages 93–104.

Lewis, G. (2021). Co-Creation: Early Steps and Future Prospects. In *In Artisticiel / Cyber-Improvisations. Phonofaune, 2021, Dialogiques d'Uzeste*.

Nika, J., Bouche, D., Bresson, J., Chemillier, M., and Assayag, G. (2015). Guided improvisation as dynamic calls to an offline model. In *Sound and Music Computing (SMC)*, Maynooth, Ireland.

Nika, J., Chemillier, M., and Assayag, G. (2017). Improtek: Introducing scenarios into human-computer music improvisation. *Computers in Entertainment*, 14.

Nika, J., Muller, A., Borg, J., Assayag, G., and Ostrowski, M. (2022a). Dicy2 for Max. Technical report, Ircam UMR STMS 9912.

Nika, J., Muller, A., Borg, J., Poletti, M., and Assayag, G. (2022b). Dicy2 for Ableton Live. Technical report, Ircam.

Nistal, J., Lattner, S., and Richard, G. (2022). Drumgan: Synthesis of drum sounds with timbral feature conditioning using generative adversarial networks. (arXiv:2008.12073). arXiv:2008.12073 [cs, eess].

Pasquier, P., Eigenfeldt, A., Bown, O., and Dubnov, S. (2017). An introduction to musical metacreation. *Comput. Entertain.*, 14(2).

Ragano, A., Benetos, E., and Hines, A. (2023). Learning music representations with wav2vec 2.0. In *2023 31st Irish Conference on Artificial Intelligence and Cognitive Science (AICS)*, pages 1–6.

Rice, T. (2013). *Ethnomusicology: A Very Short Introduction*. Oxford University Press.

Robillard, G. and Nika, J. (2024). Critical climate machine: A visual and musical exploration of climate misinformation through machine learning. *Proc. ACM Comput. Graph. Interact. Tech.*, 7(4).

Rouard, S. and Hadjeres, G. (2021). Crash: Raw audio score-based generative modeling for controllable high-resolution drum sound synthesis. (arXiv:2106.07431). arXiv:2106.07431 [cs, eess].

Schnell, N., Röbel, A., Schwarz, D., Peeters, G., Borghesi, R., et al. (2009). Mubu and friends—assembling tools for content based real-time interactive audio processing in max/msp. In *ICMC*.

Schwarz, D. (2007). Corpus-based concatenative synthesis. *IEEE signal processing magazine*, 24(2):92–104.

Schwarz, D., Beller, G., Verbrugghe, B., and Britton, S. (2006). Real-time corpus-based concatenative synthesis with catart. In *9th International Conference on Digital Audio Effects (DAFx)*, pages 279–282.
Stiegler, B. (2015). *La Société automatique. 1. L'avenir du travail*. Fayard, Paris.

Szpirglas, J. (2021). C'est pour ça: Artificial Hippocampus. https://www.ircam.fr/article/remi-fox-jerome-nika-hippocampe-artificiel. Ircam Blog, accessed July 11, 2025.

Szpirglas, J. (2022a). C'est pour ça: In Order of Appearance. https://www.ircam.fr/article/remi-fox-jerome-nika-44-in-order-of-appearance. Ircam Blog, accessed July 11, 2025.

Szpirglas, J. (2022b). C'est pour ça: Live Music in Athens. https://www.ircam.fr/article/remi-fox-jerome-nika-live-music-in-athens. Ircam Blog, accessed July 11, 2025.

Szpirglas, J. (2022c). Critical Climate Machine: Composing with DYCI2. https://www.ircam.fr/article/critical-climate-machine-composer-avec-dyci2. Ircam Blog, accessed July 11, 2025.

Szpirglas, J. (2022d). Dicy2 in the climate machine. https://www.ircam.fr/article/critical-climate-machine-saison-1-episode-2-dyci2-dans-la-machine-climatique. Ircam Blog, accessed July 11, 2025.

Szpirglas, J. (2022e). In Ex Machina. https://www.ircam.fr/creation/compositeurs-et-artistes-en-studio/in-ex-machina. Ircam Blog, accessed July 11, 2025.

Szpirglas, J. (2022f). Jazz Ex Machina. https://www.ircam.fr/creation/compositeurs-et-artistes-en-studio/jazz-ex-machina. Ircam Blog, accessed July 11, 2025.

Tatar, K., Bisig, D., and Pasquier, P. (2021). Latent timbre synthesis. *Neural Computing and Applications*, 33(1):67–84.

Tatar, K. and Pasquier, P. (2017). MASOM: A Musical Agent Architecture based on Self-Organizing Maps, Affective Computing, and Variable Markov Models. In *Proceedings of the 5th International Workshop on Musical Metacreation (MuMe 2017)*.

Tatar, K. and Pasquier, P. (2018). Musical agents: A typology and state of the art towards musical metacreation. *Journal of New Music Research*, 47:1–50.

Thelle, N. J. W. and Pasquier, P. (2021). Spire Muse: A Virtual Musical Partner for Creative Brainstorming. In *NIME 2021*. https://nime.pubpub.org/pub/wcj8sjee.

Tsiros, A. and Palladini, A. (2020). Towards a human-centric design framework for ai assisted music production. In *Proceedings of the International Conference on New Interfaces for Musical Expression*, pages 399–404.