<figure><img src="image_1.png" /></figure>

HAL
open science

# Interactive Music Co-Creation with an Instrumental Technique-Aware System: A Case Study with Flute and Somax2

Nicolas Brochec, Marco Fiorini, Mikhail Malt, Gérard Assayag

► To cite this version:

Nicolas Brochec, Marco Fiorini, Mikhail Malt, Gérard Assayag. Interactive Music Co-Creation with an Instrumental Technique-Aware System: A Case Study with Flute and Somax2. International Computer Music Conference (ICMC 2025), Jun 2025, Boston (MA), United States. hal-05061669

HAL Id: hal-05061669

https://hal.science/hal-05061669v1

Submitted on 9 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Interactive Music Co-Creation with an Instrumental Technique-Aware System:
A Case Study with Flute and Somax2

**Nicolas Brochec***

Tokyo University of the Arts
nicolas.brochec@pm.me

**Mikhail Malt**

IRCAM, Sorbonne Université
CNRS, Ministère de la Culture, STMS
F-75004 Paris, France
mikhail.malt@ircam.fr

**Marco Fiorini***

Sorbonne Université, IRCAM
CNRS, Ministère de la Culture, STMS
F-75004 Paris, France
marco.fiorini@ircam.fr

**Gérard Assayag**

IRCAM, Sorbonne Université
CNRS, Ministère de la Culture, STMS
F-75004 Paris, France
gerard.assayag@ircam.fr

## ABSTRACT

This paper presents a significant advancement in the co-creative capabilities of Somax2 through the integration of real-time recognition of a broad range of flute Instrumental Playing Techniques (IPTs). Building on the concept of interactive systems for musical co-creation, this implementation enables the recognition and response to diverse IPTs, enhancing the depth of interaction during live performance. The approach was tested in collaboration with a professional flutist performing Cassandra's Dream Song by Brian Ferneyhough, a piece renowned for its complex intricacy of extended playing techniques. Through this experiment, the artificial agents demonstrated the ability to adapt and engage with the performer's techniques, creating a more dynamic and responsive experience. This functionality relies on a CNN-based flute IPT recognition model, which informs the agents' real-time responses. The paper examines the broader implications of these innovations for music co-creation, interactive performance, and human-AI collaboration, offering new workflows for creative musical systems and potential applications in composition and live improvisation.

## 1. INTRODUCTION

Within the diversity of available tools for human-machine improvisation, many systems rely on specific reactive listening mechanisms, coupled with the user's ability to control the generation of musical agents in real-time. Early systems such as Voyager [1] and Omax [2] focused on controlling specific low-level parameters, while later developments introduced parameter abstractions for managing model behaviors and complex agent interactions [3]. To further enrich generative paradigms, various agent interaction techniques have been proposed, particularly within the

Corpus-Based Concatenative Synthesis (CBCS) paradigm [4]. This approach has significantly transformed real-time music feature navigation, facilitating novel creative applications. Foundational studies on human-machine co-creativity emphasize the importance of emergent situations [5] and non-linearity [6], which are characteristic of complex systems and are crucial for real-world contexts of musical co-improvisation [7].

A family of software descending from the Omax interaction paradigm [2] has concentrated many of these advanced features in their design by leveraging machine learning concepts to build generative musical memory, and by using audio descriptors-based machine listening techniques to manage the reactive component of the system in response to an unpredictable external sonic context involving human musicians (or other agents). These systems vary in the way they balance their skills regarding responsiveness and planning, scanning the whole spectrum from free improvisation to open composition. Improtek, Djazz [8], and Dicy2 [9] use a symbolic structure of defined labels to guide musical generation within a corpus through the concept of “scenarios” [10], allowing hybrid temporality in multimodal interactions [11]. Somax2 [12], the more advanced system to date in the Omax family, is skilled in creating rich and highly dynamic interaction between performers and musical content straight out of the box by combining an original model of cognitive memory, a powerful listening machine, a latent space trained on a large data set, and a set of interaction policies determining when and how to react to external and internal “influences”. By integrating the instrumental techniques recognition methods proposed in this paper to Somax2, it will be possible to dramatically enhance the listening machine and boost its interactions with the cognitive memory module by finding matches at a morphological level rather than the unit level of audio descriptors opening the way to enhanced “listening” skills and better awareness of musical gestures in the unfolding of interaction.

This research sets the stage for new interaction paradigms based on improved machine listening, where the real-time recognition of Instrumental Playing Techniques (IPTs) allows for more precise and contextually relevant responses

<footnote>*These authors contributed equally to this work</footnote>
during performance. In this study, we aim to advance this
emerging concept by introducing the integration of IPT
recognition directly into Somax2 output generation pro-
cess. Our focus is on extending the system's flexibility
through label-based control techniques, enabling real-time
generation of musical events guided by IPT labels. This
novel workflow facilitates a dynamic interaction model,
where recognized IPT labels influence the system output
and musical decision-making. By combining these inno-
vations with Somax2 existing capabilities, this work con-
tributes to the development of more expressive and adap-
tive systems for human-AI musical collaboration *.

The structure of this paper is as follows: Section 2 re-
views existing approaches to IPT detection and describes
our proposed model for the real-time recognition of flute
IPTs. In Section 3, we introduce our new workflow in So-
max2, incorporating the notion of classification labels re-
lated to the real-time recognition of flute IPTs. We also
present two strategies for controlling and managing these
labels over time. Section 4 details the experimental setup
for our evaluation, Section 5 provides a discussion of our
findings, and Section 6 presents our conclusions.

## 2. REAL-TIME IPT RECOGNITION

This section describes the methodology we used to design
our real-time IPT recognition system that provides Somax2
with the label of the recognized IPT. The experiment eval-
uates our system performance in real-world situations.

### 2.1 Existing Studies

Recent studies have widely utilized machine learning algo-
rithms to identify Instrument Playing Techniques (IPTs).
Examples of these algorithms include those for the recog-
nition of IPTs of Chinese bamboo flute [13], guzheng [14],
violin [15], electric guitar [16], acoustic guitar [17], cello
[18], and flute [19, 20]. To our knowledge, the only stud-
ies that have tested their system in real-world scenarios are
those on the acoustic guitar [17] and the cello [18]. Never-
theless, the existing methods for the real-time recognition
of IPTs are still insufficiently accurate and tackle a nar-
row range of IPTs, which makes them not yet applicable to
creative practices. We selected the flute as the focus instru-
ment for this study because it is a widely played instrument
and because the first author is familiar with it.

### 2.2 The Flute Playing Techniques

The sound of the flute is produced by the friction of the
air blown on the mouthpiece [21]. Varying the blowing
pressure, the degree of the coverage of the mouth hole, or
the blowing direction changes allow for the performance
of different playing techniques [22]. Several playing tech-
niques are based on articulations, such as the *staccato*,
*pizzicato*, *atterzunge*, and *tongue ram* techniques. Some
are based on the blown air flux, such as the *ordinario*, *aeo-
lian*, and *whistle tone* techniques. Some techniques involve
fingerings, such as the *multiphonics* and *trill* techniques.
Others require singing while blowing, *play and sing*, or hit-
ting the keys with fingers *key click* [23]. The standard flute

in C produces a chromatic scale from medium (C3 ≈ 261 Hz) to treble pitch (Eb6 ≈ 2489 Hz) [22], with some exceptions. Thanks to a unique key, some professional flutes allow playing B2 ≈ 247 Hz and facilitate reaching G6 ≈ 3136 Hz [23]. Our real-time IPT recognition system aims to recognize these various flute-playing techniques as audio data in real time.

### 2.3 Datasets

To train our flute real-time IPT recognition system, we se-
lected the audio files from the *GFDatabase* [24] because
this sound bank includes diverse and numerous recordings
of flute IPTs. The *GFDatabase* consists of recordings from
seven microphones placed at different distances and an-
gles from the source, annotated A, B, C, D, and E. It in-
cludes 11 flute playing techniques recorded pitch by pitch
that we chose for our experiment, totaling 165 minutes
of recordings. These playing techniques are the *aeolian*,
*flatterzunge*, *key click*, *multiphonics*, *ordinario*, *pizzicato*,
*play and sing*, *staccato*, *tongue ram*, *trill*, and *whistle tone*.
An existing study [20] shows that combining the A, B, C,
and D recordings improves the performance of their classi-
fier, which is why we chose to use the same recordings for
our experiment. Playing techniques have different lengths,
from long (e.g., *ordinario*) to very short (e.g.,*staccato*), so
classes have different total lengths. This class imbalance
is a serious concern when training a classification model,
as the majority class may bias the model. To address this
concern when building our training dataset, we used an ex-
isting balanced sampler* to ensure that each training batch
contains the same number of samples per class, so no class
is favored.

It is crucial to evaluate our classifier on a separate test
dataset (heterogeneous datasets) for a comprehensive as-
sessment of its performance. We selected the *FullSOL*
sound bank [25] considering the same flute playing tech-
niques as in our training dataset. An existing study [20]
concludes that combining perceptually similar playing tech-
niques into the classes would help the classifier because
it increases the number of samples per class. The *GF-
Database* follows this methodology, so we applied it to the
*FullSOL* sound bank. The second minor trill and the sec-
ond major trill are placed in the *trill* category. The *whistle
tone* and *whistle tone sweeping* are placed in the *whistle
tone* category. Singing at the same pitch as the note played
or not is considered performing a *play and sing* technique.
Also, we placed the audio files from *note-lasting* folder to
the *ordinario* category, considering playing an *ordinario*
sound, one second or half a second is still an *ordinario*
sound. We then removed the unused playing techniques.
The test dataset totals 48 minutes of recordings.

The *GFDatabase* includes simultaneous recordings with
microphones placed at different distances and angles from
the source. Building the validation dataset from the train-
ing dataset may result in data leakage, as the same note
recorded with two or more different microphones can be
placed both in the training and validation datasets. To tackle
this concern, we recorded our own flute playing techniques
sound bank to use as the validation dataset. We recorded

<footnote>*Demo video of the system: https://youtu.be/
OzNZkECfB7c</footnote>

<footnote>*Available on the web: https://github.com/khornlund/pytorch-balanced-sampler</footnote>
the same IPTs with a semi-professional performer at a distance of 1.5m with a Sony EMC670 microphone at 48kHz and 24-bit resolution. Our validation dataset totals 17 minutes of recordings.

## 2.4 Audio Files Preprocessing

As flute harmonics can go well above 10kHz [22], we first tested several sampling rates in preliminary experiments to determine which was suitable for our task. We observed that 44.1kHz improved performance. Thus, to preprocess our audio files, we first downsampled them to 44.1kHz. We trimmed the silence because it is irrelevant. Following existing studies [19, 20], we sliced the audio files into a maximum number of adjacent sequences of 325ms. When the sequence was too short, we padded it with zeros.

## 2.5 Data Augmentation

Data augmentation is a technique to address the lack of training data in audio classification tasks [26]. It is based on generating new data samples by applying audio transformations to the original ones, expanding the training dataset, and increasing its diversity.

We applied the following transformations based on real-world performance conditions. We detuned the recording of ±100 Hz around A440 to simulate natural tuning variations. We added Gaussian noise to simulate the noise generated by signal amplification. We adjusted audio playback speed by a random factor between 0.9 and 1.1, simulating the variability in IPT execution speeds. We applied these transformations separately to each sequence and increased the total amount of data by four.

## 2.6 System Architecture

### 2.6.1 Data Samples Preprocessing

Sequences previously sliced from audio files (see Section 2.4) are analyzed using a Log-Mel-Spectrogram (LMS) analysis, implemented as a custom layer within our model's architecture. Considering that short and percussive IPTs require high-resolution features [14], we computed the LMS using 256 Mel Bands, an FFT window of 2048 samples, and a hop length of 512 samples (≈11.6ms at 44.1kHz). We set the minimum frequency to 150Hz as a compromise to mitigate the impact of residual noise (e.g., studio ventilation) in the data, which could interfere with the learning process while preserving most of the register of the *tongue ram* technique which starts at ≈130Hz (B2 fingering). A higher minimum frequency would limit the system's ability to classify *tongue ram* technique accurately. The Log-Mel Spectrograms are normalized and then inputted into the neural network.

### 2.6.2 CNN-based Classifier

To achieve real-time recognition of instrumental playing techniques, prior studies [17, 18, 20, 27, 28] have demonstrated the effectiveness of deep Convolutional Neural Networks (CNNs). Building on a state-of-the-art architecture [20], we conducted a preliminary ablation study to identify and remove layers that did not contribute to performance improvements. Moreover, reducing the number of layers

<figure><img src="image_3.png" /><figcaption>**Figure 1.** Proposed architecture using four two-dimensional convolutional blocks and three fully connected blocks. *N* is the number of batches, for training *N* = 128, for real-time inference *N* = 1.</figcaption></figure>

would decrease latency, enhancing suitability for real-time uses.

Our model architecture illustrated in Figure 1 consists of four convolutional blocks at the top, each composed of a 2D convolutional layer followed by BatchNorm2d, LeakyReLU, MaxPooling2d, and Dropout. The number of channels in the convolutional layers is set to 40, 80, 160, and 160, with kernel sizes of 4x8, 3x6, 2x4, and 1x2, respectively. At the bottom end of the architecture, three fully connected blocks are used, each consisting of a Linear (fully connected) layer followed by BatchNorm2d, ReLU, and Dropout. The sizes for these layers are set to 160, 80, and 40, respectively. A flatten layer is inserted between the convolutional and fully connected blocks to ensure dimensional compatibility. Finally, a softmax function is applied to get a probability distribution over the classes, and the most probable class is selected using an argmax function. The model has approximately 230,000 parameters.

### 2.6.3 Real-time Inference

To perform real-time inference, we utilized a Python script that executed the model and handled the audio stream in the background. The audio stream is sent from Max/MSP to PyAudio through BlackHole, a virtual audio loopback driver*. The classification result is then sent to Max/MSP using OSC protocol [29]. The inference is performed each time the PyAudio callback function is called, i.e., every time the internal buffer is refreshed. The internal buffer of PyAudio is set to 256 samples (≈ 5.8ms at 44.1kHz). We used an M1-chip MacBook Pro laptop without running any other applications in the background. This configuration is used for the evaluation in Section 4.

<footnote>*Available on the web: https://github.com/ExistentialAudio/BlackHole</footnote>
<figure><img src="image_4.png" /><figcaption>**Figure 2.** Normalized confusion matrix averaged over ten tests (%).</figcaption></figure>

## 2.7 Training

We set the batch size to 128 to train our CNN-based classifier, taking into account both memory usage and computational efficiency. The network's weights were initialized using Xavier Normal Initialization [30]. The cross-entropy loss function was chosen as it is suited for multi-class classification tasks. The training was performed using the ADAM optimizer with a weight decay of $1 \times 10^{-5}$. The model was trained for 100 epochs on an A6000 GPU, and the training process was completed in approximately 30 minutes.

## 2.8 Measurements

We measured the accuracy and macro F1 score across ten tests to comprehensively evaluate the model's performance. After training for 100 epochs, measurements are performed on the model state with the lowest validation loss. Accuracy and macro F1 scores are then averaged, and standard deviations are computed.

We measured the latency between successive classification results by running the best-performing model in real-time using a sequence of audio files randomly selected from the test dataset. The latency between successive classification results was measured in Max 9 using a *timer*, *array.group*, *array.mean*, and *array.stdev* objects. Latency results were averaged over 1000 inferences, and standard deviations were computed. Measurements were taken first without running Somax2 agents, then running one, two, and three agents.

## 2.9 Results

Our model performs well on ten tests with an average accuracy score of 91.37%, and an average macro F1 score of 83.07%. Standard deviations are 0.71% for the accuracy and 3.99% for the macro F1 score. The averaged confusion matrix on the ten tests Figure 2 shows that the model performs well on most playing techniques.

Table 1 shows that the addition of Somax2 agents has little impact on the latency of our real-time IPT recognition system. The addition of further agents may increase the latency.

<table><caption>**Table 1.** Latency measurements averaged over 1000 inferences while running no agent, one, two, and three Somax2 agents loaded with corpus (ms).</caption><thead><tr><td>Configuration</td><td>Average ±σ</td></tr></thead><tbody><tr><td>No agent</td><td>10.03 ± 4.53</td></tr><tr><td>One agent</td><td>10.23 ± 4.54</td></tr><tr><td>Two agents</td><td>10.96 ± 4.73</td></tr><tr><td>Three agents</td><td>11.67 ± 5.05</td></tr></tbody></table>

## 3. WORKFLOW WITH REAL-TIME IPT CLASSIFICATION IN SOMAX2

Our research addresses existing Somax2 limitations, as detailed in Section 1, by introducing higher planning possibilities into the system, specifically by integrating real-time IPT classification. By combining the system's inherent reactivity with new mechanisms for dynamic filtering, enhanced navigation models, and customizable control parameters, we propose a workflow that links a more thorough offline corpus construction with real-time performance. This workflow enables context-aware generative output that aligns more closely with artistic intentions, providing flexibility and control.

This section details this new workflow, presented in Figure 3, starting from the preparation of a multidimensional corpus and progressing through the integration of IPT labels and their application in event generation. Additionally, we highlight how users can employ Somax2 configurable parameters to tailor the system's responsiveness and output to specific scenarios. While these parameters are not dynamically altered in our controlled experimental setup, described in Section 4, they remain an essential part of the system's broader capabilities, ensuring that users can adapt it to a wide range of co-creative contexts.

### 3.1 Offline Corpus Building

The generative capabilities of Somax2 rely on a structured and analyzed *corpus* of musical material. This corpus consists of audio or MIDI data, segmented into discrete *slices* based on detected onsets. Each slice undergoes a multidimensional analysis to extract key features, such as pitch and chroma, creating a structured material for an internal navigation model that guides the generative process. This format-agnostic process ensures stylistic consistency within the corpus [12].

With the integration of a real-time IPT classifier into the workflow (as detailed in the following subsections), it becomes necessary to associate a specific IPT label with each slice of the corpus during the offline corpus building stage. These labels, specified by the user after corpus segmentation, are added as a new feature layer to the corpus (see black dotted line section in Figure 3). This additional dimension will enrich the navigation model of the system, providing a new dimension for controlling the generative process.

### 3.2 Integration of IPT Labels in Real-Time Interaction

The core innovation of our approach lies in the real-time detection and integration of IPT labels into Somax2's in-
<figure><img src="image_5.png" /><figcaption>**Figure 3.** Diagram illustrating our new workflow integrating real-time IPT recognition into Somax2: offline corpus building with IPT annotation (black), real-time CNN classification for IPT labels (pink), conventional Somax2 feature extraction and influence processes (blue), and audio signal flow (red).</figcaption></figure>

teraction model. During a live performance, the system analyzes the musician's input, classifying IPT alongside pitch and chroma features detected by the Somax2 Influencer module. These IPT labels are dynamically incorporated into the player's decision-making process, enabling contextually relevant output generation. The integration of this new thread in the workflow is depicted in Figure 3, where the top pink branch illustrates the process of real-time IPT classification and how these newly retrieved labels are sent to the Somax2 Player. The bottom blue branch presents the regular process of onset, pitch, and chroma feature extraction (performed by the Somax2 AudioInfluencer module), which sends this information to the Player. These two branches now operate concurrently and in real time, resulting in a new structure of influences.

Specifically, IPT labels enhance real-time interaction in two key ways:

* **Dynamic Filtering:** IPT labels serve as a filter for corpus navigation, restricting the player's selection to slices matching the detected IPT class. This dynamic filtering adapts in real time to the musician's performance.

* **Model Extension:** IPT labels are incorporated as additional layers in the multidimensional navigation model, complementing the existing pitch and chroma features. These layers contribute to memory activation peaks, which guide the system's selection of output slices. The memory model simulates short-term memory by retaining peaks from prior influences and shifting them over time, allowing the system to prioritize consistent patterns. This approach enhances coherence and enables fuzzy pattern recognition, making the system adaptable to complex musical nuances.

In more detail, each input influence, segmented and analyzed in real-time, triggers the creation of peaks in the memory model [31]. These peaks represent locations in the corpus that match the musician's input across multi-

ple dimensions, including IPT characteristics. The system simulates short-term memory by not discarding old peaks immediately but by allowing them to decay and merge with newer peaks, resulting in a more contextually relevant output. The merging of peaks across all layers allows the system to select output slices probabilistically, balancing the fidelity to the musician's input with the corpus's inherent structure [12].

While this process requires a more thorough offline corpus preparation stage, it enhances the workflow by incorporating IPT information that would be challenging to analyze and retrieve solely from basic descriptors such as pitch and chroma. Adding this new dimension of machine listening and interactivity allows the system to respond to real-world instrumental characteristics often employed in improvisation and contemporary music, such as advanced performance techniques.

## 3.3 Control Interaction Parameters

Somax2 offers several parameters to control the interaction with its player's generative output. Below, we highlight three key parameters and their potential applications, particularly suited for our research:

* *Cut:* This parameter forces the generation of a new event whenever a new influence (e.g., onset or IPT label) is detected, regardless of whether the current event has finished. It ensures tight temporal alignment between input and output. For instance, activating *cut* enables the player to respond immediately to the IPT class of a new onset.

* *Timeout:* Defines the maximum duration for which the player can continue generating material in the absence of new influences. This parameter helps control pacing, ensuring that generative output does not become overly dense or sparse when input activity is limited.

* *Quality:* A threshold value for memory activation peaks that determines whether an output event is gen-
erated. Higher *quality* values require closer matches between input and corpus content, leading to more selective responses. Adjusting *quality* allows users to balance coherence and variability in output generation.

These parameters enhance Somax2's interaction capabilities, enabling dynamic control over scheduling and player responsiveness. However, they are not essential for the integration of real-time IPT classification. In our workflow, shown in Figure 3, these parameters are omitted for simplicity, as they are part of the regular Somax2 documentation and features [31]. Instead, the following section focuses on their fixed configuration, which is specifically tailored for the evaluation of our system with a professional flutist. While these parameters are not explicitly altered in our experiment's fixed configuration, they provide users with significant flexibility to customize interactions according to their specific artistic or research goals.

## 4. EVALUATION EXPERIENCE

To evaluate the impact of IPT labels in the system, an interaction experience was conducted with a professional flutist in a professional recording studio. The performer, chosen for her strong background in contemporary flute repertoire, was asked to interpret *Cassandra's Dream Song* by Brian Ferneyhough, a piece known for its extensive use of complex playing techniques. The system responded in real-time using a corpus derived from *Unity Capsule* by the same composer, which had been previously segmented and annotated with the corresponding IPTs. Two sessions were performed: one with IPT recognition activated and one without. The performer followed her personal interpretation of *Cassandra's Dream Song*, structured around numbered cells as suggested by the composer, ensuring consistency between both sessions. During the sessions, the flutist was not provided with visual feedback from the system to focus solely on auditory interaction. Parameters for the Somax2 Player and AudioInfluencer modules were identical in both conditions. Specifically, the AudioInfluencer was configured with an Onset Pitch detection algorithm (yin~ [32]), quality of 0.75, gain threshold of -45 dB, minimum frequency of 100Hz, and a chroma decay rate of 300 ms. The Player operated in reactive mode with cut parameter enabled, continuity set at 1.00, and a timeout of 1 second. As for the listening module of the Player, the balance of weights for the different dimensions were 0.25 for both internal melody and harmony, 1.00 for external melody, and 0.31 for external harmony, with a memory length of 2 events on all the dimensions. All audio outputs and performances were recorded for post-analysis. This setup enabled a controlled environment to isolate the effect of IPT annotations on the flutist's experience and the system's behavior.

## 5. DISCUSSION

A critical evaluation of the system's capabilities was conducted through a session with a professional flutist ("P"). During the session, "P" interacted with two variations of the co-creative system: one with and one without the integration of IPT labels. Feedback highlighted that the ver-

sion incorporating IPT labels significantly enhanced the perceived coherence and responsiveness of the system. "P" frequently described moments of interaction as feeling "accompanied" or "cohesive," citing specific instances where the system appeared to respond directly to her playing. For example, she noted a sense of musical dialogue when transitioning between specific techniques, such as *harmonics* and *atterzunge*, where the system mirrored these playing styles, creating a more engaging improvisatory experience.

However, "P" also identified areas where system responsiveness was inconsistent, particularly during rapid transitions between techniques. These inconsistencies were attributed to limitations in real-time recognition accuracy and corpus selection. The flutist further highlighted that while the IPT-enhanced version felt less "frustrating" compared to the baseline, occasional abrupt silences or mismatched outputs disrupted the sense of flow. These interruptions were attributed to the underlying segmentation method and parameterization of the system, which require further refinement.

Overall, the integration of IPT labels introduced a new dimension of interaction, as "P" described the IPT-enabled system as allowing her to "act" within the improvisation rather than merely reacting to a pre-programmed framework. Despite the limitations noted, the flutist's preference for the IPT-enabled system underscores the potential of real-time instrumental technique recognition in fostering deeper and more dynamic human-computer musical collaboration. These findings suggest paths for improvement, including refining corpus preparation and enhancing the temporal coherence of IPT recognition outputs, to further align the system's behavior with performers' expectations and intentions.

The results show that the model's performance is strong, with an accuracy of 91.37% and a macro F1 score of 83.07% averaged over ten tests. Despite these high global metrics, the confusion matrix reveals some localized issues with specific playing techniques, particularly *key click*, *tongue ram*, *play and sing*, and *multiphonics*. The misclassification of *key click* as *trill* may be attributed to the execution of *trill*. With sufficient strength, the trilled key produces a sound similar to *key click*, which may confuse the classifier. The misclassification of *tongue ram* could be attributed to its perceptual similarity to *pizzicato*, as both techniques involve strong percussive tongue attacks. For *play and sing*, singing and playing on the same pitch is also perceptually similar to *ordinario*, which may cause its misclassification. These two misclassifications could be alleviated by increasing the temporal and frequency resolution of the LMS, thereby providing the model with more detailed information about the subtle temporal variations in the frequencies of these techniques. Finally, for the *multiphonics*, we observed that it is sometimes classified as *ordinario*. Occasionally, performers produce a transitory "single-pitched" sound before finding the correct blowing angle to produce the *multiphonics*. This transitory sound may confuse the classification as it is sometimes similar to *ordinario*. To further improve the robustness of our system, we think it is worth exploring the *minus-1db* methodology and online data augmentation techniques. The *minus-1db* methodology is based on combining multiple sound banks to build the training dataset, which has shown in-
creasing model generalization capacity [18]. Online data augmentations apply transformations on data samples directly in the training loop, improving data variability and enhancing the model's robustness.

While our system can be run without any latency concerns, it is known that OSC protocol induces variable latency [29]. Developing a dedicated Max/MSP external object with a multi-threaded architecture would provide low latency for audio processing and classification. Besides performance improvements, this would allow inexperienced users in machine learning to use our system for their piece of music or live improvisation by directly instantiating the object in Max.

This study has focused on using flute IPTs to guide co-improvising musical agents. However, future developments of this work on multidimensional label management could ensure the integration of any user-specified labels. We think that the detection of vocal prosody, based on multiple levels of syllabic and semantic segmentation, integrated with word segmentation systems [33, 34], may open up relevant possibilities in the context of using software such as Somax2 in vocal and textual improvisation and composition contexts. Previous research showed that a multi-dimensional probabilistic model had already been adapted to the factor oracle [35] in the case of Omax [36]. However, in the case of Somax2's heuristic guidance model, our approach allows for greater management of navigation for musical purposes. This is because of the possibility of weighting the activity profiles of each of its activation levels. This implementation now runs specific scenarios based on the succession of detected IPT classes and prepares them in advance for longer-term scenarios.

## 6. CONCLUSIONS

In this paper, we presented a novel approach to guide co-improvising musical agents of Somax2 by using the real-time recognition of flute Instrumental Playing Techniques (IPTs). To make this possible, we designed a system for real-time recognition of flute IPTs and detailed how recognition results are integrated into Somax2 event generation. We assessed the performance of our flute IPT recognition system in real-world situations. We found it was accurate and trustworthy in real-time, with an accuracy of 91.37%, a macro F1-score of 83.07%, and a latency ranging from 10.03ms, with one Somax2 agent, to 11.67ms, when three Somax2 agents were improvising. Then, a professional flutist experienced our recognition system connected to Somax2, where our proposed interaction strategies were applied without visual feedback. The flutist felt a cohesive interaction when the IPT labels were integrated into Somax2. While our work stands out as a solid end-to-end workflow for real-time IPT integration in human-computer interactive systems, we think further improvements can be made for label integration in the event generation of co-improvising musical agents. Managing any kind of label would allow the users actually to create better label taxonomies related to their musical material and, as a result, to expand the creative possibilities. Nevertheless, we hope that our approach can help provide new perspectives to extend the current interactive music systems for music improvisation and composition.

## Acknowledgments

This research is supported by the European Research Council (ERC) as part of the Raising Co-creativity in Cyber-Human Musicianship (REACH) Project directed by Gérard Assayag, under the European Union's Horizon 2020 research and innovation program (GA #883313). Funding support for this work was provided by a Japanese Ministry of Education, Culture, Sports, Science and Technology (MEXT) scholarship to Nicolas Brochec. Authors thank Sophie Deshayes and Simone Conforti for their flute mastery and precious feedback, and Joakim Borg for his continuous Somax2 expertise.

## 7. REFERENCES

[1] G. E. Lewis, “Too many notes: Computers, Complexity and Culture in Voyager,” *Leonardo Music Journal*, vol. 10, pp. 33–39, 2000.

[2] G. Assayag, G. Bloch, M. Chemillier, A. Cont, and S. Dubnov, “Omax brothers: a dynamic topology of agents for improvization learning,” in *Proceedings of the 1st ACM workshop on Audio and music computing multimedia*, 2006, pp. 125–132.

[3] K. Tatar and P. Pasquier, “Musical agents: A typology and state of the art towards Musical Metacreation,” *Journal of New Music Research*, vol. 48, no. 1, pp. 56–105, 2019.

[4] D. Schwarz, G. Beller, B. Verbrugghe, and S. Britton, “Real-Time Corpus-Based Concatenative Synthesis with CataRT,” in *9th International Conference on Digital Audio Effects (DAFx)*, Montreal, Canada, Sep. 2006, pp. 279–282.

[5] C. Canonne and N. B. Garnier, “Cognition and Segmentation In Collective Free Improvisation: An Exploratory Study,” in *International Conference on Music Perception and Cognition*, 2012.

[6] P. Mouawad and S. Dubnov, “On Modeling A ff ect in Audio with Non-Linear Symbolic Dynamics,” *Advances in Science, Technology and Engineering Systems Journal*, vol. 2, no. 3, pp. 1727–1740, 2017.

[7] D. Borgo, “Sync or Swarm: Musical Improvisation and the Complex Dynamics of Group Creativity,” in *Lecture Notes in Computer Science*, 2006.

[8] J. Nika, M. Chemillier, and G. Assayag, “ImproteK: introducing scenarios into human-computer music improvisation,” *ACM Computers in Entertainment*, Jan. 2017.

[9] J. Nika, A. Muller, J. Borg, G. Assayag, and M. Ostrowski, “Dicy2 for Max,” *Ircam UMR STMS 9912, Tech. Rep.*, Dec. 2022. [Online]. Available: https://hal.science/hal-03892611

[10] J. Nika, D. Bouche, J. Bresson, M. Chemillier, and G. Assayag, “Guided improvisation as dynamic calls to an offline model,” in *Sound and Music Computing (SMC)*, Maynooth, Ireland, Jul. 2015.
[11] J. Nika, K. Déguernel, A. Chemla, E. Vincent, G. Assayag et al., “DYCI2 agents: merging the "free", "reactive", and "scenario-based" music generation paradigms,” in *International Computer Music Conference*. Shangai, China, 2017.

[12] G. Assayag, L. Bonnasse-Gahot, and J. Borg, “Cocreative Interaction: Somax2 and the REACH Project,” *Computer Music Journal*, vol. 46, no. 4, pp. 7–25, Dec. 2022.

[13] C. Wang, E. Benetos, V. Lostanlen, and E. Chew, “Adaptive scattering transforms for playing technique recognition,” in *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 30, pp. 1407–1421, 2022.

[14] D. Li, M. Che, W. Meng, Y. Wu, Y. Yu, F. Xia, and W. Li, “Frame-level multi-label playing technique detection using multi-scale network and self-attention mechanism,” in *ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2023, pp. 1–5.

[15] L. Su, H.-M. Lin, and Y.-H. Yang, “Sparse modeling of magnitude and phase-derived spectra for playing technique classification,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 22, no. 12, pp. 2122–2132, 2014.

[16] Y.-P. Chen, L. Su, Y.-H. Yang et al., “Electric Guitar Playing Technique Detection in Real-World Recording Based on F0 Sequence Pattern Recognition.” in *ISMIR*, 2015, pp. 708–714.

[17] A. Martelloni, A. P. McPherson, and M. Barthet, “Real-time Percussive Technique Recognition and Embedding Learning for the Acoustic Guitar,” *arXiv preprint arXiv:2307.07426*, 2023.

[18] J.-F. Ducher and P. Esling, “Folded CQT RCNN for real-time recognition of instrument playing techniques,” in *International Society for Music Information Retrieval*, 2019.

[19] N. Brochec and T. Tanaka, “Toward Real-Time Recognition of Instrumental Playing Techniques for Mixed Music: A Preliminary Analysis,” in *International Computer Music Conference (ICMC 2023)*, 2023.

[20] N. Brochec, T. Tanaka, and W. Howie, “Microphone-based Data Augmentation for Automatic Recognition of Instrumental Playing Techniques,” in *International Computer Music Conference (ICMC 2024)*, Seoul, South Korea, Jul. 2024. [Online]. Available: https://hal.science/hal-04642673

[21] N. H. Fletcher and T. D. Rossing, *The physics of musical instruments*. Springer Science & Business Media, 2012.

[22] J. Meyer, *Acoustics and the performance of music: Manual for acousticians, audio engineers, musicians, architects and musical instrument makers*. Springer Science & Business Media, 2009.

[23] C. Levine and C. Mitropoulos-Bott, *The Techniques of Flute Playing I/Die Spieltechnik der Flöte I*. Bärenreiter-Verlag, 2019.

[24] N. Brochec and W. Howie, “GFDatabase: A Database of Flute Playing Techniques,” Oct. 2024. [Online]. Available: https://doi.org/10.5281/zenodo.13898549

[25] C. E. Cella, D. Ghisi, V. Lostanlen, F. Lévy, J. Fineberg, and Y. Maresz, “OrchideaSOL: a dataset of extended instrumental techniques for computer-aided orchestration,” *arXiv preprint arXiv:2007.00763*, 2020.

[26] S. Wei, S. Zou, F. Liao et al., “A comparison on data augmentation methods based on deep learning for audio classification,” in *Journal of physics: Conference series*, vol. 1453, no. 1. IOP Publishing, 2020, p. 012085.

[27] V. Lostanlen and C.-E. Cella, “Deep convolutional networks on the pitch spiral for musical instrument recognition,” *arXiv preprint arXiv:1605.06644*, 2016.

[28] Y. Han, J. Kim, and K. Lee, “Deep convolutional neural networks for predominant instrument recognition in polyphonic music,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 25, no. 1, pp. 208–221, 2016.

[29] M. Wright, “Open Sound Control: an enabling technology for musical networking,” *Organised Sound*, vol. 10, no. 3, pp. 193–200, 2005.

[30] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in *Proceedings of the thirteenth international conference on artificial intelligence and statistics*. JMLR Workshop and Conference Proceedings, 2010, pp. 249–256.

[31] J. Borg, M. Fiorini, and G. Assayag, “Somax2 User’s Guide,” *STMS-IRCAM*, Technical Report, 2024.

[32] A. De Cheveigné and H. Kawahara, “YIN, a fundamental frequency estimator for speech and music,” *The Journal of the Acoustical Society of America*, vol. 111, no. 4, pp. 1917–1930, 2002.

[33] N. Obin, F. Lamare, and A. Roebel, “Syll-O-Matic: an Adaptive Time-Frequency Representation for the Automatic Segmentation of Speech into Syllables,” in *IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)*, Vancouver, Canada, May 2013.

[34] N. Obin, J. Beliao, C. Veaux, and A. Lacheret, “SLAM: Automatic Stylization and Labelling of Speech Melody,” in *Speech Prosody*, France, May 2014, p. 246.

[35] G. Assayag and S. Dubnov, “Using Factor Oracles for machine Improvisation,” *Soft Computing*, vol. 8, no. 9, pp. 604–610, 2004.

[36] K. Déguernel, E. Vincent, and G. Assayag, “Using Multidimensional Sequences For Improvisation In The OMax Paradigm,” in *13th Sound and Music Computing Conference*, Hamburg, Germany, Aug. 2016.