# Real-Time Musical Instruments Recognition for
Scenography Purposes

Hugues Kadi, Mathis Devidal, Romain Michon, Johan Erbani

<figure><img src="image_1.png" /></figure>

## To cite this version:

Hugues Kadi, Mathis Devidal, Romain Michon, Johan Erbani. Real-Time Musical Instruments Recognition for Scenography Purposes. Linux Audio Conference 2025, Jun 2025, Lyon, France. hal-05096057

HAL Id: hal-05096057

https://hal.science/hal-05096057v1

Submitted on 3 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# REAL-TIME MUSICAL INSTRUMENTS RECOGNITION FOR SCENOGRAPHY PURPOSES

Hugues Kadi

Institut National des Sciences Appliquées (INSA)
Lyon, France
huguesk@attssystem.fr

Romain Michon

INRIA, CITI Laboratory, INSA Lyon, GRAME-CNCM
Villeurbanne, France
romain.michon@inria.fr

## ABSTRACT

In recent years, Artificial Intelligence (AI) has been increasingly applied across various domains, including music, where its adoption has grown significantly.

This study leverages modern deep learning architectures, implemented using the Rust programming language, to analyze audio streams. Specifically, the objective is to develop a deep learning model capable of real-time musical instrument recognition, optimized for execution on cost-effective hardware and published as an open-source tool.

This research aligns closely with the Real-Time Music Information Retrieval (RT-MIR) domain.

## 1. INTRODUCTION

### 1.1. Goals

Our primary objective is to provide a preliminary contribution to the field of musical instrument recognition, a specialized area within Music Information Retrieval (MIR). MIR has been extensively explored using AI methodologies, but real-time polyphonic instrument recognition remains a challenging and evolving domain.

This work is focused on investigating the potential of deep learning techniques to develop an initial, open-source solution for this task. To the best of our knowledge, there are currently no open-source solutions or commercial products specifically designed to address this task. While some AI-based tools are capable of separating instruments from audio signals—such as those used for generating STEMS—they typically do not operate under real-time constraints. Moreover, the goal of this work is not to extract the actual waveforms of individual instruments, which is a significantly more complex task.

The integration of this technology into scenography tools is considered a long-term and complex goal, serving as a practical motivation for the research. However, it is important to first clarify the concept of scenography and the potential benefits such a tool could bring to this field. Scenography involves designing and setting up the visual and spatial environment in which a performance takes place, in order to support and enhance the artistic expression. In the context of musical performances, scenography primarily relies on the use of lighting and pyrotechnic effects to complement the music. These effects are typically managed by professionals who either pre-program them to occur at specific moments or control them manually during the performance. The proposed AI tool could enable a new level of automation by triggering these scenographic effects based on the real-time detection of specific instruments. For example, the QLab software [1], which provides an API for cue management, could be used to automatically trigger lighting cues when a piano is detected. This capability would be particularly beneficial in

Mathis Devidal

Institut National des Sciences Appliquées (INSA)
Lyon, France
mathis.devidal@gmail.com

Johan Erbani

Institut National des Sciences Appliquées (INSA)
Lyon, France
johan.leydet@insa-lyon.fr

improvised or semi-structured performances, where traditional pre-programming is less feasible.

The focus of this study is to design a lightweight, easily trainable, and efficient model capable of recognizing multiple musical instruments in real-time from polyphonic audio streams. This solution is intended as a foundation for further improvements and evolution, encouraging contributions from the community to refine and expand its capabilities.

### 1.2. Constraints

Our research project is bounded by two main constraints:

*The real-time constraint* is associated with the necessity of achieving rapid instrument identification to maintain a real-time experience. Initially, a buffer size of 10 ms was chosen based on an interpretation of the latency limit mentioned in a study on Real-Time Percussive Technique Recognition [2]. However, this interpretation proved to be overly simplistic, as it did not account for the loss of information from waveforms with wavelengths exceeding 10ms. Consequently, this buffer size was deemed unsuitable for our purposes, as it failed to yield meaningful results. Instead, a more practical limit of approximately 30ms was adopted, aligning with established guidelines for maintaining a real-time perception [3]. It is important to note that inference time is not the only contributor to latency; the model must be sufficiently lightweight to ensure rapid inference, and additional delays may arise from the response times of scenography tools interfacing with the system [1].

*The hardware constraint* is associated with the deliberate choice of ensuring accessibility, making it unreasonable to require the use of HPC<fnref n="1" /> clusters for training or inference. The experiments were designed to be reproducible on consumer-grade devices. While training may benefit from gaming-level computers equipped with discrete GPUs,<fnref n="2" /> the inference process is intended to be feasible on systems without discrete GPUs, ranging from modern Raspberry Pi devices to older laptops.

### 1.3. State Of The Art

If deep learning has already reached impressive results in the field of image recognition, it is less the case in the field of musical instruments recognition. However, a specific domain emerged in the 2010s, MIR which stands for Music Information Retrieval. This domain even became more specific with RT-MIR which adds the Real-Time component of it.

Several studies have been identified [4] [5], dating back to the early 2010s and using deep-learning techniques, and more specifically CNN. These works are considered preliminary, as

<footnote n="1">¹High Performance Computing</footnote>

<footnote n="2">²Graphics Processing Unit</footnote>
they propose simple yet effective solutions. More recent publications have demonstrated improved inference performance, although the architectural details of the models employed are often not thoroughly documented.

In recent years, alternative approaches have been proposed, including hierarchical methods [6] (Hornbostel-Sachs classification) and various CNN-based techniques [7] [4] [2] [8]. Some studies have also explored the possibility of bypassing feature extraction preprocessing, such as FFT, in favor of raw data processing [9] (deep 34 layers CNN, raw waveform).

Datasets such as NSynth [10] (monophonic), CoCoPops [11] (hum and harm files), and Slakh2100 [12] (polyphonic) were considered. The selection of an appropriate dataset proved to be a challenging aspect of this research, and further details regarding this process are discussed in subsequent sections.

In the field of deep learning, Python is often considered the standard choice, as highlighted in [13]. However, after careful consideration, Rust was chosen for this project §1.4.

## 1.4. Why Rust For Real-Time Audio Processing

Initially, Python was considered as the primary programming language for this project. However, a transition to Rust, along with the Burn ML/DL framework [14], was made for several reasons:

* The performance and safety features of Rust were deemed essential for the development of a real-time application. These capabilities are attributed to Rust's lower-level nature and its robust memory safety mechanisms;

* The Rayon framework, available in Rust, facilitates efficient parallelization of the preprocessing pipeline applied to the datasets;

* Although not a primary objective of this project, Burn's ability to operate in a no_std environment allows for bare-metal execution of a significant portion of the process;

* A limitation of the Rust/Burn combination is its relative immaturity compared to Python-based alternatives. However, this is mitigated by the support of a rapidly growing community.

## 1.5. Paper Organization

This paper is structured as follows: Section 2 describes our dataset selection criteria and the implemented preprocessing pipeline. Section 3 presents our methodology for quantitative performance evaluation. Section 4 details the iterative development process leading to our final model architecture. Finally, we conclude with an analysis of experimental results and propose directions for future research.

# 2. THE DATASET

## 2.1. Choosing Inputs

Identifying a suitable dataset is often a challenging task in AI research. In the context of audio processing, where datasets are less prevalent compared to fields like computer vision, this challenge is even more pronounced. As an initial approach, we used the NSynth dataset [10], which includes 305,979 musical notes, each characterized by unique pitch, timbre, and envelope. These notes are stored as 4-second-long WAV files. To augment the dataset, we divided each sample into smaller segments of 10ms. This segmentation aimed to increase the effective number of training samples and enhance the model's ability to generalize, as the model was initially designed to perform real-time inference on 10ms audio chunks.

Subsequently, we transitioned to the Slakh dataset [12], which offers a more complex and comprehensive collection of audio data. The Slakh dataset consists of approximately 145

hours of music stored in FLAC format. Each track includes both the full mix and individual stems corresponding to the separate instruments played. This structure enabled us to detect the presence of specific instruments in the mix by analyzing the stems in parallel. If a stem contained non-zero waveform data, the corresponding instrument was deemed present in the audio segment. Initially, we applied the same 10ms chunking strategy as with NSynth. However, the primary advantage of the Slakh dataset lies in its stem separation, which facilitated the development of a robust preprocessing pipeline.

## 2.2. Our Preprocessing Pipeline

<figure><img src="image_3.png" /><figcaption>Figure 1: Preprocessing pipeline. Looping through all tracks' directory then cleaning data and applying FFT.</figcaption></figure>

The preprocessing pipeline is a significant component of this research, as it evolved substantially throughout the project. While the initial objective was to extract relevant features from the dataset, it became increasingly evident that additional steps were necessary. Specifically, a considerable portion of the preprocessing effort was dedicated to addressing dataset imbalance.

The following section focuses on the preprocessing pipeline applied to the Slakh dataset, which serves as the primary dataset for this study. This pipeline builds upon and extends the simpler approach initially developed for the NSynth dataset. A visual representation of the complete pipeline is provided in Fig. 1.

## 2.3. Reduce The Dataset

As previously mentioned, one of the primary objectives of this research was to ensure that the model could be trained on consumer-grade hardware. While the hardware used during this study included a professional workstation equipped with an A3000 GPU and an 11th generation Intel i7 CPU with 16 cores, it is important to note that even such machines were incapable of processing the entire dataset. The Slakh dataset comprises approximately 145 hours of music, making it impractical to train on the full dataset due to the excessive computational time required.
To tackle this challenge, we devised a strategy to reduce the over-all size of the input dataset while maintaining sufficient diversity for the model to generalize effectively. This was achieved by introducing a set of hyperparameters specifically designed for dataset reduction:

*   **CHUNK JUMP:** This parameter, expressed in seconds, defines the temporal interval between consecutive audio chunks selected for training.

*   **TRACKS_LIMIT:** This parameter specifies the number of tracks to be included in the training process.

*   **TRACKS_START:** This parameter indicates the duration, in seconds, to be skipped at the beginning of each track before selecting audio segments for training.

*   **TRACKS_LENGTH:** This parameter determines the duration, in seconds, of the audio segment to be used for training after the initial skip defined by TRACKS_START.

These hyperparameters were instrumental in reducing the dataset size while maintaining a diverse set of inputs, thereby enabling the model to achieve a balance between computational feasibility and generalization capability.

## 2.4. Extract Features

Feature extraction was primarily left to the model, which is common in deep learning. However, after conducting several experiments, two key preprocessing transformations were applied to the input data:

*   The audio signal was downsampled by a factor of 8. This step effectively reduced the dimensionality of the input data while retaining sufficient auditory information for instrument recognition. The downsampling factor was empirically determined to ensure minimal perceptual loss of instrument characteristics by ear. Preliminary tests indicated that this reduction had negligible impact on the model's performance.

*   A Fast Fourier Transform (FFT) was applied to transform the audio signal from the time domain to the frequency domain. This transformation enhances the model's ability to identify patterns in the spectral representation, which are more distinct and informative for the task of instrument recognition.

## 2.5. The Balancing Logic

The final component of our preprocessing pipeline is the dataset balancing mechanism. Ensuring a balanced dataset was critical to mitigate overfitting and to enhance the model's ability to generalize to unseen inputs. Initial experiments revealed significant overfitting issues, particularly for instruments that were overrepresented in the dataset.

To address this issue, we implemented a straightforward yet effective balancing strategy. Consider a scenario with $I = 5$ instruments: $X^1, X^2, X^3, X^4$, and $X^5$. The dataset is processed in $I$ iterations, with each iteration introducing a small variation to ensure that the same tracks are analyzed but not identical audio segments. During the first iteration, items containing only one instrument are generated. In the second iteration, items containing two instruments are created, and so forth, until the $I$-th iteration, where items with all $I$ instruments are included.

For each iteration, instruments not selected for analysis are treated as noise and removed. The remaining instruments are shuffled, and only the first instrument is retained in the first iteration, the first two instruments in the second iteration, and so on. Additionally, counters are maintained to track the frequency of each instrument across the entire output dataset. Instruments are excluded from further inclusion if their frequency exceeds

the average frequency of all instruments by 50 or more, thereby preventing overrepresentation.

This iterative and controlled approach ensures a balanced representation of instruments in the dataset. A visual representation of this balancing strategy is provided in Fig. 2.

<figure><img src="image_4.png" /><figcaption>Figure 2: Balancing strategy. Included in preprocessing. Basically going through the dataset multiple times without reading the same chunks.</figcaption></figure>

# 3. MEASURING PERFORMANCE

In this section, we present our methodology for evaluating our model's performance. Multiple complementary metrics were developed and used throughout our research to ensure robust assessment of the model's capabilities. We explore both traditional evaluation approaches and novel metrics specifically tailored to address the unique challenges of multi-label instrument recognition in real-time audio processing.

## 3.1. A Naive Approach

An initial approach involved consulting the Burn documentation, which guided us toward employing the Hamming score. This metric is commonly used in multi-label classification tasks and is defined mathematically as follows:

$$HammingScore = \frac{1}{N} \sum_{i=1}^{N} \frac{|y_i \cap \hat{y}_i|}{|y_i \cup \hat{y}_i|}$$

Where $N$ is the total number of samples, $|y_i \cap \hat{y}_i|$ is the intersection of the true labels $y_i$ and predicted labels $\hat{y}_i$ for the $i$-th sample and $|y_i \cup \hat{y}_i|$ is the union of the true and predicted labels for the $i$-th sample.

We combine it with the binary cross entropy loss whose formula is:

$$BCE = -\frac{1}{N} \sum_{i=1}^{N} [y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i)]$$

Where $N$ is the total number of samples, $y_i$ the true label for the $i$-th sample, $\hat{y}_i$ is the predicted probability for the $i$-th sample. Explanations about its usage are given in 4.5.

However, the Hamming score exhibits certain limitations, particularly in the context of imbalanced datasets. It tends to penalize rare labels disproportionately and emphasizes the accuracy of individual labels rather than the correctness of the entire prediction vector. These shortcomings occasionally led to misleadingly high scores (exceeding 85%), even when the model's predictions were biased towards the most frequent instrument
in the dataset (e.g., the guitar), rather than providing accurate multi-label predictions.

## 3.2. Our Own Metric: Exact Match Ratio

To address the limitations of the initial metric, we introduced an alternative approach, referred to as the Exact Match Ratio (EMR). This metric is conceptually straightforward and is defined as follows:

$$EMR = \frac{FT}{N}$$

Where N is the total number of samples and FT the number of sample *FullyTrue* (where all labels were guessed correctly).

While this metric provides additional insights, it remains limited in its ability to capture the full performance of the model. Furthermore, unlike the Hamming score, the Exact Match Ratio can yield low values even when the model demonstrates meaningful predictive capabilities.

## 3.3. Our Great Saviour: The TCM

One common performance criterion for a deep learning model is the confusion matrix. Such a tool enables one to:

* identify which instruments are confused with which ones

* extract other performance criterion, like the accuracy, the class-wise recall and precision (and hence F-scores, discussed later)

Confusion matrices are traditionally designed for single-label classification tasks. To extend this concept to multi-label classification, we have implemented the Transport Confusion Matrix (TCM), as proposed by Johan Erbani in his work [15]. The TCM enables the computation of confusion matrices and associated metrics for multi-label classification problems.

The TCM is grounded in the principles of optimal transport theory. While a detailed exploration of its mathematical properties is beyond the scope of this paper, interested readers are encouraged to consult the referenced work for an in-depth analysis. For the purposes of this study, we provide a concise summary of the key definitions and formulations used in the implementation.

Giving the following notations:

* C, the number of instruments

* N, the number of tracks samples used to train our model

* y, a vector in {0, 1}^C giving the present instruments for a track sample

* ŷ, a vector in {0, 1}^C giving the predicted instruments for a track sample

* ||·||_1, the l₁ norm

* min(u, v), a vector having, element-wise, the minimum of u and v

* diag(v), the diagonal matrix whose diagonal is equal to vector v

* ⊗, the outer product

* λ(y, ŷ), a contribution weight

For a single observation, represented by a pair of y and ŷ, we define:

$$m = \min(\frac{y}{||y||_1}, \frac{ŷ}{||ŷ||_1})$$

$$D = \mathrm{diag}(m)$$

$$\bar{D} = \frac{(\frac{y}{||y||_1} - m) \otimes (\frac{ŷ}{||ŷ||_1} - m)}{||\frac{y}{||y||_1} - m||_1}$$

$$\pi^*(y, \hat{y}) = \begin{cases} D & y = \hat{y} \\ D + \bar{D} & y \neq \hat{y} \end{cases}$$

This is true for one observation. Then, a TCM is no more than the sum of the previous $\pi^*(y, ŷ)$ matrix for all observations. With a superscript $n \in [0, N]$ indicating the index of the observation in our train set, we finally have:

$$TCM = \sum_{n=0}^{N} \lambda(y^n, \hat{y}^n) \pi^*(y^n, \hat{y}^n)$$

The contribution weight could allow us to put more importance on the predictions, the targets or having both the same weight. We have decided to give predictions and targets the same weight, so $\lambda(y^n, \hat{y}^n) = 1$. From this TCM, we can extract the accuracy:

$$accuracy = \frac{\sum_{k=1}^{C} TCM_{kk}}{\sum_{i,j=1}^{C} TCM_{ij}}$$

As told before, recall and precision are computed class wise. For a class (instrument) k, we have:

$$recall_k = \frac{TCM_{kk}}{\sum_{j=1}^{C} TCM_{kj}}$$

$$precision_k = \frac{TCM_{kk}}{\sum_{i=1}^{C} TCM_{ik}}$$

$$Fscore_k = 2 \times \frac{recall_k \times precision_k}{recall_k + precision_k}$$

As observed, the F score represents the harmonic mean of recall and precision, computed on a per-class basis. Once the C F scores are calculated, they can be aggregated using various methods. These include macro, weighted, and micro averaging. In this study, macro averaging is employed, leveraging the preprocessing pipeline, which ensures class balance within the dataset. Consequently, the final formulation is as follows:

$$Fscoremacro = \frac{1}{C} \sum_{k=1}^{C} Fscore_k$$

We now have three metrics: the TCM, the accuracy and the macro F score.

# 4. EVOLUTION OF MODEL ARCHITECTURE

## 4.1. Using Time Domain

Initially, we aimed to use the inherent feature extraction capabilities of deep learning by minimizing the amount of preprocessing applied to the input data. Consequently, apart from reducing the dataset size, we provided the model with raw audio waveforms as input for training.

However, this approach quickly proved to be ineffective. While deep learning is capable of autonomously extracting features, the raw waveform data did not provide sufficient relevance under the specific conditions of our experiments, leading to sub-optimal performance.

## 4.2. Using Frequency Domain

The rationale for transitioning from the time domain to the frequency domain is grounded in the following observation. In the frequency domain, represented by a spectrum, it becomes significantly more intuitive for a human observer to distinguish between different instruments. This is due to the visibility of their characteristic frequencies and harmonics. Based on this premise, it was hypothesized that a deep learning model might similarly benefit from the enhanced distinguishability offered by the frequency domain, thereby improving its ability to identify and classify musical instruments.

During our experiments with frequency input data, we opted to use Convolutional Neural Networks (CNNs) due to their established efficiency in processing image data. A spectrogram,
which represents the frequency content of an audio signal over time, can be interpreted as a two-dimensional image. This characteristic makes CNNs particularly well-suited for analyzing such data. To leverage this capability, we initially applied CNNs to colored spectrograms, as they were the most intuitive and accessible representation of the frequency domain available to us at the time.

Subsequently, based on expert recommendations, we explored the use of Mel-Frequency Cepstral Coefficients (MFCCs), which are widely regarded as effective features in traditional machine learning for audio processing tasks. However, the implementation of MFCCs presented several technical challenges, which hindered their integration into our pipeline. Ultimately, we reverted to employing the Fast Fourier Transform (FFT) as the primary feature extraction method. This decision was motivated by the FFT's computational efficiency and its ability to provide sufficiently relevant information about the audio signal, as previously discussed.

### 4.3. Enjoying All Possibilities

Initially, the model was trained using 10ms audio samples, sampled at a frequency of 44.1kHz. However, as previously discussed, this buffer size was found to be suboptimal, as it resulted in the loss of significant low-frequency information, effectively acting as a high-pass filter.

To address this limitation and recover the lost low-frequency content, the criteria for determining the buffer size were revised. First, a downsampling factor of 8 was introduced to reduce the input size, thereby making the data more computationally efficient to process. Additionally, based on expert recommendations, the Fast Fourier Transform (FFT) was configured to use 1024 points.

Given these adjustments, the new buffer size can be calculated as follows:

$$ 
\begin{aligned}
sample\_duration &= FFT_{size} \times downsampling\_factor \times \frac{1}{f} \\
&= 1024 \times 8 \times \frac{1}{44100} \approx 186ms
\end{aligned}
$$

Furthermore, given the symmetric nature of the resulting FFT around zero, only the positive frequency components were retained.

### 4.4. Our Final Model Architecture

The final architecture of our model is illustrated in Fig. 3. Throughout the course of this research, several alternative approaches were explored, including the use of transformers and ResNet architectures. However, we ultimately opted for a more traditional convolutional neural network (CNN) design.

This decision was motivated by both practical and performance considerations. The alternative architectures proved to be significantly more complex to implement and required substantially longer training times. In contrast, the CNN-based approach provided a balance between computational efficiency and predictive performance, yielding satisfactory results in specific scenarios, as detailed in the next section.

### 4.5. About Training Procedure

Throughout all architecture evaluations, we maintained the same training hyperparameters and did not attempt to optimize them for performance.

The optimizer employed was Adam, configured with the following parameters:

* $\beta_1 = 0.9$

* $\beta_2 = 0.999$

<figure><img src="image_5.png" /><figcaption>Figure 3: Model architecture. Simple 8 layers CNN.</figcaption></figure>

$$ \epsilon = 1.0e^{-5} $$

No weight decay (i.e., regularization) or gradient clipping was applied during training.

Additionally, no learning rate scheduling strategy was implemented; instead, a fixed initial learning rate was provided, and its adaptation was managed internally by the Burn framework.

The loss function used was the binary cross-entropy loss, as previously defined in Section 3.1. This loss was selected due to its widespread use in multi-label classification tasks, where it is beneficial that the loss for each class is computed independently of the others. This property is appropriate for our application, as the presence of one instrument (e.g., drums) does not necessarily imply the presence of another (e.g., piano).

**Note:** In the following sections, particularly in the tables and curves, the term "Loss" refers specifically to the binary cross-entropy loss.

### 4.6. Obtained Results

**Important**: for this part, our parameters were the following:

* `CHUNK JUMP` = 1000

* `TRACKS_LIMIT` = 64

* `TRACKS_START` = 10000

* `TRACKS_LENGTH` = 70000

* `EPOCHS` = 5

* `LEARNING_RATE` = $1.0e^{-5}$

The performance of the model is influenced by the number of instruments included during training. In addition to the tables, we provide loss curves for both the training and validation phases corresponding to each instrument set, generated using Python and Matplotlib. These curves provide insight into the model's capacity to learn and generalize. Other performance metrics are omitted, as they follow similar trends. To enhance readability and reduce noise, the curves were smoothed using a
<table><caption>Table 1: Results for 2 instruments: piano and drums</caption><thead><tr><th>Split</th><th>Metric</th><th>Min</th><th>Epoch</th><th>Max</th><th>Epoch</th></tr></thead><tbody><tr><td rowspan="4">Train</td><td>Loss</td><td>0.299</td><td>3</td><td>NaN</td><td>5</td></tr><tr><td>Hamming</td><td>80.468</td><td>1</td><td>87.840</td><td>5</td></tr><tr><td>EMR (%)</td><td>62.112</td><td>1</td><td>76.445</td><td>5</td></tr><tr><td>Accuracy</td><td>0.807</td><td>1</td><td>0.878</td><td>5</td></tr><tr><td></td><td>F score</td><td>0.800</td><td>1</td><td>0.872</td><td>5</td></tr><tr><td rowspan="4">Valid</td><td>Loss</td><td>0.478</td><td>4</td><td>0.589</td><td>2</td></tr><tr><td>Hamming</td><td>73.863</td><td>2</td><td>81.873</td><td>5</td></tr><tr><td>EMR (%)</td><td>53.376</td><td>2</td><td>66.471</td><td>5</td></tr><tr><td>Accuracy</td><td>0.738</td><td>2</td><td>0.819</td><td>5</td></tr><tr><td></td><td>F score</td><td>0.727</td><td>2</td><td>0.815</td><td>5</td></tr><tr><td rowspan="3">Test</td><td>EMR (%)</td><td>/</td><td>/</td><td>79.340</td><td>/</td></tr><tr><td>Accuracy</td><td>/</td><td>/</td><td>0.893</td><td>/</td></tr><tr><td>F score</td><td>/</td><td>/</td><td>0.892</td><td>/</td></tr></tbody></table>

<table><caption>Table 2: Results for 3 instruments: Piano Guitar Drums</caption><thead><tr><th>Split</th><th>Metric</th><th>Min</th><th>Epoch</th><th>Max</th><th>Epoch</th></tr></thead><tbody><tr><td rowspan="4">Train</td><td>Loss</td><td>0.438</td><td>5</td><td>0.559</td><td>1</td></tr><tr><td>Hamming</td><td>70.430</td><td>1</td><td>78.020</td><td>5</td></tr><tr><td>EMR (%)</td><td>34.209</td><td>1</td><td>45.159</td><td>5</td></tr><tr><td>Accuracy</td><td>0.679</td><td>1</td><td>0.740</td><td>5</td></tr><tr><td></td><td>F score</td><td>0.674</td><td>1</td><td>0.735</td><td>5</td></tr><tr><td rowspan="4">Valid</td><td>Loss</td><td>0.633</td><td>5</td><td>0.690</td><td>1</td></tr><tr><td>Hamming</td><td>49.789</td><td>1</td><td>57.754</td><td>5</td></tr><tr><td>EMR (%)</td><td>11.151</td><td>1</td><td>21.383</td><td>4</td></tr><tr><td>Accuracy</td><td>0.429</td><td>1</td><td>0.525</td><td>4</td></tr><tr><td></td><td>F score</td><td>0.213</td><td>1</td><td>0.434</td><td>5</td></tr><tr><td rowspan="3">Test</td><td>EMR (%)</td><td>/</td><td>/</td><td>7.188</td><td>/</td></tr><tr><td>Accuracy</td><td>/</td><td>/</td><td>0.496</td><td>/</td></tr><tr><td>F score</td><td>/</td><td>/</td><td>0.418</td><td>/</td></tr></tbody></table>

sliding window (mean) with a width corresponding to 3% of the total number of batches.

Initially, the final architecture was evaluated using two distinct instruments: drums and piano. The results, presented in Table 1, indicate relatively strong performance in the test phase. This outcome can be attributed to the distinct characteristics of these two instruments, which likely made them easier for the model to differentiate. The corresponding training loss curve, shown in Fig. 4, indicates effective learning behavior, as evidenced by a downward trend in loss values, although the difference between the highest and lowest values remains relatively small compared to what can be observed for other AIS tasks. Similarly, the validation loss exhibits a general decrease across epochs, suggesting improved generalization over time.

The NaN value observed in the table is a result of an exceptional case encountered within the Rust framework [14] used for this study. Unfortunately, this issue could not be resolved during the course of the research.

<figure><img src="image_6.png" /><figcaption>Figure 4: Loss evolution during training and validation phases for piano and drums</figcaption></figure>

Subsequently, a model was trained to recognize three instruments: piano, guitar, and drums. As shown in Table 2, the results demonstrate a decline in performance compared to the two-instrument training scenario. This reduction in accuracy can be attributed to the increased complexity of distinguishing between three instruments, which poses a greater challenge for the model. Figure 5 reflects this difficulty, as the training loss decreases more slowly and remains higher compared to the two-instrument

case shown in Fig. 4. Additionally, the validation loss does not exhibit a substantial overall decline throughout the training process.

**Note:** We present only the loss evolution for the piano, guitar, drums set, as the trends observed for the piano, bass, drums configuration are similar.

<figure><img src="image_7.png" /><figcaption>Figure 5: Loss evolution during training and validation phases for piano, guitar and drums</figcaption></figure>

It is interesting to compare results between piano, guitar and drums with piano, bass and drums.

During the test split, outside the EMR, both accuracy and F score are higher. This could have been explained by the fact that guitar and piano have closer sound properties than bass and piano are on a FFT, but the confusion matrices below (see Fig. 6 & Fig. 7) show that our model confuses more piano and bass than piano and guitar. The brighter the square is, the more the instruments on the corresponding line and column are confused. Of course, the goal is to have a white diagonal and black squares elsewhere. We acknowledge that this behavior warrants a more in-depth analysis; however, we chose to focus our efforts on evaluating the model's performance with respect to the number of instruments considered.

To further validate the observed trend of decreasing performance as the number of instruments increases, we present the results obtained for a model trained on four instruments: piano, brass, guitar, and drums.

The results demonstrate a noticeable decline in performance across all metrics as the number of instruments increases. This
<table><caption>Table 3: Results for 3 instruments: Piano Bass Drums</caption><thead><tr><th>Split</th><th>Metric</th><th>Min</th><th>Epoch</th><th>Max</th><th>Epoch</th></tr></thead><tbody><tr><td rowspan="4">Train</td><td>Loss</td><td>0.348</td><td>5</td><td>0.521</td><td>1</td></tr><tr><td>Hamming</td><td>73.149</td><td>1</td><td>83.992</td><td>5</td></tr><tr><td>EMR (%)</td><td>37.739</td><td>1</td><td>58.774</td><td>5</td></tr><tr><td>Accuracy</td><td>0.703</td><td>1</td><td>0.812</td><td>5</td></tr><tr><td></td><td>F score</td><td>0.699</td><td>1</td><td>0.809</td><td>5</td></tr><tr><td rowspan="4">Valid</td><td>Loss</td><td>0.635</td><td>4</td><td>0.708</td><td>1</td></tr><tr><td>Hamming</td><td>54.718</td><td>1</td><td>66.156</td><td>4</td></tr><tr><td>EMR (%)</td><td>17.458</td><td>1</td><td>28.535</td><td>4</td></tr><tr><td>Accuracy</td><td>0.495</td><td>1</td><td>0.681</td><td>5</td></tr><tr><td></td><td>F score</td><td>0.369</td><td>1</td><td>0.593</td><td>5</td></tr><tr><td rowspan="3">Test</td><td>EMR (%)</td><td>/</td><td>/</td><td>4.862</td><td>/</td></tr><tr><td>Accuracy</td><td>/</td><td>/</td><td>0.587</td><td>/</td></tr><tr><td>F score</td><td>/</td><td>/</td><td>0.468</td><td>/</td></tr></tbody></table>

<figure><img src="image_8.png" /><figcaption>Figure 6: TCM: piano, bass, drums</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 7: TCM: piano, guitar, drums</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 8: Loss evolution during training and validation phases for piano, brass, guitar and drums</figcaption></figure>

observation is supported by Fig. 8 which, despite having a lower total number of training batches compared to previous experiments, shows that the loss does not exhibit a clear decreasing trend during either the training or validation phases.

Additionally, a consistent trend is observed where the performance metrics for the validation and testing splits are lower compared to the training split. This indicates that the model struggles to generalize effectively to unseen data. A potential solution to address this issue would be to increase the number of tracks used during training. However, this approach would necessitate access to higher computational resources, which were not available for this study.

## 4.7. Putting Into Practice

It is important to address the apparent discrepancy between the sample duration of approximately 186 ms and the real-time constraint of 30 ms identified earlier. To reconcile this, a sliding window approach was implemented. This method processes audio samples spanning the most recent 186 ms, effectively creating a continuous stream of overlapping input data.

Inference is performed during a callback function, which is triggered by the audio processing pipeline. This pipeline uses

<table><caption>Table 4: Results for piano, brass, guitar and drums</caption><thead><tr><th>Split</th><th>Metric</th><th>Min</th><th>Epoch</th><th>Max</th><th>Epoch</th></tr></thead><tbody><tr><td rowspan="4">Train</td><td>Loss</td><td>0.544</td><td>5</td><td>0.644</td><td>1</td></tr><tr><td>Hamming</td><td>61.101</td><td>1</td><td>71.689</td><td>5</td></tr><tr><td>EMR (%)</td><td>20.522</td><td>1</td><td>25.504</td><td>5</td></tr><tr><td>Accuracy</td><td>0.573</td><td>1</td><td>0.654</td><td>5</td></tr><tr><td></td><td>F score</td><td>0.567</td><td>1</td><td>0.638</td><td>5</td></tr><tr><td rowspan="4">Valid</td><td>Loss</td><td>0.712</td><td>1</td><td>0.724</td><td>4</td></tr><tr><td>Hamming</td><td>45.708</td><td>2</td><td>49.456</td><td>5</td></tr><tr><td>EMR (%)</td><td>0.0</td><td>1</td><td>3.700</td><td>5</td></tr><tr><td>Accuracy</td><td>0.0</td><td>1</td><td>0.568</td><td>5</td></tr><tr><td></td><td>F score</td><td>0.0</td><td>1</td><td>0.170</td><td>5</td></tr><tr><td rowspan="3">Test</td><td>EMR (%)</td><td>/</td><td>/</td><td>0.665</td><td>/</td></tr><tr><td>Accuracy</td><td>/</td><td>/</td><td>0.365</td><td>/</td></tr><tr><td>F score</td><td>/</td><td>/</td><td>0.134</td><td>/</td></tr></tbody></table>

an external Rust crate, *cpal*, over which we had limited control. Empirical observations revealed that the callback function was invoked at intervals of approximately 42 ± 0.5 ms. Despite the longer sample duration, this implementation provided a seamless real-time experience, as confirmed through subjective evaluation.

## 5. CONCLUSION

As discussed throughout this paper, a promising direction for future work would be to train the model on a larger dataset. Due to computational constraints, a significant portion of the input data was excluded to ensure reasonable training times. Given the model's demonstrated ability to perform passably with a limited number of instruments, achieving medium results on the training dataset but only poor performance on the validation and testing datasets, it is reasonable to hypothesize that training on the entire dataset could substantially improve the model's generalization capabilities. However, this would inevitably lead to a significant increase in training time, which was beyond the scope of this study, as the primary objective was to avoid reliance on high-performance computing clusters.

Another avenue for future exploration would involve training and testing the model using real instrument recordings instead of synthetic audio data. This would provide valuable insights into the practical applicability of the proposed solution in real-world scenarios and could further enhance its usability.

Additionally, with more time and resources, it would be worthwhile to revisit the potential of leveraging temporal data from the audio inputs. Specifically, exploring the use of Recurrent Neural Networks as an alternative to Convolutional Neural Networks could yield interesting results. Furthermore, examining research in the domain of voice recognition, which has faced similar challenges for a longer period, may provide valuable methodologies and insights applicable to this work.
This research represents a preliminary exploration into a cutting-edge domain of artificial intelligence, specifically within the field of deep learning. We successfully developed a straightforward model architecture capable of recognizing a limited set of musical instruments. While the implementation required significant effort, the final solution relies on fundamental techniques:

* The model architecture consists of a simple design featuring eight 1D convolutional layers, complemented by max pooling and dropout mechanisms to enhance performance and generalization.

* The preprocessing pipeline employs the Fast Fourier Transform (FFT) as the primary method for transforming raw audio data into a more suitable representation for the model.

In retrospect, while evaluating the initial objectives of this research, it is evident that the outcomes are partially aligned with the intended goals. Specifically, the objective of enabling model training on consumer-grade hardware was not fully achieved. Although the training process was conducted on such hardware, the resulting performance metrics were only moderate, indicating room for improvement in this aspect.

However, the second objective, focused on real-time inference on resource-constrained devices, was successfully accomplished. The proposed model architecture is lightweight and optimized for efficient inference, enabling real-time performance even on modest hardware. Empirical tests conducted on the training machines, including scenarios where the GPU was used in battery mode (which typically impacts performance), demonstrated the system's ability to deliver real-time inference, as perceived by human users.

The complete source code for this project, along with the materials referenced in this report, is available in our public Git repository at https://git.attssystem.fr/attssystem/rrtir. The repository is distributed under the open-source MIT License, ensuring accessibility and encouraging further contributions from the research community.

## 6. REFERENCES

[1] Figure 53, "QLab," Available at: https://qlab.app/, 2024, Visited on 2024-12-05.

[2] Andrea Martelloni, Andrew P. McPherson, and Mathieu Barthet, "Real-time percussive technique recognition and embedding learning for the acoustic guitar," in Proceedings of the 24th International Society for Music Information Retrieval Conference, Milan, Italy, 2023, pp. 121–128, ISMIR, Available at: https://doi.org/10.5281/zenodo.10265236, Visited on 2024-11-18.

[3] Stefani Domenico and Luca Turchet, "On the challenges of embedded real-time music information retrieval," 2022, number 17 in Proceedings of the 25th International Conference on Digital Audio Effects (DAFx20in22), Available at: https://www.dafx.de/paper-archive/2022/papers/DAFx20in22_paper_17.pdf, Visited on 2024-11-15.

[4] Arun Solanki and Sachin Pandey, "Music instrument recognition using deep convolutional neural networks," International Journal of Information Technology, vol. 14, no. 3, pp. 1659–1668, 2022, Available at: https://link.springer.com/10.1007/s41870-019-00285-y, Visited on 2024-11-18.

[5] Maciej Blaszke and Bozena Kostek, "Musical instrument identification using deep learning approach," Sensors, vol. 22, pp. 3033, 2022, Available at: https://www.researchgate.net/publication/360046712_Musical_Instrument_Identification_Using_Deep_Learning_Approach, Visited on 2024-11-18.

[6] Dylan Sechet, Francesca Bugiotti, Matthieu Kowalski, Edouard d'Hérouville, and Filip Langiewicz, "A hierarchical deep learning approach for minority instrument detection," 2024, number 60 in Proceedings of the 27th International Conference on Digital Audio Effects (DAFx24), Available at: https://www.dafx.de/paper-archive/2024/papers/DAFx24_paper_60.pdf, Visited on 2024-11-15.

[7] Kleanthis Avramidis, Agelos Kratimenos, Christos Garoufis, Athanasia Zlatintsi, and Petros Maragos, "Deep convolutional and recurrent networks for polyphonic instrument classification from monophonic raw audio waveforms," IEEE, 2021, ISSN: 2379-190X, Available at: https://ieeexplore.ieee.org/document/9413479, Visited on 2024-11-18.

[8] Jongpil Lee, Taejun Kim, Jiyoung Park, and Juhan Nam, "Raw waveform-based audio classification using sample-level CNN architectures," no. arXiv:1712.00866, 2017, Available at: http://arxiv.org/abs/1712.00866, Visited on 2024-11-18.

[9] Wei Dai, Chia Dai, Shuhui Qu, Juncheng Li, and Samarjit Das, "Very deep convolutional neural networks for raw waveforms," no. arXiv:1610.00087, 2016, Available at: http://arxiv.org/abs/1610.00087, Visited on 2024-11-18.

[10] Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, and Mohammad Norouzi, "Neural audio synthesis of musical notes with WaveNet autoencoders," 2017, Available at: http://arxiv.org/abs/1704.01279, Visited on 2024-11-03.

[11] Claire Arthur and Nathaniel Condit-Schultz, "The coordinated corpus of popular musics (CoCoPops): A metacorpus of melodic and harmonic transcriptions," in Proceedings of the 24th International Society for Music Information Retrieval Conference, Milan, Italy, 2023, pp. 239–246, ISMIR, Available at: https://doi.org/10.5281/zenodo.10265267, Visited on 2024-11-18.

[12] Ethan Manilow, Gordon Wichern, Prem Seetharaman, and Jonathan Le Roux, "Cutting music source separation some (Slakh): A dataset to study the impact of training data quality and quantity," in Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WAS-PAA). IEEE, 2019, Available at: http://www.slakh.com/, Visited on 2024-11-18.

[13] Stefani Domenico, Luca Turchet, and Simone Peroni, "A comparison of deep learning inference engines for embedded real-time audio classification," 2022, number 16 in Proceedings of the 25th International Conference on Digital Audio Effects (DAFx20in22), Available at: https://dafx.de/paper-archive/2022/papers/DAFx20in22_paper_16.pdf, Visited on 2024-11-15.

[14] Tracel Inc., "Burn," Available at: https://burn.dev/, Visited on 2024-11-18.

[15] Johan Erbani, Pierre-Édouard Portier, Elöd Egyed-Zsigmond, and Diana Nurbakova, "Confusion Matrices: A Unified Theory," IEEE Access, pp. 1–1, 2024, Available at: https://ieeexplore.ieee.org/document/10769075/?arnumber=10769075, Visited on 2024-12-06.