# Playing Integrated Music Knowledges with Artificial Neural Networks

Frédéric Voisin, Robin Meier

► To cite this version:

Frédéric Voisin, Robin Meier. Playing Integrated Music Knowledges with Artificial Neural Networks. Journées d'Informatique Musicale - Sound and Music Computing, IRCAM; Centre Pompidou, Oct 2004, Paris, France. hal-01612524

HAL Id: hal-01612524

https://hal.science/hal-01612524

Submitted on 7 Oct 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# Playing Integrated Music Knowledges with Artificial Neural Networks

Frédéric Voisin and Robin Meier
CIRM, 33 Avenue Jean Medecin, Nice

## Résumé

This text presents some aspects of the Neuro-muse<fnref n="1" /> project developed at CIRM. We will present some experiences in using self-organizing maps (SOM)<fnref n="2" /> to generate meaningful musical sequences, in real-time interfaces, with MaxMSP and Jitter<fnref n="3" />.

## 1 Introduction

Recent advances in the last decades in cognitive sciences, neurobiology and computer sciences make artificial neural networks (ANN) more pliable in musical applications. This connectionist paradigm, which is dramatically different to traditional computing, requires a variety of approaches, from the more factual (biology, physics, computing) to the more abstract (sciences of cognition, sociology, artificial intelligence).

## 2 SOM in MaxMSP

jit.robosom is the first freely distributed self-organizing map for MaxMSP-Jitter, written at CIRM by Robin Meier<fnref n="4" />. A SOM is normally used to represent and extract meaningful content from raw data. It can be applied to classification tasks, pattern recognition, filtering, and data completion. The ability to use a SOM within a real time musical environment simplifies a number of tasks : its easy to work with audio, symbolic streams and logics. Raw-data (i.e. stimuli) can be encoded and sent directly to a SOM within Max using Jitter matrices. Furthermore it is now easier to use a SOM in real

time for high-level user control and augmented interfaces. The Jitter layer provides another way for encoding large vectors, and gives the user a graphical representation of the activation map of the neural network : neural networks are not black-boxes anymore, one can now see their full activity, just as a neurologist can visualize brain activity with medical imaging technologies. Furthermore, these activation maps of ANN can be recorded as movie files in sync with their input and output data for later observation and analysis. Using standard Max objects for programming artificial neural networks provide a convenient way of changing their code on the fly : the generic algorithm of a SOM is quite simple and can be edited using Max programming to adapt it to peculiar tasks or experiments. One can now design and experiment with complex networks using several content access memories, and feed-forward networks for data computing and encoding. Using various programming languages within Max or Pure-Data in real-time, such as LISP or Java, makes the use of distributed agent systems more flexible. One can also use genetic algorithms to generate learning rules and network architectures<fnref n="5" />.

In the public distribution of jit.robosom two simple chaotic features are provided<fnref n="6" />. A temperature factor which is inspired by the simulated annealing method found in heuristic algorithms, and adds noise to the "synaptic" weights. Another temperature factor adds slight errors to the distance evaluation of the patterns, making classification more or less precise. Each of these variations in the code and learning rules may introduce differences in the ANN behaviors. Various chaotic or highly

<footnote n="1">¹http://www.neuromuse.org</footnote>

<footnote n="2">²Teuvo Kohonen : Self-Organizing Maps, Springer Series in Information Sciences, Vol. 30, 1995 ; 1997, 2001.</footnote>

<footnote n="3">³http://www.cycling74.com</footnote>

<footnote n="4">⁴http://www.neuromuse.org/downloads</footnote>

<footnote n="5">⁵See Balakrishnan Honavar : Evolutionary Design of Neural Architectures, 1995, Iowa State University</footnote>

<footnote n="6">⁶see Robin Meier, Applications musicales de cartes auto organisatrices : jit.robosom. http://robin.meier.free.fr/memoire.pdf</footnote>
dynamic rules may be applied and studied<fnref n="7" />.

Changing the architecture of the ANN can be done in different ways, at different levels in the network. This can range from slight tunings to strong changes of the generic rules and network topology. Experimenting with pathologic or minimalistic ANN can help to understand their general behavior as much as successful ones. These experiences in music production constitute a very interesting empirical material not only for the study of ANN, but also for the study of musical knowledge.

## 3 From classification to cognition

Classification is a standard application of self organizing maps. High dimensional data can be reduced to a lower dimensional space. The relevancy of the classification is due to i) the encoding (pre-processing), ii) the topology, and iii) the networks process of learning. Preprocessing can be done both by distributed memories and feed-forward networks, or by classical algorithms. Figure 1 shows a basic example using jit.robosom in Max for vowel classification. Here, the preprocessing consists only in extracting the first 256 bins of a 512 bins FFT, computed directly from the sound samples (vowels).

The activation map of the SOM in Jitter gives the details of the classification during learning and processing in an arbitrary space chosen by the user (see figure 2). In this example, a 16 unit SOM makes a square map in which four vowels may be classified. The observation and the analysis of the activation during learning can give an abundance of relevant information about the processed data. The history of SOM activations and errors in the learning process is part of the data analysis, showing how a classification is processed within an arbitrary context. As the video HAL04.mpg<fnref n="8" /> demonstrates in figure 2, the well-known learning process of a SOM reveals phonological aspects of the stimuli through the activations of its 16 units. This also demonstrates how similarity is not a stationary category, but can refer to dynamic and temporal aspects defining a context. Since a SOM learns and processes

solely by impression and imitation, it is simply processing a stimulus through the image of its history that has been collapsed into one frame (as well as some recurrent connections). An example of such an actual activation is shown in figure 2, where the image represents the activation map of the 16 units-SOM, as a memory recall of the stimulus during the learning process. The sound resynthesis is performed by a FFT-1 from these activations.

The observation of this example clearly shows that the functional area of the network corresponds to implicit small units of knowledge (phonems)

<figure><img src="image_1.png" /><figcaption>FIG. 1 – Example-patch for jit.robosom</figcaption></figure>

<figure><img src="image_2.png" /><figcaption>FIG. 2 – activation map of a 16 units SOM with Max-Jitter</figcaption></figure>

<footnote n="7">⁷Christophe Philemotte, Etude des rseaux neuronaux en tant que systmes dynamiques chaotiques, http://iridia.ulb.ac.be/cphilemo/tfe/tfe.html and : Emmanuel Daucé, Adaptation dynamique et apprentissage dans de réseaux de neurones recurrents aléatoires, http://esm2.imt-mrs.fr/~dauce/these/</footnote>

<footnote n="8">⁸http://www.neuromuse.org/txt/tutorial/Hal04.html</footnote>
which are transmitted via ordered sets of stimuli, making more or less explicit relations. It also shows that the control of stimuli and learning rules gives full control of a SOM behavior : if new stimuli are presented to a SOM following any arbitrary path, the SOM will classify first, according to its initial state and then it adapts, if necessary, to these new aspects. The study of these relationships between knowledge representations and memory topologies, is an interesting approach for generating new and original musical material with a SOM.

4 Playing with knowledge units

The following example is a program in Max using two small-sized SOM (26 memory units, neurons ) generating musical sequences in MIDI format, ac- cording to given examples. Short patterns of 15 to 30 notes encoded in a 3D-space (pitch, velocity, inter-onset interval) are played to a single unit SOM doing a time loop processing of the input with feed- back. Temperature and sensitivity (learning factor) of the SOM give a control of looping variations of the musical sequence, making, in a minimalist way (one unit-automat), deviations from original pat- terns (figure 3).

<figure><img src="image_3.png" /><figcaption>FIG. 3 – single unit SOM doing just a time loop
process</figcaption></figure>

At the same time, this looped-with-variation out-
put is sent to a 25 unit SOM trained to repeat the

stimuli, according to its actual state (see example
figure 4), which depends on what it has recently
learned. Temperatures and sensitivity of the SOM
are particularly interesting parameters to control in
real-time. They permit numerous methods of varia-
tion, from perfect repetition to meaningful interpo-
lations, spontaneous generation of new relevant ma-
terial or chaotic cycles, in relation to the given and
ordered stimuli. In this example, the 25 unit SOM
outputs new patterns related to previous (already
heard) and actual (unknown) patterns. Its adapting
itself to the new (melodic) context coming from the
single unit looping SOM, or from the user (a mu-
sician playing with Max interface). It shows how a
SOM can react as a knowing agent, receiving new
stimuli, memorizing and classifying them according
to the implicit rules it was just exposed to. Tem-
perature control and other dynamic features may
help a SOM jump to new types of representations,
producing new and relevant variations.

<figure><img src="image_4.png" /><figcaption>FIG. 4 – 25-units SOM trained to repeat the stimuli</figcaption></figure>

As soon as we are interested in these dynamic
processes of the network, the traditional fitness er-
ror, at some stage, may not be a good criterion for
evaluating a neural networks capabilities. In such
a context, the output of a trained SOM with large
fitness errors is not hazardous (as is the case when
using untrained SOM). Some aspects of this empir-
ical material may not only question the connec-
tionist approach involved; it also questions origi-
nal human representations of musical knowledge.
Then, one can experiment with the hypothesis in
which so-called natural knowledges and languages
can adapt themselves to neurophysiologic proper-
ties of the brain<fnref n="9" />: computational conflicts, errors,
and mismatches can reveal conflicts at a higher cog-
nitive level that can be advantageously used for ge-

<footnote n="9">⁹Cf. Simon Kurby, Morten Christiansen, NewScientist,18/01/03, p. 30</footnote>
nerating original musical material.

As shown with the following musical example
for sampled piano and jit.robosom (For Alan Tu-
ring, by Robin Meier, 1030), the minimalistic self-
organizing maps are not able to integrate the en-
tire musical material presented and therefore are
becoming dynamic musical constraints. Later de-
velopments involving more sophisticated architec-
tures and more complex musical structures remain
to be studied.

5 Conclusion

This approach shows the contemporary ability
for a large community of musicians to easily per-
form cognitive processing to generate music with
artificial neural networks. The understanding of the
complex behavior of ANN needs much investigation
and experimentation for their use in musical appli-
cations. Experimenting with such knowledge based
systems is not only valuable from a scientific pers-
pective but can also be the source of various musical
inspirations.