# Brain Computer Interface, Visual Tracker and Artificial Intelligence for a Music Polyphony Generation System

Carmelo Ardito, Tommaso Colafiglio, Tommaso Di Noia, Eugenio Di Sciascio

► To cite this version:

Carmelo Ardito, Tommaso Colafiglio, Tommaso Di Noia, Eugenio Di Sciascio. Brain Computer In-
terface, Visual Tracker and Artificial Intelligence for a Music Polyphony Generation System. 18th
IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.368-371,
10.1007/978-3-030-85607-6_39 . hal-04291195

HAL Id: hal-04291195

https://inria.hal.science/hal-04291195v1

Submitted on 17 Nov 2023

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Brain Computer Interface, Visual Tracker and Artificial Intelligence for a Music Polyphony Generation System

Carmelo Ardito[0000-0001-8993-9855], Tommaso Colafiglio[0000-0001-7184-310X], Tommaso Di Noia[0000-0002-0939-5462], Eugenio Di Sciascio[0000-0002-5484-9945]

Politecnico di Bari – Via E. Orabona 4, Bari (I-70125), Italy
{name.surname}@poliba.it

**Abstract.** In the Brain Computer Interface domain, studies on EEG represent a huge field of interest. Interactive systems that exploit low cost electroencephalographs to control machines are gaining momentum. Such technologies can be useful in the field of music and assisted composition. In this paper, a system that aims to generate four-part polyphonies is proposed. An artificial intelligence algorithm permits to generate polyphonies based on the N. Slonimsky's theory by elaborating data coming from a Leap Motion device, to detect user's hand movement, and a five-channel EEG signal detection device.

**Keywords:** EEG, Brain Computer Interface, Leap Motion, Slonimsky

## 1 Introduction

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Techniques for inserting notes within interval axes</figcaption></figure>

A music composer continuously develops music models that represent her idea of music. Such models can be intended as musical materials useful in the construction of the final piece. Thus, composers explore multiple combinations of pitches, rhythmic profiles, dynamics, and all the other parameters that represent sound in general. In 1947, the composer N. Slonimsky published a treaty entitled "Thesaurus of Pattern and Melodic Scale" [1], in which he presented a technique for constructing melodic patterns based on dividing one or more octaves into equal parts. According to Slonimsky, the construction of melodic patterns can be made systematic by inserting notes below, within and above a musical interval-axis (see Fig.1). Once the type of interval division
of the octave is chosen<fnref n="1" />, notes can be inserted within (Interpolation), beyond (Ultrapo-
lation) and below the division interval (Infrapolation). By combining these techniques
for filling interval-axes, sinusoidal profiles of Infra-Interpolation, Infra-Ultrapolation
and Infra-Inter-Ultrapolation are obtained.

In this research work, a technological tool is presented that exploits Artificial Intel-
ligence algorithms, a Leap Motion controller [2], and a five-electrode Emotive headset
[3] to support composers in exploring melodic profiles that represent the basis for com-
posing new music pieces via the Slonimsky's Infra-Inter-Ultrapolation technique.

## 2 The polyphony generation system

The system architecture is schematized in **Fig. 2**. The *Pitch Profile Generator* module on the left side is based on an algorithm that transforms into note pitches the coordinates of the composers' fingers detected by the Leap Motion device. The result is a pitch profile, i.e. a sequence of notes without indications about their duration. The module on the right side is characterized by *FeedForward Net Classifier* that, with a Deep Network algorithm, analyzes the EEG signal coming from the five electrodes of the Emotiv headset worn by the composer and classifies her mental status as “focused” or “relaxed”. Here, a mental status is intended a sequence of power spectrum values obtained from the EEG signal. The algorithm has been previously trained by the specific composer's mental status dataset. The *Polyphonic Structure Generator*: 1) receives the pitch profile, 2) multiplies it by four to have a draft polyphony, 3) according to the composer's mental state, gives a duration to the notes of the four pitch profiles, which are differentiated by means of a scramble method.

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** System architecture</figcaption></figure>

<footnote n="1">¹ In Fig 1, the octave between the C note under the pentagram and the C note in the second upper space has been chosen and is represented in the Principal Tones section.</footnote>
By means of the Cortex API SDK [4], the system derives a series of numerical values in the overall frequency 1-43 Hz, classified in different bands of the power spectrum of interest, i.e., Delta (0-4Hz), Theta (4-8Hz), Alpha (8-12Hz), Beta (12-35Hz) and Gamma (>35Hz). The Keras framework [5] is used to implement the Deep Network algorithm as shown in **Fig. 3**.

<figure><img src="image_5.png" /><figcaption>Fig. 3. Deep Network algorithm implemented in Keras with Python</figcaption></figure>

Before using the system for composing, it has to be trained to the mental states of a specific composer, so that the underlying network can adapt to the bandwidth variation characteristics, which is a specific function of each user. Such a “calibration” has to be done once for each user. The composer wears the BCI headset and samples herself in two different mental states, i.e., focused and consciously relaxed. Focused mental status means a higher amplitude of high frequencies in the range 13-40 Hz, which corresponds to the Beta and Gamma frequency bands. Consciously relaxed means a greater activation of low frequencies and specifically the Alpha band between 8-13Hz.

If the mental status is classified as Focus, the polyphony is generated with rhythmic figures such as quarter note, height notes, sixteenth notes, and their corresponding pauses. If, on the contrary, the mental status is Consciously relaxed, the polyphony generated has rhythmic values of short, whole notes, half notes, quarter notes and their corresponding pause values. In addition, each bandwidth value received in real-time from the BCI headset is used to assign a dynamic value to each generated note.

<figure><img src="image_6.png" /><figcaption>Fig. 4. User interface of the system.</figcaption></figure>
**Fig. 4** shows the system user interface, which is divided in three sections, as for the main phases of the polyphony composition process. Starting from the top, the first section is devoted to the two mental states training; a summary of the overall process is provided in the green box, which also acts as user guide. The second section is dedicated to generating the pitch profile using the Leap Motion visual tracker; a graph of finger positions is drawn in the black panel, while on the right the composer can see in real-time the pentagram with the sequence of generated notes. By means of the field on the right, the composer can possibly change the interval-axis. The third section shows how, in real-time, the pitch profile is transformed in the resulting polyphony according to the composer's EEG. A few further parameters can be set by using the other widgets in this section of the interface.

The generated polyphony can be saved in a format compatible with popular software tools, e.g. MAX/Msp [6], that the composer can use for refining it and transform in the final piece of music.

## 3 Conclusions

In this paper we presented an approach aimed at generating complex polyphonic structures related to the theoretical model of Slonimsky, useful for the composer who wants to optimize her compositional process. The polyphonic material obtained is deliberately not complete in a strict musical sense, because we want to leave to the composer the possibility to work and perfect it according to her artistic needs. The creative process is a process that involves different technical and emotional aspects and in this sense, the final goal is to demonstrate that the BCI, in our opinion, could be useful as a support, and not as a substitute, tool for the composer. The system can be evolved by expanding the compositional theoretical models, thus providing a wider support to the work of contemporary musicians/composers.

**Acknowledgments.** The authors acknowledge partial support of the following projects: PON Casa delle tecnologie emergenti di Matera "CTEMT" (CUP I14E20000020001), Servizi Locali 2.0, PON ARS01_00876 Bio-D, PON ARS01_00821 FLET4.0, PON ARS01_00917 OK-INSAID, H2020 PASSPARTOUT.

## References

1. Slonimsky Nicola, 1984 – Thesaurus of scales and melodic patterns. Previously published: New York C.Scribner, 1947. Schirmer Books, s division of Macmillan Publishing Co. Inc. 856 Third Avenue, New York, N. Y. 10022.

2. Ultraleap. Leap Motion Controller. Last visit April 2021. https://www.ultraleap.com/product/leap-motion-controller/

3. Emotiv Insight 5-channel mobile EEG. Last visit April 2021. https://www.emotiv.com/product/emotiv-insight-5-channel-mobile-eeq/

4. Emotiv Cortex API. Last visit April 2021. https://emotiv.gitbook.io/cortex-api/

5. Keras Framework. Last visit April 2021. https://keras.io/

6. Max/Msp. Last visit April 2021. https://cycling74.com/