Generating an Integrated Musical Expression
with a Brain–Computer Interface
Takayuki Hamano

Tomasz M. Rutkowski

Hiroko Terasawa

JST, ERATO, Okanoya
Emotional Information Project2-1 Hirosawa Wako
Saitama, Japan
hamano@japan.com

University of Tsukuba
Life Science Center of TARA
1-1-1 Tennoudai Tsukuba
Ibaraki, Japan
tomek@tara.tsukuba.ac.jp

University of Tsukuba
Life Science Center of TARA
1-1-1 Tennoudai Tsukuba
Ibaraki, Japanterasawa@tara.tsukuba.ac.jp

Kazuo Okanoya

Kiyoshi Furukawa

JST, ERATO, Okanoya
Emotional Information Project2-1 Hirosawa Wako
Saitama, Japankazuookanoya@gmail.com

Tokyo University of the Arts
12-8 Ueno Park Taito
Tokyo, Japan
kf@zkm.de

ABSTRACT

Thus, we developed an expressive musical instrument forconcert performances, which renders musical chords withorganic nuances (dynamics and timbre) in real time basedon the brain waves measured during musical imagination,using a brain–computer interface (BCI) technique and sonification techniques. A study by Herholz et al. showed that
there are overlapping areas in the brain during melody perception and imagery[5]. Thus, we aimed to employ thesetypes of brain waves related to musical imagery for generating musical expression. Table 1 shows the features of our
BCI instrument compared with an acoustic instrument.
A traditional acoustic instrument is highly expressive when
it has been mastered after extensive training. A survey
showed that humans add expressive performance actions
that are not marked in the score, which can also be foundin pre-20th century music. Many instruments also provide
the ability to change intonation, vibrato, and timbre in an
expressive manner [8]. We consider that this expressivity often relies on the capacity for intended codes, such as notes
and rhythms, and non-coded expressions, such as dynamics,
timbre, and subtle micro-timing.
Using our new instrument, we aimed to generate the same
degree of expressivity based on coded and non-coded information from brain waves. The coded musical elements(e.g., musical chords) are generated based on the classification of brain waves, i.e., qualitative information whereas
the non-coded, organic nuances (e.g., dynamics and timbre) are generated from the transitional characteristics ofthe brain waves, i.e., quantitative information. In addition,
our instrument has a video projection element that corresponds to the state of the brain waves, which aims to reflect
the music imagery.
This instrument was developed for practical musical expression but we hope also that it has the potential for understanding the neural activity in the brain when a player
is engaged in musical expression.

Electroencephalography (EEG) has been used to generatemusic for over 40 years, but the most recent developments inbrain–computer interfaces (BCI) allow greater control andmore flexible expression to use new musical instruments via
EEG. We developed a real-time musical performance systemusing BCI technology and sonification techniques to generate chords with organically fluctuating timbre. We aimed toemulate the expressivity of traditional acoustic instrumentsby adding “non-coded” expressions that were not marked in
the score. The BCI part of the system classifies patterns
during neural activity while a performer imagines a chord.
The sonification part of the system captures non-stationarychanges in the brain waves and reflects them in the timbre
by additive synthesis. In this paper, we discuss the conceptual design, system development, and the performance of
this instrument.

Keywords
Brain–computer interface (BCI), qualitative and quantitative information, classification, sonification

1.

INTRODUCTION

A musician can dream about an ideal performance without
any physical limitations, where the performer plays withthe expressivity imagined in their mind. At the heart ofthis dream, however, there is an assumption that the humanmind is capable of free expression (our imagination of music
is supposed to be spontaneous, without constraint, and in
perfect accord) but its physical rendering, i.e., the musicalperformance, is a limited delivery of the overall imagined
performance. In this project, we attempted to capture a
glimpse of the imagination of music to generate a performance with the aim of making music an organic expressionthat reflects the lively and subtle transitions of psychological states.

1.1

Advantages of Using Electroencephalography to Produce MusicElectroencephalography (EEG) is a method for measuring
the electrical activity produced by brain neurons. Compared with other biological methods, it has some appealing
points for use in music generation. The time resolution of
the data acquired using EEG is very fine, which makes itsuitable for generating musical changes over time. Other
methods used to observe brain activity, such as fMRI and

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.
NIME’13, May 27 – 30, 2013, KAIST, Daejeon, Korea.
Copyright remains with the author(s).

49Table 1: Comparison of Musical Expressivity Using an Acoustic Instrument and our Instrument
Acoustic Instrument
Our BCI Instrument
Control
Physical motion to
Imagery, psychological motion to induce neural activity
generate physical soundMusic (coded, qualitative)
Notation
Chords generated by classification (BCI)Music (non-coded, quantitative) Non-notated articulation Dynamics and timbre as sonification (sonification)
VisionGestural presentation
Video corresponding to music imagery

MEG, require extremely large equipment, whereas many
portable and handy EEG systems are available on the market. However, the most attractive aspect of EEG is that thebrain waves reflect thoughts and emotional states. Thus,
producing music using EEG may reveal actual emotions,which is an attractive prospect.
In addition, research into BCIs has been growing in recent
decades. The ultimate goal of BCI is to build a system that
reads human thoughts, which can provide a bridge between
the brain and an external device. Many analytical methods
have used BCI to identify thought states and emotions in
the mind. Thus, BCI provides the possibility of extracting
the patterns of musical imagery from neural activity.

1.2

for collecting biosignals [7]. Early research was based on
verbal descriptions provided by patients, which were subjective and qualitative. By contrast, the use of technical tools
to make biological measurements has reduced the level ofsubjectivity because these approaches are based on quantitative data. Interestingly, a flute teacher, Francois Nicolas
Marquet, attempted to code heart pulses as a musical notation. This case is regarded as a combination of qualitativeand quantitative data.
Kaniusas regarded the difference between qualitative andquantitative information as a problem of subjectivity versus
objectivity. When considering the use of biosignals for musical expression, this approach facilitates a clear mapping tomusical parameters. This is because music may be considered to be composed of coded information, such as notatedpitch, and non-coded information, such as non-notated articulation, which correspond to qualitative and quantitative
data, respectively. Therefore, playing an instrument is considered to be an action where the player produces sounds
based on their subjective (or sometimes subconscious) ideas
and feelings, whereas the sound produced by the player conveys both qualitative and quantitative information to theaudience.
In this context, we propose a method that integrates
the qualitative and quantitative information obtained using EEG before its use for musical expression. The majoradvantage of our approach is that quantitative information
can be contextualized using qualitative information. Bothqualitative and quantitative information are then mapped
onto musical parameters, which are coded musical elements
and non-coded organic nuances, respectively. In this manner, the integrated information is transformed into musicalexpression.

Related Work

Many notable composers have attempted to produce music using EEG. The American composer Alvin Lucier was
the first person to use EEG as a source of music. In hispiece Music for Solo Performer (1965), alpha waves wereused to vibrate percussion instruments. David Rosenboom
is another famous composer who used EEG for sound andmusic creation. One of his contributions was the use of abiofeedback process during musical performances where hemade the brain state of the performer audible [14].
In recent years, many artists have worked using EEG,
which has been facilitated by the smaller size and low priceof the latest EEG devices. For example, Angel [1] produced interactive art using EEG data, while visual effects
and sounds were generated based on brain, heart, and respiration signals by Filatriau [3]. Some notable projects havebeen presented at past NIME conferences [4, 10, 11, 13].
Previous studies have used BCI for music creation. Miranda developed a brain–computer music interface (BCMI)
that comprised a MIDI-controlled mechanical acoustic piano controlled by EEG using generative musical rules[12].
Another group, The MiND Ensemble, delivered stage performances using a portable EEG system1 . In their performances, psychological parameters were estimated frombrain waves and assigned to musical parameters.

1.3

2.

Problems and Solutions

All of the aforementioned studies developed functional methods for music performance using EEG, which allowed theaudience to perceive musically expressed neural activities
from various perspectives. These studies typically analyzed
brain waves and presented them as sounds. Thus, they extracted useful information from neural activity and mapped
it to musical parameters, before presenting the music to audiences. In our study, we considered the different typesof biosignal information that can be obtained and used formusic generation.

1.3.1

Qualitative and Quantitative Information

Most biosignal information can be classified as qualitativeor quantitative.
The book Biomedical Signals and Sensors I by Eugenijus
Kaniusas provides a historical account of the methods used
1

GOALS

Based on the solutions proposed above, we aimed to produce a musical performance by integrating the qualitativeand quantitative data extracted from EEG outputs. Thus,
a player wearing an EEG system imagines chords, i.e., setsof musical notes, in their mind. Our BCI instrument extracts various patterns and characteristics from the EEG
data and converts them into sounds and visual images. The
amplitude of the EEG data affects the timbre of chords via
sonification. This allows music to be played using brain
waves as an instrument where the sounds reflect the brain
activity during the imagination of music. Based on thisidea, we aimed to design our instrument for practical use inmusical performances, such as a concert.
To accomplish this goal, the Brain dreams Music (BdM)
Project was initiated in April 2011 by researchers from diverse domains, such as music composition, neuroscience,and computer science. This group has conducted research
to develop this new instrument.2

2

http://www.themindensemble.com/

50

http://brain-dreams-music.net/

3. DESIGN AND IMPLEMENTATION
3.1 Design of the Performance

Preprocessing.
The acquired EEG data are processed using a band-passfilter during the Preprocessing stage. This removes any artifacts, including noise components such as the electric hum
noise or the pulse from the heart.

Before formulating the specifications of the instrument, we
designed the overall process of our musical performance asfollows. Figure 1 shows the overall procedure used to acquire EEG data, which are processed and delivered to theaudience using BCI. In this procedure, the instrument estimates the music imagined by a player and generates sounds
and visual images automatically in real time.

Feature Extraction, Classifier Learning, and Classification.
To estimate the patterns imagined by the player, it isnecessary to make the computer learn the correlations between features in the brain-wave patterns and the imagined
chords. However, the electrophysiological brain signals captured non-invasively via the human scalp using EEG are
non-linear and non-stationary. We addressed this problem
by combining two methods, i.e., the common spatial patterns (CSP) method is used for feature extraction and binary linear discriminant analysis (LDA) is used for classification.
To determine the order of processing, common features
are extracted from each imagery pattern using the CSP feature extraction process and the filtered EEG signals. CSP
has been used widely in BCI research to identify linear spatial filters for extracting discriminative brain activities during two different mental imagery tasks. We used CSP tofind the optimal spatial filter for each imagery class, which
maximizes the average energy of that particular class while
minimizing the other remaining classes.
The extracted features are stored and organized during
the classifier learning process. This learning process is conducted every time before a performance because the state of
the brain waves changes on a daily basis and the computer
needs to be recalibrated.
After the computer learns the feature of the EEG pattern,
it is ready for classification. During the Classification process, the newly acquired data is classified according to four
predefined patterns. To implement a multi-class machinelearning approach, we utilized a cascade of LDA classifiersin a one-versus-all configuration. During each classification
step, “a winner takes all” method is applied by choosing thebest one-versus-all classification posterior probability result.
The results of the pattern analysis and the probability areused to generate sounds and visual images.
In another path, the filtered EEG signal is used directly
without processing as a source for sonification.
This instrument applies different information properties
using the classification results and the filtered EEG signals
as qualitative and quantitative information, respectively.

[A] EEG Analysis

EEG data acquisition
Brain Player
imagines chords.

Raw EEG data stream

Preprocessing (filtering)

Feature extraction

Classifier
learning

Triggers[B] Generation

Probability

Quantitative

Classified
patterns

Quantitative

Classification

Qualitative

Sequence

Filtered
EEG

Sound

ChordsDynamics

TimbreVisualization

Color

Speed oftransition

Brightness

Qualitative

Quantitative

Audience

Figure 1: Schematic diagram of the procedure usedduring musical performances with BCI. The extracted qualitative and quantitative information are
integrated to produce sound and vision representations.

3.1.2

The procedure can be divided into two stages, which areindicated by the grey areas [A] and [B] in the figure.

3.1.1

Generation ([B])

Various types of data produced by the analysis process are
mapped to the parameters of the sounds and images, which
are used to generate an entity that will be received by theaudience.

EEG Analysis ([A])

Sound.

The first stage is the EEG analysis. The computer estimates
the patterns of imagined musical chords as qualitative information and extracts other quantitative information. These
processes are conducted in parallel.

The synthesizer receives the classification results and generates the sound of chords with an additive synthesis. The
dynamics of the sound vary depending on the probability
value, which indicates the degree to which the classification
results are likely to be correct. Each note sound comprises
one fundamental tone and seven overtones, i.e., a total ofeight sinusoidal waves. These waves are multiplied by processing the raw EEG data from each electrode channel, andmodulated. To enhance the effect of timbre, the overtones
are set with various degrees of inharmonicity with weighting
coefficients for each overtone. This is an extension of overtone mapping [15] and it produces a unified sound effectwhen integrating multichannel EEG signals. Finally, each

EEG Data Acquisition.
First, the player known as the Brain Player wears an
EEG cap and imagines notes as if they were actually playing
music. In the latest version, the player selects one of four
different types of chords during each imagination session.
The time frame for each chord is 3 s and it is repeatedmore than 10 times. During this period, the EEG signal isrecorded by the computer.

51

overtone is played separately from eight speakers, whichmixes all of the overtones in the acoustic space.
The timing of the imagery, EEG recording, classification,and onset of the playing sound are triggered by the score
data. The score is made by the composer beforehand using four predefined chords. During our performances, theplayer attempts to play music by imagining each chord ashort time before it is actually played. This can be slightly
confusing for the player and the player has to know when
to imagine. To address this situation, we implemented atime-synchronized score display. The score reflects the composer’s musical intention to some extent. After the score
display starts playing, each chord the player needs to imagine is shown on the player’s iPad in sequence so the player
can concentrate on imagining that chord. In other words,
the score simply indicates the timing but does not affectany of the result of the analysis, including the classification
results.

Table 2: List of Programming Languages and APIs
Used in the BCI
Process
Languages and APIs used
EEG Streaming Server MATLAB
Messaging Server
Java
Web Server
Node.js
Classifier
MATLAB
Synthesizer
Max/MSP
Visualizer
Java, OpenGL
Control Surface
Java, OpenGL

3.2.1

Visualization.
Visual images are also generated by the same information used to produce sounds and they are presented to theaudience at the same time.
The positional arrangement of the electrodes is shownon the screen so the audience can observe the classification
results and the brain activity simultaneously. Thus, theoverall color represents the classified pattern, the transition
speed of the color reflects the probability value, and thebrightness of each electrode position represents the amplitude of the filtered EEG signal.

3.2

Implementation of the Instrument

To implement this performance paradigm, we constructed
the BCI shown in Figure 2.

Hardware

EEG Electrodes (g.SAHARA)

SoftwareandData stream
Text messaging

EEG Amplifier (g.MOBIlab+)

3.2.2«bluetooth»
Raw EEG Data

LaptopEEG Streaming Server
Filtered EEG

Score Display

Messaging Server

Classifier
Classification
results

Web Server

Synthesizer

Visualizer

Surrounded
Speakers

Dual
Projections

4.

PERFORMANCE

We conducted five performances during 1.5 years using thismusical BCI, including domestic and overseas events. Threeof the performances were given in large concert halls in international conferences or exchange concert programs. Others were parts of concerts in small lecture rooms.
The latest performance took place at Kubus (Cube) in
Zentrum für Kunst und Medientechnologie Karlsruhe on«IEEE 802.11g»
iPads

Software

The software configured several internal processes using multiple programming languages and APIs (Table 2).
Each process operates independently in parallel while information is exchanged between processes via networking.
The exchanged information belongs to two types, i.e., astream of EEG data and text messages. Both types of data
are transferred via UDP. Three types of servers, i.e., an
EEG Streaming Server, Messaging Server, and Web Server
were developed using MATLAB and Node.js4 .
All of the controls required for a performance are handled
on the Control Surface. A preview monitor of the raw EEG
waveforms and the topographic map, the controls used forthe classification process, and the sequencer for playing control were implemented on the Control Surface (Figure 3).

Control Surface
EEG Monitor

Hardware

A mobile wireless EEG amplifier g.MOBIlab+ was used in
combination with dry EEG electrodes g.SAHARA manufactured by g.tec 3 . g.MOBIlab+ also sent EEG data to
the computer in real time at a 256 Hz sampling rate via
Bluetooth serial communication. Unlike typical wet EEGelectrodes, the g.SAHARA dry electrodes did not require
gel or paste. Thus, the preparation was faster and there
was no need to wash the head after use. The number ofmeasurement positions was up to eight each time and thearrangement could be changed freely. Based on the resultsof our experiment using another wet 32-point EEG electrode system, we selected the eight channels that worked
the most efficiently and effectively. The portability of thisEEG device indicated that it was suited perfectly to ourmusical performance.
We used a MacBook Pro with Windows 7, which was installed with BootCamp. Some physical MIDI controllers
were connected to it to allow more convenient control during performances. In addition, we used iPads to provide a
visual presentation of the imagery and to give other accompanying acoustic instrument players cues that facilitatedmusical synchronization.
Finally, the generated sound was played by 8-channel
speakers and the visualization was shown on dual screen.
The multi-channel speakers produced an acoustic space where
the audience could hear precise changes in brain activity.

Figure 2: Diagram of the system architecture of our
BCI instrument.

3
4

52

http://www.gtec.at/
http://nodejs.org/

Screen 1

AmplifierOperator
EEG
Sound
card

8 Speakers
Clarinet
PlayerBrain
PlayerLaptop

Projector

Scr

een

2

P
Prroojectr jectoor

To Mixing Console

Audience

Figure 5: Stage diagram.

5.

DISCUSSION

In this section, we discuss the technical novelty of BCI for
communicating the aesthetics of musical expression.

5.1
Figure 3: Screenshot of the control surface.
September 20125 (Figure 4). The seating capacity was around50 and eight speakers were arranged at regular intervals on
the stage. Visual projections were displayed on a large central screen for the Visualizer and another monitor at theside displayed the Control Surface. We provided a demonstration and gave a live performance to verify the usability
and expressivity of our instrument. The brain player was
not allowed to move his body so the audience could seethat the player was sitting still. During the performance,
the audience listened to the sound and recognized certain
musical structures that partly reflected the player’s intentions. The fluctuating timbre also allowed the audience toperceive dynamical changes in brain activity. The visual
images synchronized with sounds allowed the audience toappreciate the brain activity.

Figure 4: Performance at Zentrum für Kunst undMedientechnologie Karlsruhe (ZKM), September
2012.

5.2

Musical Elements Used for Imagery

The choice of the music elements used for imagery is important from another perspective. In our instrument, we used
long tones of chords with a certain degree of complexity. We
used them because the player could imagine the temporally
steady state of a sound and each chord had a characteristicthat was distinct from another. We conducted an experiment to compare various types of musical elements during
imagery and determined the classification accuracy. We assumed that a steady-state imagery while sustaining chords
was favorable for our classification method, which analyzed
the pattern of specific time frame in the EEG data.

The brain player usually plays music with one or several
accompanists. We have delivered performances on many occasions with a clarinet player and once with a string quartet.
These acoustic instrument players interact with the soundgenerated by the brain player, which produces a dialogue
between the players (Figure 5).
Video recordings and pictures of past performances havebeen uploaded on the website of the BdM Project.
5

Analysis methods

The classification algorithm was the main factor that improved our instrument. The average accuracy was 50% using four classes. This result was sufficiently high because
the chance level was 25%. During a musical performance,
however, it was difficult to get the feeling that the player
was controlling the instrument on their own. Thus, the accuracy needs to be improved to enhance the level of control.
Possible solutions are to improve the algorithm or to use
more advanced methods such as SVM or cross-validation.
These techniques may improve the results but they require
more computational time, which is why methods need to bedevised for online implementation. Another possibility ismaking improvements using portable EEG devices. Experimental evaluations are required in future to address thesepoints.
Two other possible analytical methods could be used in
our BCI instrument. First, music is a temporal art formso temporal change appears to be an important element in
the imagery of music, which may also involve the anticipation of different musical states. In future, we could use an
EEG analysis method to evaluate the causal relations between channels relative to time, such as a Directed Transfer
Function where imagery patterns related to changes in themusic might be captured.
Another possibility is the measurement of emotional states.
The components of music are considered to be associatedwith specific types of emotions so the imagery of music
may contain images of emotional states. Previous studies have measured biological signals to estimate emotional
states during musical performances [6, 9]. Estimating emotions during musical imagery would be a novel applicationof the BCI instrument.

http://on1.zkm.de/zkm/stories/storyReader$8116

53

5.3

Improvement of Human Imagery

8.

The computer we used was not responsible entirely for theclassification results because the human operator also hadroom for improvement. If the human operator could imagine the music better, the computer would find it considerably easier to classify the brain-wave patterns. Many instruments require training before an expert musical performance can be produced and this was also the case for our instrument. Thus, if the controllability could be increased byuser imagery enhancement training, it would be closer to its
ideal form and the user could produce an instrument with awide range of expressivity. At present, we have not verified
whether non-musicians with a limited ability for imagining music can operate the instrument. If we can develop amethodology for learning musical imagery, this would open
up the possibilities of brain music performance by many
people. A previous study demonstrated that brains learn tocontrol BCI adaptively citeblake:control. This type of human self-transformation process based on interactions withan instrument may improve expressivity.

5.4[1] C. R. Angel. EEG data in interactive art. In
ISEA2011 Istanbul Proceedings, September 2011.
[2] J. W. Davidson and J. S. Correia. Body movement. In
R. Parncutt and G. McPherson, editors, The Scienceand Psychology of Music Performance : Creative
Strategies for Teaching and Learning. Oxford
University Press, USA, 2002.
[3] J.-J. Filatriau and L. Kessous. Visual and soundgeneration driven by brain, heart and respiration
signals. In Proceedings of the 2008 International
Computer Music Conference, 2008.
[4] S. L. Groux, J. Manzolli, and P. F. Verschure.
Disembodied and collaborative musical interaction in
the multimodal brain orchestra. In Proceedings of theInternational Conference on New Interfaces forMusical Expression, 2010.
[5] S. C. Herholz, A. R. Halpern, and R. J. Zatorre.
Neuronal correlates of perception, imagery, andmemory for familiar tunes. J. Cognitive Neuroscience,24(6):1382–1397, June 2012.
[6] J. Jaimovich and R. B. Knapp. Pattern recognition ofemotional states during musical performance fromphysiological signals. In Proceedings of the 2009
International Computer Music Conference, pages
461–464, August 2009.
[7] E. Kaniusas. Biomedical Signals and Sensors I.
Springer-Verlag, Berlin, 2012.
[8] A. Kirke and E. R. Miranda. A survey of computer
systems for expressive music performance. ACM
Comput. Surv., 42(1):3:1–3:41, Dec. 2009.
[9] R. B. Knapp and E. Lyon. The measurement of
performer and audience emotional state as a newmeans of computer music interaction: a performancecase study. In Proceedings of the 2011 International
Computer Music Conference, pages 415–420, 2011.
[10] G. Leslie and T. Mullen. Moodmixer : EEG-based
collaborative sonification. In Proceedings of theInternational Conference on New Interfaces forMusical Expression, pages 296–299, 2011.
[11] S. Mealla, A. Väljamäe, M. Bosi, and S. Jordà.
Listening to your brain: Implicit interaction incollaborative music performances. In Proceedings ofthe International Conference on New Interfaces forMusical Expression, 2011.
[12] E. R. Miranda. Brain-computer music interface forcomposition and performance. International Journal
on Disability and Human Development, 5(2):61–67,
2006.
[13] T. Mullen, R. Warp, and A. Jansch. Minding the(transatlantic) gap: An internet-enabled acoustic
brain-computer music interface. In Proceedings of theInternational Conference on New Interfaces forMusical Expression, pages 469–472, May 2011.
[14] L. Polansky and D. Rosenboom. Interview with David
Rosenboom. Computer Music Journal, 7(4):40–44,
Winter 1983.
[15] H. Terasawa, J. Parvizi, and C. Chafe. Sonifying
ECoG seizure data with overtone mapping: A
strategy for creating auditory gestalt from correlated
multichannel data. In Proceedings of the 18th
International Conference on Auditory Display, pages
129–134, June 2012.

Evaluation

As described above, we consider that it is always important
to explore the effectiveness of musical BCI by coupling ananalysis method with imagery elements and human imagery
enhancement. These issues can be verified by technical evaluation in a scientific manner.
However, yet another type of evaluation is still required,
which is the artistic meaning of the musical performanceitself and the results produced by that performance. For
example, it is not clear whether the player might exhibit
better musical expression of the neural musical imagery withbodily movement. Davidson and Correia showed that the
body is vital for the generation of expressive ideas aboutmusic, as well as being essential for the physical manipulation of the instrument[2]. Thus, we would like to compare
physically embodied performances with disembodied brain
performances.

6.

CONCLUSION

We developed a BCI-based musical instrument that uses
a combination of qualitative and quantitative information,i.e., the classification of musical imagery and the sonification of brain activity. This instrument was designed for use
during a concert performance. It is our dream to convert
natural expressions such as the mental images of emotions
or memories directly into musical imagery, which would befree from physical constraints. This goal has only been
partly achieved so far, but this was a first step that helps toelucidate the relationship between musical expression andthe performance intent. As mentioned in Section 5.4, themultidimensional evaluation of this project need to addressscientific and artistic issues. Thus, we will continue to explore our experiments and performances interactively because they are the most fascinating parts of this project.

7.

REFERENCESACKNOWLEDGMENTS

This research was conducted as a joint research project between JST, ERATO, Okanoya Emotional Information Project(OEIP), Tokyo University of the Arts, and University ofTsukuba. The author would like to thank Ryu Nakagawa(Tokyo University of the Arts) for his help with visualization, as well as Reiko Hoshi-Shiba (The University ofTokyo), Hidefumi Ohmura (OEIP), and Gen Hori (Asia
University) for advice related to the experiments and analyses.

54