Proc. of the 15th International Symposium on CMMR, Online, Nov. 15-19, 2021

Towards an Aesthetic of Hybrid Performance Practice:
Incorporating Motion Tracking, Gestural and TelematicTechniques in Audiovisual Performance
Haruka Hirayama1 and Ioannis Zannos2
1

Hokkaido Information University, Dept. of Information Media2
Ionian University, Dept. of Audiovisual Arts
hiraryama@do-johodai.ac.jp
zannos@gmail.com

Abstract. This paper discusses composing works for interactive live performancebased on the comparison between recent artworks and experimental work methods of the two authors. We compare the different interaction design strategies
employed and discuss the factors which influenced the choice of methods formotion tracking and their influence on body movements when coupled with thegeneration of sound during the performance. We consider the resulting artworks
as hybrid artforms that combine aspects of music composition, improvised soundperformance, and stage performance or dance. A high-level comparison of thetechnical and practical aspects of said works is provided. It can be argued that the
new expressive potential and the wealth of possibilities to be explored warrants
further work in this direction, and that systematic comparison of the interactive
characteristics and expressive affordances of the systems developed are useful inguiding further research in the development of novel hybrid performance forms.
Keywords: Interactive Music Performance, Audio-Visual Art, Gesture Mapping,Telematic Art, Embodied Performance

1Introduction

In acoustic music performance we can say that the actions or gestures of performers
also provide visual cues conveying the character and shape of sound. At the same time,a performer’s actions or gestures are directly coupled to the characteristics of produced
sound and its musical expressive characteristics. Overall, we can say that instrumental
music performance is a form of multimodal interaction synthesis [1].
The shapes of performer’s gestures are dictated to a large extent by the physicalproperties of the instrument they are using, and appropriate techniques of performanceare required to play it functionally and effectively. In addition, performers’ intentions
with regard to musical expression influence the form of characteristic gesture shapes.
Therefore, acoustic music performance creates a fairly strict framework within which
performers must stay in order to interact with their instruments in a musically effectiveway. The actual shape of gestures is usually regarded as playing an auxiliary role in theexperience of the performance.

251Proc. of the 15th International Symposium on CMMR, Online, Nov. 15-19, 2021

However, in the context of interactive computer music composition we can create
new relationships between performance gestures and sound and we can choose more
freely both the types of movements and the degree and type of their influence on theresulting sound. This leaves greater margins of freedom to explore the expressive potential of performance gestures from the viewpoint of their visual expressive impact.
This leads to a hybrid form of expressive art that lies between visual art and music orother types of stage performance.
This paper discusses composing works for interactive live performance based on
the comparison between recent artworks and experimental work methods of the twoauthors. The works are: People in the Dunes, [2] and [3] created by Haruka Hirayama
in collaboration with a visual artist and choreographer Bettina Hoffmann, and IDEFantasy, created by Iannis Zannos in collaboration with dancers Jun Takahashi andAsayo Hisai (Japan) and Tasos Pappas-Petrides, Vasiliki Florou, Natali Mandila andMary Randou (Greece) [4].
We compare the different interaction design strategies employed in the above works
and discuss the factors which influenced the choice of methods for motion tracking
and their influence on body movements during the performance, when directly coupled with the generation of sound during the performance. We consider the resulting
artworks as hybrid art forms that combine aspects of music composition, improvised
sound performance, and stage performance or dance. We discuss the degree to which
system design allowing dancers to develop their individual or intuitive style of performance, with reference to the affordances created by the technical characteristics of thesystems employed. Several unresolved problems arise with regard to both performance
practice and the aesthetic appreciation of such works.
The extent of possible couplings of body movement to sound forms is vast, and thetask of choosing or designing interaction strategies is daunting. This problem is furthermore compounded by technical limits in the accuracy and response time of movement
tracking devices and by the complex, at times almost entirely unpredictable behaviourof the couplings between movement and the resulting sounds, both in terms of the physical or mathematical behaviour and from the perceptual viewpoint. However, we argue
that the new expressive potential and the amount of possibilities to be explored warrants
further work in this direction, and that systematic comparison of the interactive characteristics and expressive affordances of the systems developed are useful in guiding
further research in the development of novel hybrid performance forms. The present
paper presents a simple methodology based on a classification of the interaction techniques used in the works mentioned, and evaluating their potential based on practicalfactors experienced during our work.

2The Performances

2.1

People in the Dunes

The People in the Dunes project explores expressive potential of performance with realtime sound processing as a live audio-visual art that exists at the intersection of interactive music performance and visual art involving human bodies. In this work, human252

Proc. of the 15th International Symposium on CMMR, Online, Nov. 15-19, 2021

body movement plays a theatrical role while at the same time working as a medium forsound conveyance and a form of music embodiment in a manner similar to instrumental
performance in music.
The People in the Dunes project consists of three works: People in the Dunes I, TheEmbodiment I - Strings, and People in the Dunes II. These have been created and performed in Tokyo, Montreal, and Gatineau in Canada between 2018 and 2020. The titleof the project is inspired by the novel The Woman in the Dunes by Kobo Abe, that depicts the situation of a man trapped in the dunes fighting the ever flowing sand, reflecting
about his life and in the end becoming aware of its essence and finding freedom: howhuman bodies and movements eventually find new directions under the influence of theforces acting from multiple directions between multiple individual actors, particularly
under restricted circumstances? This project has been further developed by involving
local dancers and instrumentalits working in Butoh and other contemporary styles.

2.2

IDE-Fantasy

The objective of IDE-Fantasy is to create an interactive performance which can be realized in remote locations at the same time, through the collaboration of dancers ineach location, and relying entirely on motion capture data from the dancers. The pieceeschews any transmission of images or sounds between the locations of the performances. The presence of the performers is transmitted between the remote stages of theperformances based solely on the influence of their tracked movements on the sounds
which are produced locally at each stage. Both the performers and the audience must
rely on the sounds locally created by sound synthesis software to reconstruct in their
imagination the actions or states of the performers in remote locations. The objective
is to explore the narrative and interpretive potential of strictly reduced means for representation and the capability for sensing the states and of the performers based on thedata traces left by their body movements, but without having direct visual or auditory
contact.
The subject matter of the performance is inspired by the story of Izutsu, a Japanese
Noh Play, which talks of the encounter of a monk with the ghost of a woman that islonging for reunion with her lover and husband from her previous life. Additionally, asa cultural reflection of the idea of correspondences between remote locations, symbolic
correspondences between Izutsu and the myths of Echo and Narcissus and of Daphnis
and Chloe are being explored for future realisations of this work.
The piece was developed through a series of rehearsals in Tokyo, Athens and Corfu.
So far, telematic rehearsals have been realised between Athens and Corfu and Athensand Jerusalem. A performance between Stanford (USA), Athens and Corfu was presented in March 2019 at the LAC19 conference. This performance was combined witha presentation of the software framework used to create the piece [5]. A local only
performance was presented at TAMA Music Festival in 2020. Further realisations arebeing prepared in conjunction with ongoing rehearsals and the development of newtechniques for data capture and transmission.

253

Proc. of the 15th International Symposium on CMMR, Online, Nov. 15-19, 2021

3

Methodolody, Design Considerations, Discussion

From the perspective of an interactive music composer, the following research questions
are addressed in both projects discussed: What are the expressive possibilities of music
composition motivated by performance gestures? What is the theoretical framework
required to create links between the shape formed by human bodies and sound, andbetween changes of shapes and sound transformations? Also, what are the technicalmeans for linking physical body movement to sound production, and how can these
influence the artistic process and its final outcome?
In the case of People in the Dunes, the experience of the composer’s previous work
FRISKOTO raised questions about the difference between performance gestures andcontrol gestures in music. The hypothesis was posed, that the difference consists inthe possibility of perceiving gestures as the animating force of music or not [1]. To
make bodily movement perceivable as an animating force of music it is important todevelop a system where sound can give an instant reaction to the movement, and viceversa. Furthermore, it is important to consider the correspondences between visual andauditory percepts. Visual and auditory sensations need to be properly coordinated orcorresponded, in other words, their correspondences should be readily recognizable toperformers as well as viewers. The following aspects guided the creative process andthe design of the performances as a whole:
1. The availability of movement capture technologies and their technical performance
characteristics (accuracy, reliability, temporal and measuring resolution, latency);
2. Affordances of the movement tracking devices for the performers (which movements are easy and comfortable for the performers to execute while using the tracking devices, and how do they understand the relationship between their movements
and the resulting data when using the device);
3. Design of the mechanisms for influencing the sound produced based on the data
received from the tracking devices;
4. Correspondence of the perceptual characteristics of the available or chosen sonicvocabulary to those of the gestural vocabulary developed by the performers;
5. Narrative effects of the sequential ordering of sequence of motions types and associated sound textures. The alternation of different motion types and types of soundtextures produced by these can provide cues to the audience for understanding thecausal relationship between movements and sounds, and thus aid their understanding of or identification with the performance. In addition, the ordering of motion
types and sound textures can form a type of sequence of scenes that create theimpression of a narrative, albeit of a fairly abstract and vague type. This plays animportant role in capturing the attention of the audience, by offering hints for fabricating an interpretation of the events of the performance in their own imagination;
6. Subtle changes and minimalism. At certain parts of the performance, it is helpful
to heighten the sensibility and awareness of the audience by purposely focusing on
minute movements or changes of sound. This can intensify the sense of tension andthe interpretive potential of the piece.
In the development of IDE-Fantasy we started with simple mappings between movement and synthesis parameters as few as 1 or 2, and gradually introduced more param-254

Proc. of the 15th International Symposium on CMMR, Online, Nov. 15-19, 2021

eters. Even extremely simple parameter - sound mappings proved to be useful performance tools for dancers, providing them with instruments which they could explore
very easily, but where nevertheless rich and responsive enough for short improvisations. In this approach, 3 or 4 parameters were already sensed as being hardly possible
to handle or adapt to. 6 parameters per person were definitely outside the realm of feasible. We also experimented with 6-parameter chaotic algorithms cojointly performed bytwo dancers. These did capture the attention of the performers, even though they proved
to be difficult to master. In table (1) we summarise the techniques used in our works.

1. Movement
tracking device
2. Number ofattached sensors
per person

People in the Dunes I People in the Dunes II
IDE-Fantasy
Built-in sensors ofa) 9-axis movement
KinectiPhoneb) 3-axis accelerometers-2 iPhones per person Up to 2 per person

3. Sensor positioning-4. Tracking information

Horizontal
boundary position

5. Data transmission
protocol
6. Software(s) forinteractive systems

Left and rightlower arms
Acceleration,
Magnetic field,
Gyroscope

USB

WiFi

Max, JitterMax, ZIGSIM (iOS)Wrists
Acceleration,
Magnetic field,
Gyroscope
WiFi,
Xbee Mesh Network
SuperCollidera) Boundary
Prerecorded voice,a) Prerecorded samples
microphones
cello, traffic,b) Simple or Complexb) Prerecorded
environmental sound, UnitGenerator graphs
voice soundsynthesiserswith or without feedback
Table 1. Technologies used in People in the Dunes and IDE Fantasy

7. Audio source forcomposition

4Conclusion, Future WorkIn all, a common trait observable in both works discussed here is the use of the technicalaffordances of motion tracking devices and sound generation or processing algorithms
to design a sort of performance language which combines body movements and theirassigned sound textures or events to create narratives of a more or less abstract type. In
both pieces, concrete narratives of previously existing and well known works provided
a reference framework in order to create the more abstract narrative of the pieces.
In conclusion, the main challenges confronting this kind of work stem from the abstract and indirect nature of digital mediation between bodily movement and generated
audiovisual stimuli. The causal relationship between movement and generated soundor video tends to be difficult to recognise. In some cases it can be entirely absent, as
is when employing chaotic synthesis algorithms. To counterbalance these obstacles, it255

Proc. of the 15th International Symposium on CMMR, Online, Nov. 15-19, 2021

is necessary to create interpretive or narrative links with the performers and the audience. Simple mappings and complex or chaotic correspondences both present advantages and disadvantages, and the decisive design criteria for developing a functioning
performance seem to lie in semiotic domains such as the choice of sounds, images, andmovements for their associative semantic charge, and the devising of narrative devices
through trial and error during rehearsals. Currently we are interested in employing Machine Learning algorithms in order to devise improved methods for translating motion
to sound, and in particular in experimenting with unsupervised learning and in adaptive
techniques that modify their behaviour during the performance itself.
At the same time, the characteristics and affordances of motion tracking devices(shapes of sensors, kind of detecting data, mobility etc.) have a direct impact on theavailable movements and thereby on the kind of body movement language that theperformers develop. We feel that a combination of different type of tracking method
can enhance performance expression.
The body is capable of constantly adapting and changing shapes or forms [6]. We
realised that there is an interdependence between isolated movements of individual partsof the body and the perception of forms created by movements of the body as a whole.
This will serve as a guiding principle for the currently planned experiments for future
work.

References
1. Collins, Karen, Bill Kapralos, and Holly Tessler, eds. The Oxford Handbook of Interactive
Audio. Oxford University Press, Oxford (2014).
2. People
intheDunes
project
page,
https://www.facebook.com/
peopleinthedunes/, last accessed 2021/06/15
3. People in the Dunes II videos, https://vimeo.com/user88406194, last accessed
2021/06/15
4. IDE-Fantasy videos, https://ide-fantasy.tumblr.com/, last accessed 2021/06/15
5. Zannos, I.: SC-Hacks: A Live Coding Framework for Gestural Performance and Electronic
Music. In: Proceedings of the 2019 Linux Audio Conference. pp. 121-128. CCRMA, Stanford
University (2019).
6. Kobayashi, Y., Matsuura, H., eds. The Discourse of Représentation Skinship: The Rhetoric ofthe Body. University of Tokyo press, Tokyo (2000).

256