# The LuminUs: Providing Musicians with Visual Feedback on the Gaze and Body Motion of Their Co-performers

Evan Morgan, Hatice Gunes, Nick Bryan-Kinns

► To cite this version:

Evan Morgan, Hatice Gunes, Nick Bryan-Kinns. The LuminUs: Providing Musicians with Visual Feedback on the Gaze and Body Motion of Their Co-performers. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.47-54, 10.1007/978-3-319-22668-2_4. hal-01599872

HAL Id: hal-01599872

https://inria.hal.science/hal-01599872

Submitted on 2 Oct 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# The LuminUs: Providing Musicians with Visual Feedback on the Gaze and Body Motion of their Co-Performers

Evan Morgan, Hatice Gunes, and Nick Bryan-Kinns

School of Electronic Engineering and Computer Science
Queen Mary University of London, Mile End Road, E1 4NS, London, U.K.
{e.l.morgan, h.gunes, n.bryan-kinns}@qmul.ac.uk

**Abstract.** This paper describes the LuminUs - a device that we designed in order to explore how new technologies could influence the inter-personal aspects of co-present musical collaborations. The LuminUs uses eye-tracking headsets and small wireless accelerometers to measure the gaze and body motion of each musician. A small light display then provides visual feedback to each musician, based either on the gaze or the body motion of their co-performer. We carried out an experiment with 15 pairs of music students in order to investigate how the LuminUs would influence their musical interactions. Preliminary results suggest that visual feedback provided by the LuminUs led to significantly increased glancing between the two musicians, whilst motion based feedback appeared to lead to a decrease in body motion for both participants.

**Keywords:** Musical interaction, computer-supported cooperative work, groupware, eye-tracking, social signals, non-verbal communication.

## 1 Introduction

There has been much focus recently on the use of wearable sensors for lifestyle, sports and activity tracking. Our research concerns the use of wearable devices as tools for understanding and enhancing interactions between musicians during co-present performances. In a previous study we assessed the suitability of various sensors for providing continuous information on the affective and behavioural aspects of collaborative music making [1]. Following on from this work, we decided to design and test a device that would enable musicians to receive real-time sensor-based feedback on aspects of their collaborative interactions. We chose to focus specifically on two non-verbal interaction features that have important expressive and communicative roles in musical performance - *eye gaze* and *body motion*.

Studies have shown that gaze has a variety of functions in non-verbal communication, including the expression of emotional and interpersonal information such as liking, attentiveness, competence, attraction, and dominance [2, 3]. It may also be used as a means of directing attention towards shared references
and surveying activity in the external environment [4]. Schutz notes that visual co-presence during collaborative music making allows each performer to share “in vivid present the Other’s stream of consciousness in immediacy” [5, pp. 176]. Despite its apparent significance, surprisingly few studies have been conducted on the use of gaze during collaborative music making. Davidson and Good [6] observed that members of a string quartet used “conversations with the eyes” to convey important information while playing. A study with a string duet found that visual contact had a positive influence on timing synchrony between musicians for certain types of music [7].

Eye-tracking technology makes it possible to automatically track where someone is looking. Existing studies have predominantly investigated how eye-tracking might be used to support and evaluate the success of computer-supported collaborative work [8,9]. Music related studies have generally focused on the potential for eye-tracking to be used as a control medium (e.g. selecting notes by glancing at specific markers - for a review see [10]). We are not aware of any studies that have used eye-tracking to investigate or support the interpersonal and affective functions of gaze during collaborative musical interactions.

As an intrinsic feature of non-verbal communication, body motions are closely tied to gestural [11] and emotional expression [2]. Humans are able to successfully identify emotions from simple point-light representations of body motion [12,13]. More specifically, studies have found evidence for a link between measures of body motion activity and the arousal/activation aspects of emotion [14,15]. Body motion analysis has also been applied to the study of affect in musicians [16]. Motion measurements can be obtained using accelerometers worn at positions of interest. Many consumer electronic devices - such as smart phones, smart watches, and activity trackers - already contain accelerometers, from which data can be accessed using custom built applications.

In this paper we describe the design of the LuminUs - a device that provides musicians with real-time visual feedback on either the gaze or body motion of their co-performers. We also report preliminary results from a study that investigated the impact that the LuminUs has on dyadic improvised performances.

## 2 The LuminUs

We designed the LuminUs with the aim of exploring how technology might be used to provide performing musicians with an increased awareness of each other, especially in situations where their mutual attention might be hindered by complex musical interfaces and physical obstructions. The LuminUs consists of a strip of 16 coloured LEDs that can be controlled individually to provide dynamic visual information (see Fig. 1(a)). The device is mounted on a flexible arm so that it can be positioned in front of the musician. Control of the lights is provided via USB connection to a computer. The design was chosen based upon three criteria for the way in which feedback should be presented to the musician: *visual* (as opposed to haptic or audio); *minimal* (so as not to be overly distracting); and *dynamic* (to represent time varying signals).
<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 1: (a) The LuminUs: different colours were used for gaze (1) and motion (2) feedback; (b) Annotated image of the experimental setup.</figcaption></figure>

The LuminUs has two modes of operation - *motion feedback* and *gaze feedback*. In the motion feedback mode we measure the body movements of each musician using small wireless accelerometers worn around the waist. These movements are then processed and displayed on the LuminUs. The greater the movement, the more lights are illuminated - with the colours of the lights ranging from green (low) to red (high). In the gaze feedback mode we use eye-tracking glasses to detect when one musician is glancing towards the other. The LuminUs then visually notifies the musician who is being glanced at. In this case, more lights become illuminated as the duration of the glance increases. For gaze feedback the colours of the lights range from blue (short) to purple (long).

For eye-tracking we used the Pupil headset, which is an open-source hardware and software project [17]. The headset tracks the movement of the right eye using a single infra red camera, and simultaneously captures the wearer's point-of-view (POV) using a separate forward-facing video camera. Following a short calibration (~20 sec), the software is able to map the gaze point of the wearer onto the live POV video. We modified the pupil software so that it could recognise 2D markers within the POV images. Consequently, by placing markers on the headsets of each musician (see Fig. 1(b)), we are able to automatically register the moments when one person is glancing at the other.

# 3 The Experiment

Our hypotheses concerning the way in which the LuminUs would influence musical interactions were as follows:

**H1** Providing gaze feedback would increase the overall amount of gaze during collaborative interaction.

**H2** Providing motion feedback would stimulate increased awareness of the other participant, as indicated through more glancing at the other.

**H3** Providing motion feedback would increase the overall amount of motion during collaborative interaction.
To test these hypotheses we designed an experiment in which pairs of pianists and percussionists were asked to create live improvised accompaniments to a silent animation. This task was chosen for a number of reasons. Firstly, the improvisational nature of the task encouraged the musicians to work collaboratively and heightened the need for mutual awareness. Secondly, the animation meant that the musicians had more to attend to than simply their instrument and co-performer - as would often be the case in real-world performances. In addition to this, it provided a visual stimulus to guide and influence the improvisations created by each pair.

Four two minute long animations were short-listed for use in the study. We then asked a separate group of 24 musicians to watch and rate the animations based upon their suitability for improvised musical accompaniment. The animation that received the highest average rating was chosen for use in the study.

## 3.1 Method

15 percussionists and 15 pianists were recruited from music colleges in London. The participants comprised 7 females and 23 males aged between 18 and 38 (M = 22.7, SD = 4.7). Their playing experience ranged from 3 to 33 years (M = 11.6, SD = 5.6). The participants were assigned to percussionist-pianist pairs such that the individuals in each pair did not know one another.

There were seven experimental conditions, coded as: G-G, G-X, X-G, M-M, M-X, X-M, and X-X. The first letter represents the feedback that the LuminUs provided to the percussionist, whilst the second letter represents the feedback provided to the pianist - either gaze feedback (G), motion feedback (M), or no feedback (X).

The experiment was held in a performance space, with overhead stage lighting and black curtains surrounding the performance area. The setup consisted of a large screen, which showed the animation and provided instructions to the participants. The two participants were then positioned facing the screen, but angled slightly towards one another. The percussionist was provided with two electronic drum pads (snare and floor tom) and an electronic ride cymbal, which they played standing up. The pianist was provided with a 61 note keyboard and sustain pedal, which they also played standing up. Both instruments were connected to a computer via MIDI, and the audio was output through speakers positioned either side of the screen. Each participant's LuminUs was positioned on a stand in front of them, such that it was just below their line of sight to the screen. The devices were positioned such that each participant could only see their own device, and the brightness of the lights was adjusted so that no reflections were visible. Figure 1(b) shows the experimental setup as seen from below the screen.

Upon arriving for the experiment the participants were given a couple of minutes to play the instruments together. This gave them a chance to make brief introductions and familiarise themselves with the instrumental set up. The participants were then fitted with the accelerometers and eye tracking headsets, which were subsequently calibrated for each participant. All of the eye-tracking
and motion data was time-stamped and saved. The experiment began with a
warm up session, where participants watched the animation twice without play-
ing their instruments and then twice whilst playing along. Following this, the
participants were given roughly one minute to discuss ideas. This was the last op-
portunity for them to verbally interact during the experiment. The remainder of
the experiment consisted of seven improvisation sessions, each randomly assigned
to one of the seven conditions discussed above. Prior to each session each par-
ticipant's LuminUs would show either green-red (motion feedback), blue-purple
(gaze feedback), or no lights (no feedback) to indicate which type of feedback
they would receive in that session. This was done so that neither participant
would know what kind of feedback the other was receiving. In each session the
participants had two attempts to play along to the animation, with a 10 second
gap between. After each session the participants completed a short questionnaire
to rate aspects of their experience and subjective opinions relating to that par-
ticular session. The analysis of our questionnaire results is out of the scope of
this paper.

**3.2 Results and Analyses**

To test **H1** and **H2** we decided to look at the average number of glances that each participant made towards their co-performer within each condition. This information was extracted from our eye-tracking data. Figure 2(a) shows the mean and standard error for the number of glances averaged over participants within each condition. We see that the mean values for both the pianist and percussionist are lowest in the conditions where they are not receiving any feedback from the LuminUs. We can also see that, on average, the pianist tended to glance more than the percussionist.

To test **H3** we extracted the mean body motion for each participant within each condition. The results are shown in Fig. 2(b). We can see that the percussionist moved more than the pianist, as might be expected. More interestingly, it appears that the body motion values tend to be lower when the participants are receiving motion feedback, compared to gaze feedback.

We used the Wilcoxon signed rank test to perform statistical analyses of some of the relationships in figures 2(a) and 2(b). In particular, we looked at the differences between the mean values obtained in the 6 conditions where the LuminUs was active, relative to the equivalent means for the inactive LuminUs condition (X - X). The results are shown in Table 1. The results for the mean number of glances indicate that there were significant differences between all of the conditions and the X - X condition. This effect is greatest for condition 1, where both participants had gaze feedback (r = 0.7, p = .000). The second strongest effect is for condition 2, where only the percussionist had gaze feedback (r = 0.57, p = .002). The third strongest is for condition 4, where both participants had motion feedback (r = 0.52, p = .005).

For the body motion results we see that none of the differences are significant.
However, of potential interest is the fact that a mild negative effect is observed for
condition 4 (r = -0.29, p = .106), where both participants had motion feedback.
<figure><img src="image_4.png" /><figcaption>Fig. 2: Bar graphs showing (a) the mean number of glances; and (b) the mean body motion within each of the seven conditions. G = gaze feedback, M = motion feedback, X = no feedback. Error bars: ± 1 SE.</figcaption></figure>

In addition to this, the only other weak negative effects correspond to the other conditions involving motion feedback. For the conditions involving gaze feedback (1-3) the effect sizes are small (< 0.2).

## 3.3 Discussion

H1 and H2 are supported by our findings, which indicate that when participants were receiving feedback from the LuminUs they glanced towards each other significantly more than when no feedback was provided. The size of this effect was greatest for the condition where both participants had gaze feedback. However, it is possible that the main effects were simply due to the LuminUs serving as a reminder of the other musician's presence. We intend to carry out further analyses to investigate the nature of these effects in more detail. In particular,

<table><caption>Table 1: Statistical results for the differences between means in conditions 1-6 (LuminUs in use), relative to those in condition 7 (LuminUs not in use).</caption><thead><tr><th rowspan="2">Condition</th><th colspan="4">Number of Glances:</th><th colspan="4">Body Motion:</th></tr><tr><th>Z</th><th>r</th><th>n</th><th>p</th><th>Z</th><th>r</th><th>n</th><th>p</th></tr></thead><tbody><tr><td>(1) G - G</td><td>3.81***</td><td>0.70</td><td>27</td><td>.000</td><td>1.00</td><td>0.18</td><td>30</td><td>.318</td></tr><tr><td>(2) G - X</td><td>3.14**</td><td>0.57</td><td>28</td><td>.002</td><td>0.30</td><td>0.05</td><td>30</td><td>.766</td></tr><tr><td>(3) X - G</td><td>2.46*</td><td>0.45</td><td>27</td><td>.013</td><td>0.24</td><td>0.04</td><td>30</td><td>.813</td></tr><tr><td>(4) M - M</td><td>2.83**</td><td>0.52</td><td>28</td><td>.005</td><td>-1.61</td><td>-0.29</td><td>30</td><td>.106</td></tr><tr><td>(5) M - X</td><td>2.43*</td><td>0.44</td><td>28</td><td>.015</td><td>-0.87</td><td>-0.16</td><td>30</td><td>.382</td></tr><tr><td>(6) X - M</td><td>2.61**</td><td>0.48</td><td>28</td><td>.009</td><td>-1.14</td><td>-0.21</td><td>30</td><td>.254</td></tr></tbody></table>

Z = Wilcoxon signed rank z-statistic, r = effect size, n = sample size, p = two-tailed p-value. *p < .05, **p < .01, ***p < .001.
we will look at how the specific timings of glances relate to the light output of
the LuminUs. This will allow us to see whether musicians responded directly
to the feedback, or whether it had a more general influence on their behaviour
during the sessions.

Regarding **H3**, our results were less conclusive, but appeared to indicate
that motion feedback led to decreased body motion. We envisaged that pro-
viding feedback on the motion of a co-performer would, if anything, encourage
participants to move more. This hypothesis was partly influenced by studies on
behavioural mimicry, which suggest that the actions or emotions displayed by
one person can cause congruent behaviour in another person [18]. The fact that
an opposite trend was observed is a potentially interesting finding. Unfortunately
there is an absence of existing research on the behavioural effects of providing
real-time motion feedback, so at this stage we can only speculate as to any
causality. A possible explanation is that the musicians felt self-concious when
they knew their movements were being displayed on the LuminUs. However, if
this was the case then we might expect a similar effect for the gaze feedback.
Furthermore, none of the musicians indicated feelings of self-conciousness when
we informally interviewed them after the experiments.

The results presented in this paper indicate that the LuminUs had an in-
fluence upon the *objective* aspects of the musicians’ behaviours. However, at
this stage it is not possible for us to say whether these were detrimental or
beneficial to the musical interaction as a whole. Our ongoing analyses will in-
vestigate whether the use of the LuminUs influenced the subjective experiences
of the musicians; the quality of their coordination; and the musical outcomes
of their performances. In order to investigate this we will use the self report
data collected from the questionnaires alongside the audio and video recordings
of the performances. An additional consideration is that the short duration of
the study may not have provided the musicians with sufficient opportunity to
familiarise themselves with the LuminUs and make effective use of its feedback.
In future work it would be interesting to investigate how musicians appropriate
the LuminUs over a more sustained period of usage.

4 Conclusion

This paper provides a brief introduction to the LuminUs, and some preliminary
results from experiments, where we put it to the test with trained musicians. Our
initial findings suggest that providing real-time co-performer gaze and motion
feedback can have a measurable impact on aspects of non-verbal communica-
tion and behaviour during collaborative musical performances. Our continued
analyses will attempt to ascertain whether or not the LuminUs also influenced
the *quality* of the interactions between musicians and the accompaniments they
created. Furthermore, through detailed analysis of our eye-tracking, motion, and
video data, we hope to gain a better understanding of how musicians use non-
verbal communication during live performances.
**Acknowledgements** This research is supported by the Media and Arts Technology Programme, an RCUK Doctoral Training Centre in the Digital Economy.

## References

1. Morgan, E., Gunes, H., Bryan-Kinns, N.: Using affective and behavioural sensors to explore aspects of collaborative music making. Int. J. of Human-Computer Studies **82** (2015) 31–47

2. Kendon, A.: Some functions of gaze-direction in social interaction. Acta Psychologica **26** (January 1967) 22–63

3. Kleinke, C.L.: Gaze and eye contact: a research review. Psychological bulletin **100**(1) (July 1986) 78–100

4. Argyle, M., Graham, J.A.: The central Europe experiment: Looking at persons and looking at objects. Environmental Psychology and Nonverbal Behavior **1**(1) (1976) 6–16

5. Schutz, A.: Making music together. Collected papers II (1976)

6. Davidson, J.W., Good, J.M.M.: Social and Musical Co-Ordination between Members of a String Quartet: An Exploratory Study. Psychology of Music **30**(2) (October 2002) 186–201

7. Vera, B., Chew, E., Healey, P.: A Study of Ensemble Performance Under Restricted Line of Sight. In: Proc. of the Int. Conf. on Music Inform. Retrieval, Curitiba, Brazil (2013)

8. Vertegaal, R.: The GAZE groupware system: mediating joint attention in multi-party communication and collaboration. In: Proc. of the SIGCHI Conf. on Human factors in computing syst. - CHI '99, New York, USA, ACM Press (1999) 294–301

9. Chanel, G., Bétrancourt, M., Pun, T., Cereghetti, D., Molinari, G.: Assessment of Computer-Supported Collaborative Processes using Interpersonal Physiological and Eye-Movement Coupling. In: Affective Computing and Intelligent Interaction (ACII13), Geneva, Switzerland (2013)

10. Hornof, A.J.: The Prospects For Eye-Controlled Musical Performance. In: Int. Conf. on New Interfaces for Musical Expression (NIME14). (2014) 461–466

11. Goldin-Meadow, S.: Beyond words: the importance of gesture to researchers and learners. Child development **71**(1) (2000) 231–9

12. Walk, R.D., Homan, C.P.: Emotion and dance in dynamic light displays. Bulletin of the Psychonomic Society **22**(5) (November 1984) 437–440

13. Clarke, T.J., Bradshaw, M.F., Field, D.T., Hampson, S.E., Rose, D.: The perception of emotion from body movement in point-light displays of interpersonal dialogue. Perception **34**(10) (2005) 1171–1180

14. Castellano, G., Villalba, S.D., Camurri, A.: Recognising Human Emotions from Body Movement and Gesture Dynamics. In: Affective Computing and Intelligent Interaction (ACII07), Lisbon, Portugal, Springer-Verlag (2007) 71–82

15. Kleinsmith, A., Bianchi-Berthouze, N.: Affective Body Expression Perception and Recognition: A Survey. IEEE Trans. on Affective Computing (2012) 1–1

16. Dahl, S., Friberg, A.: Visual Perception of Expressiveness in Musicians' Body Movements. Music Perception: An Interdisciplinary J. **24**(5) (June 2007) 433–454

17. Kassner, M., Patera, W., Bulling, A.: Pupil: An Open Source Platform for Passive Eye Tracking and Mobile Gaze-based Interaction. CoRR abs/1405.0006 (April 2014)

18. Chartrand, T.L., Lakin, J.L.: The Antecedents and Consequences of Human Behavioral Mimicry. Annu. review of psychology (September 2012)