PHASERINGS FOR IPAD ENSEMBLE AND ENSEMBLE DIRECTOR AGENT

PhaseRings for iPad Ensemble and Ensemble Director Agent is an improvised musical work exploring the use of dynamic touchscreen instruments, tracked by a gesture-classifying agent, to enhance the creativity of an ensemble of performers. The PhaseRings app has been designed specifically for ensembles to create expressive music with simple percussive gestures. Each performer can play with a small set of notes selected from a musical scale that is common to the whole group. The agent tracks the performers’ gestures and reacts to moments of heightened gestural change by allowing the performers to access a new set of notes from a new scale. In this way, this performance links collaborative creativity to the dynamic interface of the individual touch-screen instruments. 1. DESCRIPTION OF THE WORK In this performance, an ensemble of improvising iPad performers using the PhaseRings iPad app are tracked and directed by an Ensemble Director Agent running on a server. While artificial intelligence agents are often used as improvisation partners in computer music, they have only rarely been used to conduct or direct a performance. The agent software in this performance, Metatone Classifier, tracks performers by classifying their touches according to a vocabulary of percussive gestures. It encourages and rewards the group for creative interactions by updating their app interfaces with new notes and sounds when it detects increases in gestural variety and change. Performance with a network of computer music interfaces has been explored as early as the late 1970s when The League of Automatic Music Composers connected their early personal computers as part of their compositions [1]. Weinberg has described such interconnections, where data is shared directly between interfaces, as a Local Performance Network [7]. An alternative model for networked computer was postulated by Pressing [6] who suggested that an intelligent agent could serve as a musical director, monitoring information from the performers, applying tests, and then issuing commands or interrupting processes in response. In the present work, the concept of an Ensemble Director Agent has been implemented in the Metatone Classifier software which tracks improvised ensemble interactions during the performance. As the PhaseRings instrument runs on touch-screens, the This work is licensed under Creative Commons Attribution Non Commercial 4.0 International License. The full terms of the License are available at http://creativecommons.org/licenses/by-nc/4.0 Figure 1: Screenshot of the PhaseRings app showing rings that trigger eight different notes and the GUI button to change setup (exposed by the Ensemble Director Agent). morphology of the instrument can be updated during the performance in response to signals from the agent. So, this performance connects improvised collaborative creativity to the capabilities of the musical instrument. In the following sections the technical details of the PhaseRings app and Metatone Classifier agent will be described. 2. THE PHASERINGS APP The PhaseRings app [3] is an annular interface for percussive ensemble performance. Each player is given a number of notes from a particular scale which are shown as rings on the screen (see Figure 1). These rings can be tapped for short sounds, or swirled to create different kinds of long sounds. These notes change throughout the performance in response to signals from the agent. Each performer’s notes are taken from the same scale so that, while each player can explore a unique melodic space, their harmonic position is common to the group. PhaseRings’ interaction with the gesture tracking agent has been the subject of a series of HCI studies where multiple iterations of the interface were analysed [5]. The final interface follows a mixed-initiative model [2]; signals from the agent expose a button in the app GUI and the interface changes only if a member of the ensemble chooses to tap the button. This model allows the performers to retain ultimate control over interface changes while incorporating “suggestions” from the Ensemble Director Agent. The 22nd International Conference on Auditory Display (ICAD–2016) July 2-8, 2016, Canberra, Australia Figure 2: The ANU Experimental Music Studio performing with PhaseRings and Metatone Classifier at You Are Here 2015. 3. METATONE CLASSIFIER Metatone Classifier is the Ensemble Director Agent used in this performance. This software tracks multiple touch-screen performers simultaneously and calculates measures of the overall ensemble behaviour. Each second, the performers’ touches are classified into one of nine continuous gestures, then, the recent history of calculated gestures are compiled into transition matrices, similar to calculating a first-order Markov model. These transition matrices can be calculated for individual performers, or averaged over the whole ensemble, giving a high level representation of the group’s behaviour. In this performance, a metric called flux is applied to these ensemble transition matrices which measures how much performers change between different gestures. If the agent detects a sharp increase in flux, possibly related the start of a new musical section in the improvisation, it sends a signal to PhaseRings suggesting that the interface could be updated. Further details of the implementation of Metatone Classifier are available elsewhere [4]. 4. REALISATION AT ICAD 2016 At ICAD 2016, this work will be realised by the ANU Experimental Music Studio, a flexible group of musicians with members studying musical performance, composition and other disciplines at the Australian National University. This group previously performed with PhaseRings and Metatone Classifier at the You Are Here 2015 festival in Canberra (see Figure 2) and members of the group have also participated in a series of studies and workshops to analyse and improve the design of these systems throughout 2014 and 2015.