Composing and Performing Digital Voice Using
Microphone-Centric Gesture and Control Data
Kristina Warren
University of Virginia
kmw4px@virginia.edu

ABSTRACTDigital voice is a rich area for compositional and performative research. Existing voice-technology work frequently entails a division of labor between the composertechnologist, who creates the hardware/software, devises
the formal structure, and writes about the work; and theperforming vocalist, who may have some creative or improvisational input. Thus, many scientific papers on digital voice lack an authorial performance perspective. Thispaper aims to imbue performance back into the discussion of digital voice, drawing from my own experience asa composer-technologist-vocalist. In addition, many
novel controllers for vocal performance are glove- andhand-centric, but in fact the hands are auxiliary to vocal
performance. I propose the Abacus interface, which ismounted directly on the microphone, as a more literally
voice- and mouth-centric means of controlling digitalvoice. The Abacus treats rhythm, pitch, and noise parametrically and tracks vocal gesture input to modulate
among various processing states.

1. INTRODUCTION
Digital voice comprises a vast, rich sonic palette. Though
we are accustomed to considering the abstract compositional voice, and to using voice as an inspiration for developing affective audience connection in composition[1], the area of digital voice demands more focused research. Few authors of scientific papers on digital vocal
compositions are themselves vocalists, and many voicebased controllers are rooted in hand motion and havevery little to do with the precise gestural work possible in
the lips, teeth, tongue, and vocal tract. I propose that themouth is a prime site of vocal control, and thus my micmounted interface called the Abacus takes steps toward
evaluating the gestural and control potential of digitalvoice.

2. RELATED WORK
Many early voice-technology works were tape compositions whose primary source material was recorded
speech. Such speech-based compositions still thrive, particularly in the legacy of Swedish text-sound composition
and related compositional styles [4, 9, 16]; analysis ofthese works emphasizes intelligibility of the text [2].
Copyright: ©2016 Kristina Warren. This is an open-access article distributed under the terms of the Creative Commons Attribution License 3.0
Unported, which permits unrestricted use, distribution, and reproduction
in any medium, provided the original author and source are credited.

More recent voice-technology works, by contrast, explore
timbre and the act of performance. Works by Trevor
Wishart exemplify this compositional style. Wishart typically records a vast array of sounds, many based in extended vocal techniques, and subsequently applies frequency-domain transformations, such as spectral shiftingand stretching, to create fixed media compositions [15].
Today, works for digital voice emphasize live vocal
processing – particularly video [10] and polyphonic [12]extensions of voice – and hand- and limb-centric controllers. The visual beauty of glove- and hand-based controllers, such as those employed by Imogen Heap, Laetitia
Sonami, and Pamela Z, guides the audience toward anunderstanding not only of digitized vocal sound but alsoof nuances of performer affect and identity [7, 11, 18].
Nonetheless, despite the primacy of the hands in therealm of haptic sensing and gesture [14] the hands are infact auxiliary to vocal performance, and therefore morestudy is needed of voice- and mouth-centric controllers.
Furthermore, the writing on digital vocal music is dominated by the perspectives of non-vocalist composers (notable exceptions include [6, 13, 17]), so it is necessary to
rediscover the performance perspective within digital
vocal music.

3. DIGITAL VOICE
3.1 Live Processing in Max/MSP
In performance, I employ two Max/MSP units I built
for live vocal processing: [rhyGran] and [glchVox].
These consist of rhythmic, granulation, and frequencydomain effects of my own making, as well as Michael
Norris’ Soundmagic Audio Unit plug-ins for spectral
processing.1 Both units can use either live or prerecorded
samples; [rhyGran] tends to produce normative, voicelike sounds, while [glchVox] often yields prominently
digital, glitchy sounds, but both modules can be variably
employed. Processing on the live voice signal is similar,
alternately maintaining or ablating the vocal character ofthe original signal. This patch is meant to work in conjunction with my vocal performance practice, which consists of both extended techniques and more traditionalsinging styles.
3.2 Performative Affect, Novel Controller
My recent research has turned increasingly toward thematter of performative affect. I find that the potency ofmouth as source of gestural and control data is enhanced1

http://www.michaelnorris.info/software/soundmagic-spectral

by eye contact and engagement with the audience. Thus,
the Abacus is part of an effort to streamline my own performance affect.
In early work with the aforementioned Max/MSP modules, I used a tablet (first a Wacom tablet2 and subsequently a Google Nexus tablet3) to control live processingof the voice. Later, I decided for aesthetic and performative reasons that the tablet was not sufficient as a controller. First, the sonic output of my work tends to be denseand sculptural, and the two-dimensional tablet feels unsuited to this sound. Moreover, I experienced a growing
desire to get “out from behind” the laptop in order to develop a greater connection with the audience. My work
points toward the mouth as a crucial site of voice. Whilehands and limbs aid in expressivity, they are in fact auxiliary to vocal performance, so I began to develop a novel
controller to be mounted directly on the microphone . I
argue that the most important vocal gestures originate in
the mouth and vocal tract, rather than the hands.

4. ABACUS INTERFACE
4.1 Versions 1 and 2ometer. The toggles are the primary source of control
data; the button triggers recording of live vocal samples;the LEDs report state and time information, and the potentiometer controls gain.

Figure 1. Abacus version 3.0.

Inspired by the ancient adding tool of the same name,the Abacus treats control data parametrically. There arethree possible control states: Rhythm, Noise, and Pitch.
Each control state consists of a three-dimensional control
space whose axes are a Short Term parameter, a Long
Term parameter, and a Texture parameter. Toggles 3-4
allow navigation along the x-axis, Short Term; Toggles
5-6 along the y-axis, Long Term; and Toggles 7-8 along
the z-axis, Texture. Each pair of Toggles (3-4, 5-6, 7-8)

The first two versions of the Abacus used an Arduino
Teensy4 to send digital and analog control input to thecentral Max/MSP patch, and included several LEDs toprovide visual feedback during performance. These early
Abacus versions were mounted on a breadboard/protoboard.

Figure 3. 3D control space, Toggles 3-8.

outputs four possible values: 0, 1, 2, 3. Thus, there are4^3 = 64 possible positions within this 3D control space.
Toggle #1

Figure 2. Abacus version 1.

4.2 Versions 3.0 and 3.1
Abacus 3.0 and 3.1 continue to use an Arduino Teensy
and maintain the exposed-wires aesthetic of the earlier
versions. Version 3.1 dispenses with the breadboard andinstead uses thermoplastic and suede cabling to achieve
direct adherence to the microphone and clip. Abacus 3.1
consists of 8 toggles, 1 button, 2 LEDs, and 1 potenti2

http://www.wacom.com/en-us
https://www.google.com/nexus/
4
https://www.arduino.cc/
3

23-4
5-6
7-8Function
Backward/forward among states(saved before/during performance)
Listen to RVGs
Short Term control axis (x)Long Term control axis (y)Texture control axis (z)Table 1. Toggle functions.

Routing Vocal Gestures (RVGs) determine the control
state (Rhythm, Noise, or Pitch); examples of Routing
Vocal Gestures are shown in Table 2. There are two important limitations on the control ability of the RVGs:(1) RVGs do not directly control sound processing.
Instead, they route control data from Toggles 3-8.
(2) Toggle 2 controls whether the patch listens forRVGs. If Toggle 2 is in the “off” position, an RVG will
not cause a change in control state.

My habit is to vocalize continually throughout a given
piece, establishing a symbiosis between the sounds
emerging directly from my mouth and those emerging
from the Max/MSP patch. Because of this continuousExample Routing Vocal Gesture(RVG)“inhale k” = unpitched; inhale withoccasional tightening of soft palate,yielding sucking “k” sound“hum n” = pitched; hum, rapidly
touch tongue to front palate, yielding closed-mouth “n” sounds“8vb” = pitched; harmonic undertone from false vocal fold vibration
at slower frequency (e.g. f/2)
“fast in/ex” = unpitched; rapid inhales, exhales; hyperventilation-like“ae” = pitched; short nasalized repetitions of [ae] vowel (as in “cat”)“lip squeak” = pitched; upper teeth
on moistened lower lip, inhale yielding one or more gliss pitches

ResultingControlStateRhythm

NoisePitch

Table 2. Example RVGs and control states.

The three axes of the 3D control space are Short Term,Long Term, and Texture. This organization: (1) allows
both detailed and general control during performance, (2)promotes a balance between composition and improvisation, and (3) acts as a mnemonic during performance. It issomewhat conceptually difficult to establish an exact
parallel between, for instance, the Short Term axis in theRhythm and Pitch control states, or between the Long
Term axis in the Rhythm and Noise control states. Nonetheless, the Short Term axis is meant to give some information about the immediate character of the sound; theLong Term axis, about the phrase-level organization ofsounds; and the Texture axis, about the relationship ofpatch voices or layers to one another.
Several control axes bear further explanation. “Cyber,”the Long Term axis of the Noise control state, refers to
the frequency of timbre changes and thus the implied“cyberization” of vocal samples. “Continuity,” the Long
Term axis of the Pitch Control state, refers to consonanceand apparent modulation. “Solo,” the Texture axis of thePitch control state, describes the extent to which the digital voices or layers form a singular, coherent counterpoint
to my vocal input.
Finally, Toggle 1 allows toggling backward to the previous state. If I arrive at a desirable configuration of settings, I can save this and return to it later using Toggle 1.

vocalization, it is disadvantageous for RVGs to be toosensitive. Thus, though RVGs can serve a control function, they are intended to be primarily gestural.
Control x-axis Control y-axis Control z-axis(Short Term)(Long Term)(Texture)= Toggles 3-4= Toggles 5-6= Toggles 7-8
Control State = Rhythm
Triggered by Rhythm RVG, e.g. “inhale k”
MeterLoopSync
0 = slow, ir0 = 10-20%
0 = 10-20%
regularchance repeat
slaving to mas…
same rhythm
ter rhythm
3 = fast, regu…
…
lar
3 = 80-90%
3 = 80-90%
chanceControl State = Noise
Triggered by Noise RVG, e.g. “8vb”
TimbreCyber
Density
0 = dark
0 = mostly un- 0 = indiv. lines…
altered voice
discernible
3 = bright…
…
3 = mostly
3 = wash ofprocessed
soundControl State = Pitch
Triggered by Pitch RVG, e.g. “ae”
IntervalContinuity
Solo
0 = mostly
0 = scattered,
0 = accomp.
small (< min3) granular
live voice…
…
…
3 = mostly
3 = phrasing,
3 = soloistic
large (> Maj6) key apparent
counterpoint
Table 3. Control states and values.

5. IMPLEMENTATION OF ABACUS
5.1 couldn’t, voice and stereo audio
My piece couldn’t (2016)5 is a studio-based composition
made with Abacus 3.1; it comprises two sections. The
first section consists mainly of rhythmic manipulations ofthe live voice signal and some live samples thereof. The
second section emphasizes noise and textural density; I
recorded my vocal performance with several microphones and fed these dry and processed versions backinto the Abacus as gestural data during mixing. I made acompositional decision to use “inhale k,” “8vb,” and “ae”
as the RVGs for Rhythm, Noise, and Pitch respectively;
these techniques as well as proximity work among themicrophones comprise the bulk of the vocal performance.
A primary goal of couldn’t is to use the Abacus toblend compositional and improvisational work. The main
vocal line was a single improvised performance in thestudio, eight minutes in duration. This was recorded withthe Shure SM58 microphone on which the Abacus ismounted, and simultaneously with a Rode NT1-A. This
Rode track was then fed back into the Abacus during
mixing to add layering and depth. The Rode’s greater
sensitivity yielded a more precise translation of vocal
signal into control data. I played with temporal displacement, sometimes sending control information from theAbacus at the same time points in the fixed Rode track as
I originally did when performing with the Shure, but atother times developing a new temporal control path. All
together, these layers give a sense of ghosts in the performance: the live voice and its associated processing do
not always co-occur.
5

Audio available at nitious.bandcamp.com

6. FUTURE DIRECTIONS
I intend to incorporate text into the interface design. I am
interested in text as both sonic/semantic material and instruction for action. Thus, incorporating a live text videoscore will shape my body and mouth in real time, andwill allow the audience a stylized close view of my vocalwork.
The voice inherently carries much timbral and stylistic
flexibility, and recent spectral analysis of recordings [8]and laryngoscopic studies of expressive and extended
vocal techniques [3] begin to shed light on the vast performative potential of voice. More study is needed tointegrate this directly with the practice of technologized
academic music. In addition, singing voice synthesis [5]is a vast and promising field which demands greater connection to digital-vocal work, for instance as a performance partner to a human vocalist.
Acknowledgments
I am grateful to Peter Bussigel for his guidance in designing and building the Abacus interface, and to Daniel Jolliffe, Collin Bradford, Thomas Ouellet Fredericks, andSeejay James for their development of Serial-based Arduino and Max patches that allow communication between
the Teensy and Max/MSP.

7. REFERENCES[1] C. Bahn, T. Hahn, and D. Trueman, “Physicality andFeedback: A Focus on the Body in the Performanceof Electronic Music,” Proceedings of the 2001
International Computer Music Conference, Havana,
2001, pp. 44-51.
[2] A. Bergsland, “The Maximal-Minimal Model: Aframework for evaluating and comparing experienceof voice in electroacoustic music,” Organised
Sound, vol. 18, no. 2, pp. 218-228, 2013.
[3] D.Z. Borch, J. Sundberg, P.-Å. Lindestad, and M.
Thalén, “Vocal fold vibration and voice source
aperiodicity in ‘dist’ tones: a study of a timbral
ornament in rock singing,” Logopedics Phoniatrics
Vocology, vol. 29, no. 4, pp. 147-153, 2004.
[4] W. Brunson, “Text-Sound Composition – TheSecond
Generation,”
ProceedingsoftheElectroacoustic Music Studies Network 09, Buenos
Aires, 2009.
[5] P. Cook, “Singing Voice Synthesis: History, Current
Work, and Future Directions,” Computer Music
Journal, vol. 20, no. 3, pp. 38-46, 1996.
[6] M. Guilleray, “Towards a fluent electronic
counterpart of the voice,” Masters thesis, Institute ofSonology, Royal Conservatory of the Hague, 2012.
[7] D. Haraway, A cyborg manifesto: Science,technology, and socialist-feminism in the late 20thcentury, Springer Netherlands, 2006.

[8] S. Lacasse and C. Lefrançois, “Integrating Speech,
Music, and Sound: Paralinguistic Qualifiers inPopular Music Singing,” Proceedings of 2008
IRCAM Expressivity in Music and Speech
Conference, Campinas Brazil, 2008.
[9] C. Lane, “Voices from the Past: compositionalapproaches to using recorded speech,” Organised
Sound, vol. 11, no. 1, pp. 3-11, 2006.
[10] G. Levin and Z. Lieberman, “In-Situ Speech
Visualization in Real-Time Interactive Installation
and Performance,” Proceedings of the 3rdInternational Symposium on Non-PhotorealisticAnimation and Rendering, Annecy France, 2004.
[11] G. Lewis, “The Virtual Discourses of Pamela Z,” J.
Society for American Music, vol. 1, no. 1, pp. 57-77,
2007.
[12] P.J. Maes, M. Leman, K. Kochman, M. Lesaffre andM. Demey, “The ‘One-Person Choir’: A
Multidisciplinary Approach to the Development ofan
Embodied
Human-ComputerInterface,”
Computer Music Journal, vol. 35, no. 2, pp. 22-35,
2011.
[13] C. Stamper, “Our Bodies, Ourselves, Our Sound
Producing Circuits: Feminist Musicology, Access,and Electronic Instrument Design Practice,” Masters
thesis, Mills College, 2015.
[14] C. Udell and J. Sain, “eMersion | Sensor-controlled
Electronic Music Modules & Digital Data
Workstation,” Proceedings of the International
Conference on New Interfaces for MusicalExpression, London, 2014, pp. 130-133.
[15] T. Wishart, “The Composition of ‘Vox-5,’”
Computer Music Journal, vol. 12, no. 4, pp. 21-27,
1988.
[16] A. Woloshyn, “The Recorded Voice and theMediated Body in Contemporary CanadianElectroacoustic Music,” PhD dissertation, Universityof Toronto, 2012.
[17] A. Young, “The Voice-Index and Digital VoiceInterface,” Leonardo Music Journal, vol. 24, pp. 35, 2014.
[18] M. Young, “Latent body – plastic, malleable,
inscribed: The human voice, the body and the soundof its transformation through technology,”
Contemporary Music Review, vol. 25, no. 1-2, pp.
81-92, 2006.