Proceedings of the 2nd AES Workshop on Intelligent Music Production, London, UK, 13 September 2016

ADEPT: A FRAMEWORK FOR ADAPTIVE DIGITAL AUDIO EFFECTS
Owen Campbell1 , Curtis Roads1 , Andrés Cabrera1 , Matthew Wright3 , and Yon Visell2
Media Arts and Technology
Media Arts and Technology and Electrical and Computer Engineering
University of California, Santa Barbara
3Center for Computer Research in Music and Acoustics
Stanford University
owencampbell@mat.ucsb.edu
1There are many examples of adaptive effects, such as dynamic range processors, gates, and pitch-correctors that havelong been mainstays of music production. Previous research
has investigated organizational methods for adaptive digital audio effects [1, 2]. [1] proposed a general taxonomy ofadaptive effects, describing auto-adaptive, external-adaptive,
feedback-adaptive, and cross-adaptive techniques. [2] went
beyond purely signal-level models of content with a comprehensive treatment of ’content-based transformations’. Related research has investigated adaptive processing for automated mixing tasks, for example [3, 4]. [5] and [6] focusedon implementing flexible frameworks for user-defined adaptive effects routing using audio plugins as target effects.
2. SYSTEM DESIGN AND IMPLEMENTATIONThe ADEPT adaptive audio processing software utilizes existing audio plugins and implements feature extraction anda mapping algorithm in order to facilitate ADAFx processing. As a result it can control any processing parametersthat are made available as automatable controls within thedigital audio workstation (DAW). Developed using JUCE, aC++ framework which enables cross-platform deploymentof audio plugins, it communicates with Ableton Live using
Open Sound Control [7] and the LiveOSC MIDI RemoteScript. ADEPT works with the DAW and existing plugins, providing immediate benefits due to the large variety

Input Audio

1. BACKGROUND(Samples)(Seconds)

Extracted
PitchValues (Hz)The field of music information retrieval (MIR) has enabledresearch in ’intelligent’ audio processing. Emerging applications of MIR techniques in music production might provide mixing engineers, musicians and composers control
over extant effects processing plugins based on the audio
content of the recorded music. In adaptive digital effects(ADAFx) processing, mapping functions are used to modulate algorithm parameters via features that are extracted
from the input audio signal or signals. We present an audio software plugin that is designed to facilitate featureparameter mappings within digital audio workstations. We
refer to it as the Adaptive Digital Effects Processing Tool
(ADEPT).

and quality of available effects processing and synthesis algorithms, and integration with production and performance
practices.
Each instance of the ADEPT plugin computes a stream ofaudio features using Essentia [8], a C++ audio feature extraction library. Audio features from a given source may bemapped to any plugin parameter in the DAW session, including those belonging to plugins residing on other tracks.
At present, the feature-parameter mappings are one-to-many;a single feature stream may feed multiple mappings controlling different parameters. The software makes it easy to realize evocative real-time effects (Fig. 1) via feature-driven
effects automation (Fig. 2) controlled through the graphicaluser interface of a software plugin (Fig. 3).

(Analysis Frames)

Decay TimeParameter
Values

ABSTRACT(Analysis Frames)

Output Audio

2(Samples)(Seconds)

Figure 1: Example time series of audio feature and effectparameter data with mapping from pitch to reverb decaytime
3. EVALUATION
Two pilot studies were conducted to explore potential evaluation methods and to gain insight into the strengths andweaknesses of the software prototype. Qualitative data wascollected regarding the utility of the proposed ADAFx routing framework, the user experience, and the aesthetic quality of the audio effects produced by participants in study.
Part I was based on music production tasks to be performed

Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).

Proceedings of the 2nd AES Workshop on Intelligent Music Production, London, UK, 13 September 2016

OSC
Messagesof New Music Research, vol. 32, no. 1, pp. 95–114,2003.

DAW

Decay TimeUpdates

Audio InADEPT

Unaltered
AudioReverb

Audio Out

Figure 2: Block diagram of pitch to reverb decay time mapping[3] J. D. Reiss, “Intelligent systems for mixing multichannel audio,” in 2011 17th International Conference onDigital Signal Processing (DSP), pp. 1–6, IEEE, 2011.
[4] E. Perez-Gonzalez and J. Reiss, “Automatic equalization of multichannel audio using cross-adaptive methods,” in Audio Engineering Society Convention 127,
Audio Engineering Society, 2009.
[5] M. E. Stabile, C. Roads, S. T. Pope, M. Turk, andJ. Kuchera-Morin, “Adapt: A networkable plug-in hostfor dynamic creation of real-time adaptive digital audioeffects,” 2010.
[6] Ø. Brandtsegg, “A toolkit for experimentation with signal interaction,” in Proceedings of the 18th International Conference on Digital Audio Effects (DAFx-15),
pp. 42–48, 2015.
[7] M. Wright, A. Freed, and A. Momeni, “Opensoundcontrol: State of the art 2003,” in Proceedings of the2003 conference on New interfaces for musical expression, pp. 153–160, National University of Singapore,
2003.

Figure 3: View of the ADEPT plugin GUI
with the ADEPT plugin or with conventional methods. In
Part II, expert listeners were asked to evaluate the audio output from Part I. Due to the small number of participants wecannot draw strong conclusions, but the results suggest thatsuch a system may be a valuable addition to conventional
mixing and performance workflows for effects processing
and synthesis.

[8] D. Bogdanov, N. Wack, E. Gómez, S. Gulati, P. Herrera, O. Mayor, G. Roma, J. Salamon, J. R. Zapata, andX. Serra, “Essentia: An audio analysis library for music information retrieval.,” in ISMIR, pp. 493–498, Citeseer, 2013.

4. CONCLUSION
We developed a conceptual framework for flexible adaptive
effects routing in the DAW environment and investigatedpotential use cases for such a system. A software implementation of this framework was created in the form of anaudio plugin, where we explored the utility of a number ofaudio features and an interface for defining control mappings. Finally, we conducted a small two-part pilot study toevaluate the software’s utility and integration with existing
workflows.
5. REFERENCES[1] V. Verfaille, U. Zolzer, and D. Arfib, “Adaptive digitalaudio effects (a-dafx): A new class of sound transformations,” IEEE Transactions on audio, speech, and language processing, vol. 14, no. 5, pp. 1817–1831, 2006.
[2] X. Amatriain, J. Bonada, l. Loscos, J. L. Arcos, andV. Verfaille, “Content-based transformations,” Journal
Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).