Audio Engineering Society

Student DesignCompetition
Presented at the 141st Convention
2016 September 29 – October 2, Los Angeles, CA, USA

ADEPT: A Framework for Adaptive Digital Audio Effects
Owen Campbell1 , Curtis Roads1 , Andrés Cabrera1 , Matthew Wright3 , and Yon Visell2
1 Media Arts and Technology , University of California Santa Barbara
2 Media Arts and Technology and Electrical and Computer Engineering , University of California Santa Barbara
3 Center for Computer Research in Music and Acoustics , Stanford University

Correspondence should be addressed to Owen Campbell (owencampbell@mat.ucsb.edu)

ABSTRACTThe field of music information retrieval (MIR) has enabled research in ‘intelligent’ audio processing. Emerging
applications of MIR techniques in music production might provide mixing engineers, musicians and composers
control over extant effects processing plugins based on the audio content of the recorded music. In adaptive
digital effects (ADAFx) processing, mapping functions are used to modulate algorithm parameters via featuresthat are extracted from the input audio signal or signals. We present an audio software plugin that is designed
to facilitate feature-parameter mappings within digital audio workstations. We refer to it as the Adaptive Digital
Effects Processing Tool (ADEPT).

1Introduction

The field of music information retrieval (MIR) has led
to the development of numerous signal processing algorithms for extracting semantic information from audio
signals. By repurposing some of these audio feature
extraction algorithms, emerging applications of MIR
techniques in music production might provide mixing
engineers, musicians and composers control over extant effects processing and synthesis plugins driven by
the content of musical performances.
In adaptive effects processing, mapping functions areused to modulate algorithm parameters via features thatare extracted from the input audio signal or signals suchthat the effects operate in a signal-dependent way [1].
There are many examples of adaptive effects, such as
dynamic range processors, gates, and pitch-correctorsthat have long been mainstays of music production.

However, in principle any effect could become an adaptive effect if one or more of its processing parametersettings could be dynamically driven by signal content.
For example, a vocalist could create a control mappingthat increases the amount of reverb as he or she singshigher and higher pitches, or the dissonance of electric guitar music could be mapped to the intensity or
timbre of a distortion effect. These examples are just asmall subset of the possible mappings and novel adaptive effects made possible by a flexible adaptive effects
framework.This document describes a proposed system for flexible ADAFx routing using existing music
production software tools.

2Methods

As in [2] and [3], this work proposes an approach toadaptive effects processing in which existing audio
software tools can be augmented to facilitate arbitrary

Campbell et al.

A Framework for Adaptive Digital Audio Effects

control mappings between the content of input audio
signals and the behavior of audio processing and synthesis algorithms. Modern digital audio workstations(DAWs) possess all the capabilities necessary to enable
the control mappings required for a generalized framework for adaptive effects processing. An abundance ofdiverse, high-quality effects processing and synthesis
algorithms in the form of audio plugins and support forinter-application communication for remote DAW control are two critical components of the system. DAWs
lack only the feature extraction pipeline and mapping
capabilities required to complete the system.
The ADEPT adaptive audio processing software implements feature extraction and a mapping algorithm
in order to facilitate ADAFx processing (Fig. 1). As
a result it can control any processing parameters thatare made available as automatable controls within theDAW. Developed using JUCE, a C++ framework whichenables cross-platform deployment of audio plugins, it
communicates with Ableton Live using Open Sound
Control [4] and the LiveOSC MIDI Remote Script .
ADEPT works with the DAW and existing plugins, providing immediate benefits due to the large variety andquality of available effects processing and synthesis
algorithms, and integration with production and performance practices.
Each instance of the ADEPT plugin computes a streamof audio features using Essentia [5], a C++ audio feature extraction library. Audio features from a given
source may be mapped to any plugin parameter inthe DAW session, including those belonging to plugins residing on other tracks. At present, the featureparameter mappings are one-to-many; a single featurestream may feed multiple mappings controlling different parameters. Initial experimentation with the systemsuggests that there is great potential in this capability,
and there are far more possible mappings to investigate
than can be explicitly described in this document. For
any given effect or synthesis algorithm there are a number of parameters which, when modulated in real time,
produce output that sounds markedly different fromthe same algorithm with static settings. Some of thesepotential mappings will produce useful or interesting
results and others will not. Much of this work has been
devoted to discovering the former.

3the software. Examples were tested using segments ofhigh-quality multi-track recordings from MedleyDB[6].
Case example 1: One of the simplest but most compelling examples is the mapping from vocal pitch to
the decay time of a reverb unit (Fig. 2, Fig. 3a). As
the detected pitch [7] increases, so does the reverb’s
decay time, resulting in a very dry application of theeffect for the lowest notes of the singer’s range and avery wet, sustained reverb for the highest notes. This
relationship, though applied in a rather extreme mannerin this example, is useful for automatically avoiding
muddy low notes with too much reverb while applying
a desirable amount of reverb to higher notes.
Audio Feature
DrivenControl
Messages

Parameter
AutomationAPIFX Parameter
Updates

Audio InADEPTPluginUnaltered
AudioDigital
Audio
FX Plugin

Audio OutFig. 1: Each instance of the plugin computes audio
features a stream of audio features which canfeed mapping functions for controlling audio
plugin behavior.

Results

To illustrate the capabilities of the ADEPT software, wedescribe two case examples of adaptive effects using
AES 141st Convention, Los Angeles, CA, USA, 2016 September 29 – October 2
Page 2 of 5

A Framework for Adaptive Digital Audio Effects

Input AudioCampbell et al.

(Samples)Extracted
PitchValues (Hz)(Seconds)

Decay TimeParameter
Values(Analysis Frames)

Output Audio(Analysis Frames)(Samples)(Seconds)

Fig. 2: Example time series of audio feature and effect parameter data with mapping from vocal pitch to reverb
decay time.

AES 141st Convention, Los Angeles, CA, USA, 2016 September 29 – October 2
Page 3 of 5

Campbell et al.

A Framework for Adaptive Digital Audio Effects(a) View of the ADEPT plugin GUI with mapping from vocal pitch to reverb decay time.

(b) View of the ADEPT plugin GUI with mapping from guitar dissonance to distortion drive.

Fig. 3: Users can activate mappings between selected audio features and processing parameters (background) andaccess simple controls and visualization of mapping function behavior (foreground) via the plugin’s GUI.
AES 141st Convention, Los Angeles, CA, USA, 2016 September 29 – October 2
Page 4 of 5

Campbell et al.

A Framework for Adaptive Digital Audio Effects

Case example 2: In another example, the dynamic
application of a guitar distortion effect is based on
three different audio feature mappings. The primary
adaptive processing relationship maps higher detectedlevels of dissonance [8] to lower settings for the distortion unit’s ‘drive’ parameter (Fig. 3b) , thus applying
heavy distortion to power chords and single notes andmoderate distortion to more complex chords. In thenext mapping, the ‘tone’ parameter tracks the detected
brightness value (as approximated by spectral centroid[9]). Finally, the detected loudness [10] modulates the‘dry/wet’ parameter to automatically make the distortion wetter at low levels and dryer at high levels tomanage the dynamic range of the effect’s output.

4Discussion

The adaptive reverb effect in case example 1 above
demonstrates that the introduction of even a single adaptive control mapping to a familiar effect can have dramatic results. By combining multiple such mappings,
it is easy to turn static effects into highly dynamic ones,
as in the adaptive distortion described in case example
2 above. There are many other potential applications,among them several that have been implemented using
the ADEPT system, including a cross-adaptive automated panning scheme, an onset-ducking delay effect,
and a peak-tracking resonant filter.
With more development time the ADEPT plugin will
support logical comparison or arithmetic combinationof multiple feature streams for implementing many-tomany mappings, enabling the full taxonomy of adaptive
effects outlined in [1]. Future work will also explore
the use of additional audio features, more complex mapping paradigms, and extensions to the user interface.

5Summary

We developed a conceptual framework for flexible adaptive effects routing in the DAW environment and investigated potential use cases for such a system. A
software implementation of this framework was created in the form of an audio plugin, where we exploredthe utility of a number of audio features and an interface
for defining control mappings.

References[1] Verfaille, V., Zolzer, U., and Arfib, D., “AdaptiveDigital Audio Effects (a-DAFx): A New Class ofSound Transformations,” IEEE Transactions on
Audio, Speech and Language Processing, 14(5),pp. 1817–1831, 2006, ISSN 1558-7916, doi:10.
1109/TSA.2005.858531.
[2] Stabile, M. E., Roads, C., Pope, S. T., Turk, M.,
and Kuchera-Morin, J., “Adapt: A Networkable
Plug-in Host for Dynamic Creation of Real-Time
Adaptive Digital Audio Effects,” 2010.
[3] Brandtsegg, O., “A Toolkit for Experimentation
with Signal Interaction,” in Proceedings of the
18th International Conference on Digital Audio
Effects (DAFx-15), pp. 42–48, 2015.
[4] Wright, M., Freed, A., and Momeni, A., “Opensound Control: State of the Art 2003,” in Proceedings of the 2003 Conference on New Interfaces
for Musical Expression, pp. 153–160, National
University of Singapore, 2003.
[5] Bogdanov, D., Wack, N., Gómez, E., Gulati, S.,Herrera, P., Mayor, O., Roma, G., Salamon, J.,
Zapata, J. R., and Serra, X., “Essentia: An Audio
Analysis Library for Music Information Retrieval.”in ISMIR, pp. 493–498, Citeseer, 2013.
[6] Bittner, R. M., Salamon, J., Tierney, M., Mauch,M., Cannam, C., and Bello, J. P., “MedleyDB: A
Multitrack Dataset for Annotation-Intensive MIR
Research.” in ISMIR, pp. 155–160, 2014.
[7] Brossier, P. M., Automatic Annotation of MusicalAudio for Interactive Applications, Ph.D. thesis,Queen Mary, University of London, 2006.
[8] Plomp, R. and Levelt, W. J. M., “Tonal consonance and critical bandwidth,” The journal of theAcoustical Society of America, 38(4), pp. 548–560, 1965.
[9] Peeters, G., “{A large set of audio features forsound description (similarity and classification)in the CUIDADO project},” 2004.
[10] Vickers, E., “Automatic Long-Term Loudness andDynamics Matching,” in Audio Engineering Society Convention 111, Audio Engineering Society,2001.

AES 141st Convention, Los Angeles, CA, USA, 2016 September 29 – October 2Page 5 of 5