Pro Audio is Easy, Consumer Audio is Hard
Linux Audio Confernce 2010

Lennart Poettering
lennart@poettering.net

May 2010

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Who Am I?

Software Engineer at Red Hat, Inc.
Developer of PulseAudio, Avahi and a few other Free Software
projects
http://0pointer.de/lennart/
lennart@poettering.net
IRC: mezcalero

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Not musician

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Not musician
Didn’t graduate in signal processing

Lennart Poettering

Pro Audio is Easy, Consumer Audio is HardWhy?

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Why?
Correct misconceptions

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Why?
Correct misconceptions
Improve cooperation, share code and interfaces

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Why?
Correct misconceptions
Improve cooperation, share code and interfaces
It might just be interesting

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

General Background

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

General BackgroundNobody makes money off Linux Audio

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

General Background
Nobody makes money off Linux Audio
Only 3 people payed to hack on general purpose infrastructure by
the big distributors

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

However, we have allies, particularly in RT, Mobile, also V4L

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Pro Audio: knowledgable user

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Pro Audio: knowledgable user
Consumer Audio: laymen users

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Pro Audio: good hardware

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Pro Audio: good hardware
Consumer Audio: crappy hardware (Both the PC, and the Audio
HW, Feature Monsters)

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Pro Audio: Audio is primary purpose of the machine

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Pro Audio: Audio is primary purpose of the machine
Consumer Audio: Audio is one thing among many

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Pro Audio: User owns the machine, full configurability, user ==administrator

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Pro Audio: User owns the machine, full configurability, user ==administrator
Consumer Audio: user != administrator

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Different Optimization Targets:

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Different Optimization Targets:
Pro Audio: low latencies, functionality

Lennart Poettering

Pro Audio is Easy, Consumer Audio is HardDifferent Optimization Targets:
Pro Audio: low latencies, functionality
Consumer Audio: low power usage, dynamically changing
environments, robustness

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Latency vs. Power Usage

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

In Pro Audio: minimize latency at price of power usage, anddynamic configuration changes

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

In Pro Audio: minimize latency at price of power usage, anddynamic configuration changes
In Consumer Audio: choose latency as high as possible, and as low
as necessary, allow configuration changes

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Result for consumer: we need to change latencies

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Result for consumer: we need to change latencies
Unfortunately, most hardware/drivers cannot do that

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Result for consumer: we need to change latencies
Unfortunately, most hardware/drivers cannot do that
Also: hardware/drivers limited in there range: ideally we have one
wakeup every 1sLennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Hence: schedule audio with system timers

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Hence: schedule audio with system timers
That’s difficult, because system timers deviate from sound card
timers, and sound card timers are crap

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Time interpolation

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Time interpolation
Linear Regression + Spline interpolation

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Time interpolation
Linear Regression + Spline interpolation
Exponentional backoff

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Time interpolation
Linear Regression + Spline interpolation
Exponentional backoff
Everything twice

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

To make long latencies useful: support seeking

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

To make long latencies useful: support seeking
Requires history, for remixing

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Dropouts, require dynamic adjustments of sleep times

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Timing and transfer block granularity

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Outlook, moving timing into the kernel?

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Non-interpolated timing has advantages?

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Non-interpolated timing has advantages?
Sound card IRQs are accurate!

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Algorithm to use? DLL?

Lennart Poettering

Pro Audio is Easy, Consumer Audio is HardWe cannot trust clients, one misbehaving client may not beallowed to stall the entire pipeline

Lennart Poettering

Pro Audio is Easy, Consumer Audio is HardRT CPU time calculations change, need to preemptively increase
sleep times when streams are moved, removed and added

Lennart Poettering

Pro Audio is Easy, Consumer Audio is HardRT CPU time calculations change, need to preemptively increase
sleep times when streams are moved, removed and added
Why? Never drop out

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Unreliable latencies, dynamically changing

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Compromises in RT code, IO loops are not constant timeLennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Multiple encoded formats, conversion/resampling required, butideally zero-copy

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Multiple encoded formats, conversion/resampling required, butideally zero-copy
Need to support S16LE and FLOAT32, and a lot more

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Large latency means exchanging of large blobs

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Large latency means exchanging of large blobs
Zero copy (because sizeof(metadata) ¡ sizeof(payload))

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Cached audio samples

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Cached audio samples
Means ref counting

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Cannot expose low-level hw controls

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Cannot expose low-level hw controls
Need to merge and extend in software

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Cannot expose low-level hw controls
Need to merge and extend in software
Difficult: Routing in the Sound HW seldom known (USB!)

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Common: unreliable dB data

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Common: unreliable dB data
If the mixer controls even existLennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Machine not owned by the user and user not knowledgable

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Machine not owned by the user and user not knowledgable
Means no low-level control on RT =¿ rtkit

Lennart Poettering

Pro Audio is Easy, Consumer Audio is HardNo memory locking

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Compromises, compromises, compromises

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Cooperation PA, JACK

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Use more standard desktop infrastructure. Tear down theseperating walls.

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Cooperation on APIs?

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Cooperation on Timing?

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Cooperation on Protocols? RTP?

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Come to LPC!

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

Come to LPC!
Meet RT folks, V4L folks, MM folks

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

That’s all, folks.

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard

That’s all, folks.
Any questions?

Lennart Poettering

Pro Audio is Easy, Consumer Audio is Hard