Simplify audio buffer handling

This ensures audio is always padded and simplifies the code at the same
time.