audio-caf: allow devices to only set min buffer sizes

Required for Sony MSM8960 devices.

This is a port of codeworkx's patch from the mako audio repo.
x
Merge remote-tracking branch 'upstream/cm-10.1' into cm-10.1