audio-caf: allow devices to only set min buffer sizes

Required for Sony MSM8960 devices.

This is a port of codeworkx's patch from the mako audio repo.
x