audio-caf: allow devices to only set min buffer sizes

Change-Id: I182f86549b01d4c452cc94201c638c410592083e