Don't allow 0-sized buffers in downsample