Btrfs-progs: elaborate error handling of mkfs

$./mkfs.btrfs -f /dev/sdd -b 2M
[...]
mkfs.btrfs: volumes.c:845: btrfs_alloc_chunk: Assertion `!(ret)' failed.
Aborted (core dumped).

We should return error to userspace instead of the above.

Signed-off-by: Liu Bo <bo.li.liu@oracle.com>