shorten: set invalid channels count to 0

Prevent the loop shorten_decode_close from writing and freeing out of
the array boundary.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

CC: libav-stable@libav.org

(cherry picked from commit c10da30d8426a1f681d99a780b6e311f7fb4e5c5)
shorten: K&R formatting cosmetics

(cherry picked from commit a2ad554def214d2d03b7c16f68dc081a8622f9ca)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
shorten: report meaningful errors

(cherry picked from commit 4c364eb2b856fc33cf7b42f7c7b979e69fde5f3a)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
shorten: use the unsigned type where needed

get_uint returns an unsigned value, use an unsigned to store
blocksize to make sure the comparison logic is correct and report
correctly the error for the channel count not supported.

CC: libav-stable@libav.org

(cherry picked from commit 5cf7c72757779a740e897a97710aac044fe5258c)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>