avcodec/hevc/extract_rbsp: fix error return type

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>