bitstream: fix a crash in case of oom

fixes ticket #2766

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>