rmdec: fix crash in case of oom

Fixes ticket #2724

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>