avformat/rmdec: fix return code of ff_rm_parse_packet()

Broken by aecb9d3
Fixes assertion failure
Fixes Ticket3042

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
avformat/rmdec: add some error messages

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>