dnxhddec: return the correct number of bytes from decode_frame

Fixes Ticket2022

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>