jpeg2000: check that nreslevels2decode has been initialized before use

Fixes out of array accesses

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
jpeg2000dec: Propagate error code from get_cox() correctly

Without this the context state could become inconsistent

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
jpeg2000: Make nreslevel fields int

This prevents potential integer overflows

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>