tta: make probing more robust

Signed-off-by: Paul B Mahol <onemda@gmail.com>
j2kdec: Try to fix 8bps output case

Ive no test samples for which this makes a difference but it
matches the 16bit implementation.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
j2kdec/jpeg2000dec: partially merge quantization code

The quantization code needs more work, not so much work
merging but more work investigating what is correct.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>