quant: correct QP range clipping for 10-bit input depths.

Fixes a bad hash error and artifacts introduced by commit c8973aec5e70