Split residual quantization to separate luma and chroma function.
Re-add rdo control to transformskip decision.

- It got left out when rewriting the function.