Fix automatic matrix selection for YUV -> RGB conversion with high bit-depth

Thanks to tophf