OMG IT WORKS! Camera intrinsics were not considering the scale factor!