Still trying to fix the reprojection issues.
Got it right this time. The reprojection error is almost 0. Idea, make 2 cameras, one with the camera matrix calculated by OpenCV looking at the chessboard and the other looking at the first camera.