So this here I do not understand very well, maybe you can explain to me. Normally one would need perspective projection equations to go back and forth (http://cseweb.ucsd.edu/classes/fa12/cse252A-a/lec4.pdf). CameraInfo holds the data for that.