Moving to a better math library, adding move and rotate commands

This is the math error in your camera code.  You shouldn't take values from OVR, invert them, store them inverted, and then invert them again here.  It's very hard to understand and screams "you're applying your matrices backwards".