Move up, right and forward vectors from Camera to Entity. Note there is overlap between these and rotation matrix - will have to think carefully about how these are synced (quaternions?).