Still allow calibration w/o IMU constraints between poses