Repository: nerfstudio-project/nerfstudio

Issue:
Camera Optimizer has different pose adjustments when applied to Raybundle and Camera.
Hi developers,

I found a pretty tricky computation inconsistency issue on `camera_optimizer`'s `apply_to_*` functions:

https://github.com/nerfstudio-project/nerfstudio/blob/babf57718c75e7d04f1b977f035db67c02a9c20a/nerfstudio/cameras/camera_optimizers.py#L148-L168

The rotation and translation shown in the above two functions are completely different. 
Assume the original pose is `[R, t]`, and pose adjustment in `CameraOptimizer` is `[dR, dt]`

* `apply_to_raybundle`: simply translates the `origins` (`t+dt`), and rotates the ray `directions` (`dR @ R @ ray_vec`) 

* `apply_to_camera`: new camera is `[R@dR, t+R@dt]`, when using this new camera to generate rays `origins` will be `t+R@dt`, `directions` transformation will be `R @ dR @ ray_vec`.

I initially wanted to evaluate the quality of camera optimization of a trained NeRF model by applying this `apply_to_camera` function but found such a computation mismatch. I guess these functions would also be confusing for other users (for sure 🤣 

It would be better in the future version to add some caution notes and an option to switch to different forms of pose adjustments.
For example,
```python
# raybundle_cam: bool
if raybundle_cam:
    R = torch.bmm(adj[:, :3, :3], camera.camera_to_worlds[:, :3, :3])
    t = adj[:, :3, 3:] + camera.camera_to_worlds[:, :3, 3:]
    camera.camera_to_worlds = torch.cat([R, t], dim=-1)
else:
    adj = torch.cat([adj, torch.Tensor([0, 0, 0, 1])[None, None].to(adj)], dim=1)
    camera.camera_to_worlds = torch.bmm(camera.camera_to_worlds, adj)
``` 


Discussion:
Thanks for sharing the knowledge. I think the correction transform `(dR, dt)` should be multiply to `T_camera_to_world` transform from the *left*, which is `(dR @ R, dR t + dt )`. This is neither of what was implemented in `apply_to_raybundle` and `apply_to_camera`.

Think about the case when we have pose estimate of a sensor rig that contains two cameras. The two cameras have their own pose estimates subject to their fixed extrinsics `T_cam1_to_cam2`, such that `T_cam2_to_world * T_cam1_to_cam2 = T_cam1_to_world`. A shared correction transform to the pose estimate of this sensor rig must be multiplied from the *left* to each of its cameras. 


@oseiskar Just want to learn your thoughts on this?
@jb-ye I did not actually check this in detail in https://github.com/nerfstudio-project/nerfstudio/pull/2891/commits/e8e05ff02f1c604124a0373afcca7691f2b88546, and just kept the multiplication order from the [original 3DGS PR](https://github.com/nerfstudio-project/nerfstudio/pull/2521/files#diff-574309e78cc47396f2fd1e783b1695cddbd093a62333ccdfa63bfd659b2a79dfR158)

I agree with you that it would be better to multiply camera-to-world from the left since that would also work for multi-camera rigs: If you have a single adjusment matrix for all cameras in the rig, the multiplying with that from the left would work for adjusting the pose of the rig correctly.

However, as long as there are no such rig constraints but all cameras are optimized independently, then all of the versions discussed in this thread are OK. They are just different ways to parametrize adjustments to a SE(3) pose.

The main problem in the current version seems to be that the NeRF (ray bundle) and 3DGS use different interpretations for the parameters, which is not nice since it breaks in cases like @nexuslrf 's test.

It might be safest to fix the 3DGS version to match the NeRF one, since at least for 3DGS, all versions should be fine, but for NeRFs, there could be more performance implications (just guessing here, though). If support for multi-camera setups is added, then both version should probably be fixed to use the multiply-from-the-left form.
Thanks everyone for the discussion! Some quick thoughts.

The reason for the original `apply_to_raybundle()` implementation was likely runtime: the reconstruction quality for all of these implementations (left-multiply, right-multiply, the current behavior) should all be basically the same, but `(dR @ R, t + dt)` (what's implemented) has one less matrix multiplication than `(dR @ R, dR @ t + dt)`.  I recall some tests being run and the overhead being nontrivial.

Assuming this is true, one suggestion is to keep the current behavior for NeRFs, but match the splatfacto behavior for consistency and document it better: it's a separate rotation around the camera frame origin + translation in world coordinates.[^1]

[^1]: **Edit:** on a second read this is exactly what @oseiskar suggested in the comment above. sorry for not reading more thoroughly!

> Think about the case when we have pose estimate of a sensor rig that contains two cameras. The two cameras have their own pose estimates subject to their fixed extrinsics T_cam1_to_cam2, such that T_cam2_to_world * T_cam1_to_cam2 = T_cam1_to_world. A shared correction transform to the pose estimate of this sensor rig must be multiplied from the left to each of its cameras.

If we want to switch to a single rigid transform, between left- vs right- multiplied my very weak personal bias is actually for right-multiplied, mostly because it seems like the standard in packages like Ceres and gtsam. I don't find the multi-sensor rig case super convincing, since (a) you have to write custom code anyways and (b) right-multiplication is more convenient for the complementary case where you have good global rig poses but don't trust per-sensor calibrations relative to the rig.
> If we want to switch to a single rigid transform, between left- vs right- multiplied my very weak personal bias is actually for right-multiplied, mostly because it seems like the standard in packages like Ceres and gtsam. I don't find the multi-sensor rig case super convincing, since (a) you have to write custom code anyways and (b) right-multiplication is more convenient for the complementary case where you have good global rig poses but don't trust per-sensor calibrations relative to the rig.

I would like to add that being able to handle all relevant cases with a single rigid transform, which is either left or right multiplication is not a very sustainable goal. In a system that can deal with cases such as multi-camera rigs and possibly extrinsic calibration correction, you will anyway end up simultaneously transforming the poses from _both_ left (tune the pose of the rig) and right (fixed or adjustable transform within the local rig coordinates) and some of these transforms will be shared between multiple cameras.

... and, without multi-camera support, it does not really matter so I also support this:

> Assuming this is true, one suggestion is to keep the current behavior for NeRFs, but match the splatfacto behavior for consistency and document it better: it's a separate rotation around the camera frame origin + translation in world coordinates.

Resolving patch:
diff --git a/nerfstudio/cameras/camera_optimizers.py b/nerfstudio/cameras/camera_optimizers.py
--- a/nerfstudio/cameras/camera_optimizers.py
+++ b/nerfstudio/cameras/camera_optimizers.py
@@ -157,15 +157,24 @@ def apply_to_camera(self, camera: Cameras) -> torch.Tensor:
         if self.config.mode == "off":
             return camera.camera_to_worlds
 
-        assert camera.metadata is not None, "Must provide id of camera in its metadata"
-        if "cam_idx" not in camera.metadata:
-            # Evalutaion cams?
+        if camera.metadata is None or "cam_idx" not in camera.metadata:
+            # Viser cameras
             return camera.camera_to_worlds
 
         camera_idx = camera.metadata["cam_idx"]
-        adj = self(torch.tensor([camera_idx], dtype=torch.long, device=camera.device))  # type: ignore
-        adj = torch.cat([adj, torch.Tensor([0, 0, 0, 1])[None, None].to(adj)], dim=1)
-        return torch.bmm(camera.camera_to_worlds, adj)
+        adj = self(torch.tensor([camera_idx], dtype=torch.long)).to(camera.device)  # type: ignore
+
+        return torch.cat(
+            [
+                # Apply rotation to directions in world coordinates, without touching the origin.
+                # Equivalent to: directions -> correction[:3,:3] @ directions
+                torch.bmm(adj[..., :3, :3], camera.camera_to_worlds[..., :3, :3]),
+                # Apply translation in world coordinate, independently of rotation.
+                # Equivalent to: origins -> origins + correction[:3,3]
+                camera.camera_to_worlds[..., :3, 3:] + adj[..., :3, 3:],
+            ],
+            dim=-1,
+        )
 
     def get_loss_dict(self, loss_dict: dict) -> None:
         """Add regularization"""
diff --git a/nerfstudio/exporter/exporter_utils.py b/nerfstudio/exporter/exporter_utils.py
--- a/nerfstudio/exporter/exporter_utils.py
+++ b/nerfstudio/exporter/exporter_utils.py
@@ -30,6 +30,7 @@
 from rich.progress import BarColumn, Progress, TaskProgressColumn, TextColumn, TimeRemainingColumn
 from torch import Tensor
 
+from nerfstudio.cameras.camera_optimizers import CameraOptimizer
 from nerfstudio.cameras.cameras import Cameras
 from nerfstudio.cameras.rays import RayBundle
 from nerfstudio.data.datasets.base_dataset import InputDataset
@@ -283,11 +284,14 @@ def render_trajectory(
     return images, depths
 
 
-def collect_camera_poses_for_dataset(dataset: Optional[InputDataset]) -> List[Dict[str, Any]]:
+def collect_camera_poses_for_dataset(
+    dataset: Optional[InputDataset], camera_optimizer: Optional[CameraOptimizer] = None
+) -> List[Dict[str, Any]]:
     """Collects rescaled, translated and optimised camera poses for a dataset.
 
     Args:
         dataset: Dataset to collect camera poses for.
+        camera_optimizer: Camera optimizer that has been used for adjusting the poses
 
     Returns:
         List of dicts containing camera poses.
@@ -304,7 +308,15 @@ def collect_camera_poses_for_dataset(dataset: Optional[InputDataset]) -> List[Di
     # new cameras are in cameras, whereas image paths are stored in a private member of the dataset
     for idx in range(len(cameras)):
         image_filename = image_filenames[idx]
-        transform = cameras.camera_to_worlds[idx].tolist()
+        if camera_optimizer is None:
+            transform = cameras.camera_to_worlds[idx].tolist()
+        else:
+            # print('exporting optimized camera pose for camera %d' % idx)
+            camera = cameras[idx : idx + 1]
+            assert camera.metadata is not None
+            camera.metadata["cam_idx"] = idx
+            transform = camera_optimizer.apply_to_camera(camera).tolist()[0]
+
         frames.append(
             {
                 "file_path": str(image_filename),
@@ -331,7 +343,12 @@ def collect_camera_poses(pipeline: VanillaPipeline) -> Tuple[List[Dict[str, Any]
     eval_dataset = pipeline.datamanager.eval_dataset
     assert isinstance(eval_dataset, InputDataset)
 
-    train_frames = collect_camera_poses_for_dataset(train_dataset)
+    camera_optimizer = None
+    if hasattr(pipeline.model, "camera_optimizer"):
+        camera_optimizer = pipeline.model.camera_optimizer
+
+    train_frames = collect_camera_poses_for_dataset(train_dataset, camera_optimizer)
+    # Note: returning original poses, even if --eval-mode=all
     eval_frames = collect_camera_poses_for_dataset(eval_dataset)
 
     return train_frames, eval_frames
diff --git a/nerfstudio/models/splatfacto.py b/nerfstudio/models/splatfacto.py
--- a/nerfstudio/models/splatfacto.py
+++ b/nerfstudio/models/splatfacto.py
@@ -672,10 +672,10 @@ def get_outputs(self, camera: Cameras) -> Dict[str, Union[torch.Tensor, List]]:
             return {}
         assert camera.shape[0] == 1, "Only one camera at a time"
 
+        optimized_camera_to_world = self.camera_optimizer.apply_to_camera(camera)[0, ...]
+
         # get the background color
         if self.training:
-            optimized_camera_to_world = self.camera_optimizer.apply_to_camera(camera)[0, ...]
-
             if self.config.background_color == "random":
                 background = torch.rand(3, device=self.device)
             elif self.config.background_color == "white":
@@ -685,8 +685,6 @@ def get_outputs(self, camera: Cameras) -> Dict[str, Union[torch.Tensor, List]]:
             else:
                 background = self.background_color.to(self.device)
         else:
-            optimized_camera_to_world = camera.camera_to_worlds[0, ...]
-
             if renderers.BACKGROUND_COLOR_OVERRIDE is not None:
                 background = renderers.BACKGROUND_COLOR_OVERRIDE.to(self.device)
             else:


Test patch: