internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: nerfstudio-project/nerfstudio

Issue:
The command to generate Poisson meshes is not visible in the GUI
**Describe the bug**
The command to generate Poisson meshes is not visible in the GUI (see the screenshot)

**Screenshots**
![image](https://user-images.githubusercontent.com/63703454/234820780-34e4a896-eaf5-4a98-92b0-dc1288e62f87.png)


**Additional context**
Latest version of nerfstudio, installed using `pip install -e .`



Discussion:
I got same problem with `NeRFStudio` 0.2.1

![image](https://user-images.githubusercontent.com/60387342/235317687-87bb301d-6269-43cf-a2d9-11f843362666.png)

Did you train the model with the flag `--pieline.model.predict-normals True`?
@MvWouden yes

Resolving patch:
diff --git a/nerfstudio/viewer/server/render_state_machine.py b/nerfstudio/viewer/server/render_state_machine.py
--- a/nerfstudio/viewer/server/render_state_machine.py
+++ b/nerfstudio/viewer/server/render_state_machine.py
@@ -79,6 +79,7 @@ def __init__(self, viewer: ViewerState):
         self.viewer = viewer
         self.interrupt_render_flag = False
         self.daemon = True
+        self.output_keys = {}
 
     def action(self, action: RenderAction):
         """Takes an action and updates the state machine
@@ -227,7 +228,11 @@ def _send_output_to_viewer(self, outputs: Dict[str, Any]):
         Args:
             outputs: the dictionary of outputs to choose from, from the model
         """
-        self.viewer.control_panel.update_output_options(list(outputs.keys()))
+        output_keys = set(outputs.keys())
+        if self.output_keys != output_keys:
+            self.output_keys = output_keys
+            self.viewer.viser_server.send_output_options_message(list(outputs.keys()))
+            self.viewer.control_panel.update_output_options(list(outputs.keys()))
 
         output_render = self.viewer.control_panel.output_render
         self.viewer.update_colormap_options(
diff --git a/nerfstudio/viewer/viser/message_api.py b/nerfstudio/viewer/viser/message_api.py
--- a/nerfstudio/viewer/viser/message_api.py
+++ b/nerfstudio/viewer/viser/message_api.py
@@ -485,6 +485,14 @@ def send_status_message(self, eval_res: str, step: int):
         """
         self._queue(messages.StatusMessage(eval_res=eval_res, step=step))
 
+    def send_output_options_message(self, options: List[str]):
+        """Send output options message
+
+        Args:
+            options: The list of output options
+        """
+        self._queue(messages.OutputOptionsMessage(options=options))
+
     def _add_gui_impl(
         self,
         name: str,
diff --git a/nerfstudio/viewer/viser/messages.py b/nerfstudio/viewer/viser/messages.py
--- a/nerfstudio/viewer/viser/messages.py
+++ b/nerfstudio/viewer/viser/messages.py
@@ -241,3 +241,13 @@ class TimeConditionMessage(NerfstudioMessage):
 
     time: float
     """ Time conditioning value """
+
+
+@dataclasses.dataclass
+class OutputOptionsMessage(NerfstudioMessage):
+    """Output options message which are used in the export panel.
+    TODO: remove when export panel is becomes python defined.
+    """
+
+    options: Any
+    """ List of output option strings"""


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
Opposite order of optimizer and lr_scheduler in pytorch 1.1.0+
Hi, when I am training nerf in pytorch 1.13 with cuda 11.7, one warning pops out 

> UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule.

Do you have any idea if this warning will affect the NeRF training? Thank you!



Discussion:

Resolving patch:
diff --git a/nerfstudio/engine/trainer.py b/nerfstudio/engine/trainer.py
--- a/nerfstudio/engine/trainer.py
+++ b/nerfstudio/engine/trainer.py
@@ -461,8 +461,11 @@ def train_iteration(self, step: int) -> TRAIN_INTERATION_OUTPUT:
 
             metrics_dict["Gradients/Total"] = total_grad
 
+        scale = self.grad_scaler.get_scale()
         self.grad_scaler.update()
-        self.optimizers.scheduler_step_all(step)
+        # If the gradient scaler is decreased, no optimization step is performed so we should not step the scheduler.
+        if scale <= self.grad_scaler.get_scale():
+            self.optimizers.scheduler_step_all(step)
 
         # Merging loss and metrics dict into a single output.
         return loss, loss_dict, metrics_dict


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
Pytorch 2.0 nerfacto optimizer inf check error
**Describe the bug**
Nerfacto fails under PyTorch 2.0 with the `no inf checks error`.

**Error**
```
Traceback (most recent call last):
  File "/home/kulhajon/.conda/envs/nerfstudio-torch20/bin/ns-train", line 8, in <module>
    sys.exit(entrypoint())
  File "/home/kulhajon/repos/nerfstudio/scripts/train.py", line 247, in entrypoint
    main(
  File "/home/kulhajon/repos/nerfstudio/scripts/train.py", line 233, in main
    launch(
  File "/home/kulhajon/repos/nerfstudio/scripts/train.py", line 172, in launch
    main_func(local_rank=0, world_size=world_size, config=config)
  File "/home/kulhajon/repos/nerfstudio/scripts/train.py", line 87, in train_loop
    trainer.train()
  File "/home/kulhajon/repos/nerfstudio/nerfstudio/engine/trainer.py", line 236, in train
    loss, loss_dict, metrics_dict = self.train_iteration(step)
  File "/home/kulhajon/repos/nerfstudio/nerfstudio/utils/profiler.py", line 93, in inner
    out = func(*args, **kwargs)
  File "/home/kulhajon/repos/nerfstudio/nerfstudio/engine/trainer.py", line 425, in train_iteration
    self.optimizers.optimizer_scaler_step_all(self.grad_scaler)
  File "/home/kulhajon/repos/nerfstudio/nerfstudio/engine/optimizers.py", line 130, in optimizer_scaler_step_all
    grad_scaler.step(optimizer)
  File "/home/kulhajon/.conda/envs/nerfstudio-torch20/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py", line 368, in step
    assert len(optimizer_state["found_inf_per_device"]) > 0, "No inf checks were recorded for this optimizer."
AssertionError: No inf checks were recorded for this optimizer.
```




Discussion:

Resolving patch:
diff --git a/nerfstudio/engine/optimizers.py b/nerfstudio/engine/optimizers.py
--- a/nerfstudio/engine/optimizers.py
+++ b/nerfstudio/engine/optimizers.py
@@ -127,7 +127,8 @@ def optimizer_scaler_step_all(self, grad_scaler: GradScaler) -> None:
             if max_norm is not None:
                 grad_scaler.unscale_(optimizer)
                 torch.nn.utils.clip_grad_norm_(self.parameters[param_group], max_norm)
-            grad_scaler.step(optimizer)
+            if any(any(p.grad is not None for p in g["params"]) for g in optimizer.param_groups):
+                grad_scaler.step(optimizer)
 
     def optimizer_step_all(self) -> None:
         """Run step for all optimizers."""


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
a simple mistake in ExponentialDecayScheduler
When training nerfacto with --optimizers.fields.scheduler.warmup-steps 5000, it will crash soon.

**Expected behavior**
The lr should ramps up to 1e-2 in 5000 steps.  However, the lr increase like this.
![image](https://github.com/nerfstudio-project/nerfstudio/assets/132132397/46b10c50-529c-433a-9dfb-078e2f92f714)

**Additional context**
This is caused by a typo in [ExponentialDecayScheduler](https://github.com/nerfstudio-project/nerfstudio/blob/e94d9031ab711bd755655adafba1b986e980f27b/nerfstudio/engine/schedulers.py#LL125C27-L125C27),  
where `lr = self.config.lr_pre_warmup + (1 - self.config.lr_pre_warmup) * np.sin(0.5 * np.pi * np.clip(step / self.config.warmup_steps, 0, 1))` 
should be `lr = self.config.lr_pre_warmup + (lr_init - self.config.lr_pre_warmup) * np.sin(0.5 * np.pi * np.clip(step / self.config.warmup_steps, 0, 1))`



Discussion:

Resolving patch:
diff --git a/nerfstudio/engine/schedulers.py b/nerfstudio/engine/schedulers.py
--- a/nerfstudio/engine/schedulers.py
+++ b/nerfstudio/engine/schedulers.py
@@ -122,7 +122,7 @@ def get_scheduler(self, optimizer: Optimizer, lr_init: float) -> LRScheduler:
         def func(step):
             if step < self.config.warmup_steps:
                 if self.config.ramp == "cosine":
-                    lr = self.config.lr_pre_warmup + (1 - self.config.lr_pre_warmup) * np.sin(
+                    lr = self.config.lr_pre_warmup + (lr_init - self.config.lr_pre_warmup) * np.sin(
                         0.5 * np.pi * np.clip(step / self.config.warmup_steps, 0, 1)
                     )
                 else:


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
Doc Description Wrong
Hello, 

I find python doc in [get_depth_image_from_path](https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/data/utils/data_utils.py) is wrong about the return tensor shape, 

it should be [height, width, 1] not [width, height, 1].

![图片](https://github.com/nerfstudio-project/nerfstudio/assets/20349525/e6acb5ee-4f66-44ad-b21e-1161b1df61b3)

a simple verification:

![图片](https://github.com/nerfstudio-project/nerfstudio/assets/20349525/dfd65f57-7f8b-44ab-897b-d06793c03375)




Discussion:

Resolving patch:
diff --git a/nerfstudio/data/utils/data_utils.py b/nerfstudio/data/utils/data_utils.py
--- a/nerfstudio/data/utils/data_utils.py
+++ b/nerfstudio/data/utils/data_utils.py
@@ -74,7 +74,7 @@ def get_depth_image_from_path(
         interpolation: Depth value interpolation for resizing.
 
     Returns:
-        Depth image torch tensor with shape [width, height, 1].
+        Depth image torch tensor with shape [height, width, 1].
     """
     if filepath.suffix == ".npy":
         image = np.load(filepath) * scale_factor


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
[Regression] Nerfacto-huge system memory usage went from 12GB to 175+GB (1200 images)
**Describe the bug**
As per https://github.com/nerfstudio-project/nerfstudio/pull/2003#issuecomment-1594261444, a recent change in the nerfacto-huge PR skyrocketed its memory usage.  
Nerfacto-huge's memory usage went from ~12GB on commit 374d7fe4b2f81363b3eeb4b10229aae84ab1a7b5, to 175+GB (before OOM) on commit f3ac598e27bef1fa691ea5623e75841525c30596 (or 4acc06bdb27b26bc0d16ef1fbc2ad96e47ea2265 but that one crashes, so I can't verify).
The problem also happens with lower image counts, 300 images now uses around 60GB RAM.

**To Reproduce**
Steps to reproduce the behavior:
1. Pull commit 374d7fe4b2f81363b3eeb4b10229aae84ab1a7b5;
2. Run a ns-train nerfacto-huge, take note of the memory usage;
3. Pull commit f3ac598e27bef1fa691ea5623e75841525c30596;
4. Run a ns-train nerfacto-huge, take note of the memory usage. It is several times higher now.

**Expected behavior**
Memory usage stays within sensible bounds

**Screenshots**
N/A

**Additional context**
Command tested with: `ns-train nerfacto --max-num-iterations 100000 --viewer.quit-on-train-completion True --pipeline.model.use_gradient_scaling True --pipeline.model.predict-normals True --data /home/sebastiaan/src/nerf/work/out/colmaps/VID_20230615_163207/1200 --output-dir /home/sebastiaan/src/nerf/work/out/models/VID_
20230615_163207/100000`

The commits mentioned in this issue are from before #2003 was merged, commits from the PR, making it difficult to further bisect this issue without cherry picking commits from the main branch.

Note: This applies to system memory, not GPU VRAM.


Discussion:
Any better after #2082 ?
> Any better after #2082 ?

Nope, this issue was raised only after #2082, which fixed another problem I had, did not fix this problem.
Just re-confirmed, current main branch head (7859f19f50736e477f0128a4c7b9942f41f597c0) still has the same issue.

Resolving patch:
diff --git a/nerfstudio/configs/method_configs.py b/nerfstudio/configs/method_configs.py
--- a/nerfstudio/configs/method_configs.py
+++ b/nerfstudio/configs/method_configs.py
@@ -158,7 +158,7 @@
     mixed_precision=True,
     pipeline=VanillaPipelineConfig(
         datamanager=VanillaDataManagerConfig(
-            dataparser=NerfstudioDataParserConfig(downscale_factor=1),
+            dataparser=NerfstudioDataParserConfig(),
             train_num_rays_per_batch=16384,
             eval_num_rays_per_batch=4096,
             camera_optimizer=CameraOptimizerConfig(


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
masked (mustard) dataset much slower with main than PR2025
On my machine
- using the mustard dataset with masks (see https://github.com/nerfstudio-project/nerfstudio/issues/1498#issue-1599115766 ) with PR #2025 had default nerfacto iteration times around 80 to 90 ms
- using same dataset with main (currently https://github.com/nerfstudio-project/nerfstudio/commit/d18588d5d43fafa87aa819cdb3dc5b4e09f05549 ) has default nerfacto iteration times around 400 ms
so 4-5x worse.


Discussion:
Suggested viewer crop settings for mustard:
```
Enable
Crop Min -0.15 -0.20 -0.25
Crop Max 0.15 0.20 0.33
```
https://github.com/nerfstudio-project/nerfstudio/commit/d4b04376abd46d9bddf8c299d1687177fe027951 is 80 to 90ms
https://github.com/nerfstudio-project/nerfstudio/commit/3b0f7581d8df44045f0a0822f0cbcb7e6c0223b8 is ~400ms

So looks to be a side effect of #1730 for datasets using masks.
Removing `"mask"` from https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/data/datasets/base_dataset.py#L45 restores full speed for mustard dataset.

Perhaps there should be a parameter to request masks on GPU that would do the above at runtime.
I faced similar issue. The training speed drops from 170k/s to 6k/s so more than 20 times slower. Indeed #1730 reduced GPU memory when using mask, but the speed is compromised

Resolving patch:
diff --git a/nerfstudio/data/datamanagers/base_datamanager.py b/nerfstudio/data/datamanagers/base_datamanager.py
--- a/nerfstudio/data/datamanagers/base_datamanager.py
+++ b/nerfstudio/data/datamanagers/base_datamanager.py
@@ -111,6 +111,8 @@ class DataManagerConfig(InstantiateConfig):
     """Source of data, may not be used by all models."""
     camera_optimizer: Optional[CameraOptimizerConfig] = None
     """Specifies the camera pose optimizer used during training. Helpful if poses are noisy."""
+    masks_on_gpu: Optional[bool] = None
+    """Process masks on GPU for speed at the expense of memory, if True."""
 
 
 class DataManager(nn.Module):
@@ -396,6 +398,8 @@ def __init__(
         self.train_dataset = self.create_train_dataset()
         self.eval_dataset = self.create_eval_dataset()
         self.exclude_batch_keys_from_device = self.train_dataset.exclude_batch_keys_from_device
+        if self.config.masks_on_gpu is True:
+            self.exclude_batch_keys_from_device.remove("mask")
 
         if self.train_dataparser_outputs is not None:
             cameras = self.train_dataparser_outputs.cameras


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
Make default implementation tcnn
Hi, can you make all the implementation: [tcnn, torch] arguments default to tcnn (e.g. HashMLPDensityField and nerfacto are not)? Upgraded to the latest nerfstudio and spent quite a while trying to figure out why my code was 4x slower.



Discussion:

Resolving patch:
diff --git a/nerfstudio/field_components/encodings.py b/nerfstudio/field_components/encodings.py
--- a/nerfstudio/field_components/encodings.py
+++ b/nerfstudio/field_components/encodings.py
@@ -264,7 +264,7 @@ def __init__(
         log2_hashmap_size: int = 19,
         features_per_level: int = 2,
         hash_init_scale: float = 0.001,
-        implementation: Literal["tcnn", "torch"] = "torch",
+        implementation: Literal["tcnn", "torch"] = "tcnn",
         interpolation: Optional[Literal["Nearest", "Linear", "Smoothstep"]] = None,
     ) -> None:
         super().__init__(in_dim=3)
diff --git a/nerfstudio/fields/density_fields.py b/nerfstudio/fields/density_fields.py
--- a/nerfstudio/fields/density_fields.py
+++ b/nerfstudio/fields/density_fields.py
@@ -56,7 +56,7 @@ def __init__(
         base_res: int = 16,
         log2_hashmap_size: int = 18,
         features_per_level: int = 2,
-        implementation: Literal["tcnn", "torch"] = "torch",
+        implementation: Literal["tcnn", "torch"] = "tcnn",
     ) -> None:
         super().__init__()
         self.register_buffer("aabb", aabb)


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
Fast downscale breaks raw input images
The fast-downscale path apparently relies on being able to overwrite the original file (eg. with crop etc. applied) which fails for TIFF, and unfortunately that is currently what RAW input images use for conversion.  #2295 should fix it.


Discussion:

Resolving patch:
diff --git a/nerfstudio/process_data/process_data_utils.py b/nerfstudio/process_data/process_data_utils.py
--- a/nerfstudio/process_data/process_data_utils.py
+++ b/nerfstudio/process_data/process_data_utils.py
@@ -36,7 +36,7 @@
 """Lowercase suffixes to treat as raw image."""
 ALLOWED_RAW_EXTS = [".cr2"]
 """Suffix to use for converted images from raw."""
-RAW_CONVERTED_SUFFIX = ".tiff"
+RAW_CONVERTED_SUFFIX = ".jpg"
 
 
 class CameraModel(Enum):
@@ -244,7 +244,7 @@ def copy_images_list(
             CONSOLE.log(f"Copying image {idx + 1} of {len(image_paths)}...")
         copied_image_path = image_dir / f"frame_{idx + 1:05d}{image_path.suffix}"
         try:
-            # if CR2 raw, we want to read raw and write TIFF, and change the file suffix for downstream processing
+            # if CR2 raw, we want to read raw and write RAW_CONVERTED_SUFFIX, and change the file suffix for downstream processing
             if image_path.suffix.lower() in ALLOWED_RAW_EXTS:
                 copied_image_path = image_dir / f"frame_{idx + 1:05d}{RAW_CONVERTED_SUFFIX}"
                 with rawpy.imread(str(image_path)) as raw:


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
Dycheck Dataparser rescale shape error
**Describe the bug**
When using downscaling with the dycheck dataparser, the automatic downscaling has a shape error between `depth_raw` and `cosa` [here](https://github.com/nerfstudio-project/nerfstudio/blob/9528a3f717fe28a6c6500199b5b37d4f2c6e10ca/nerfstudio/data/dataparsers/dycheck_dataparser.py#L176). Could possibly only exist for record3d-captured scenes?

**To Reproduce**
Steps to reproduce the behavior:
Use dycheck dataparser  with downscale_factor > 1 for a scaling that does not already exist. I used the backpack scene from [here](https://drive.google.com/drive/folders/1cBw3CUKu2sWQfc_1LbFZGbpdQyTFzDEX) with a downscale factor of 2 and received a dimension mismatch error:
`ValueError: operands could not be broadcast together with shapes (480,360,1) (480,360)`

**Expected behavior**
There should be no shape error.

**Additional context**
Adding an extra dimension to `depth_raw` if it is not 3-dimensional fixes the issue. Does anyone know what the extra dimension is for?



Discussion:
[This line](https://github.com/nerfstudio-project/nerfstudio/blob/9528a3f717fe28a6c6500199b5b37d4f2c6e10ca/nerfstudio/data/dataparsers/dycheck_dataparser.py#L325C5-L325C5) should also pass in width first then height.

Resolving patch:
diff --git a/nerfstudio/data/dataparsers/dycheck_dataparser.py b/nerfstudio/data/dataparsers/dycheck_dataparser.py
--- a/nerfstudio/data/dataparsers/dycheck_dataparser.py
+++ b/nerfstudio/data/dataparsers/dycheck_dataparser.py
@@ -173,7 +173,10 @@ def _rescale_depth(depth_raw: np.ndarray, cam: Dict) -> np.ndarray:
     viewdirs /= np.linalg.norm(viewdirs, axis=-1, keepdims=True)
     viewdirs = viewdirs.reshape((*batch_shape, 3))
     cosa = viewdirs @ (cam["camera_to_worlds"][:, 2])
-    depth = depth_raw / cosa[..., None]
+    if depth_raw.ndim == cosa.ndim:
+        depth = depth_raw[..., None] / cosa[..., None]
+    else:
+        depth = depth_raw / cosa[..., None]
     return depth
 
 


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
Add flag to disable apperance embedding



Discussion:
Sorry, I had auto-merge turned on :( I would have updated it.

Resolving patch:
diff --git a/nerfstudio/fields/nerfacto_field.py b/nerfstudio/fields/nerfacto_field.py
--- a/nerfstudio/fields/nerfacto_field.py
+++ b/nerfstudio/fields/nerfacto_field.py
@@ -112,6 +112,8 @@ def __init__(
         self.appearance_embedding_dim = appearance_embedding_dim
         if self.appearance_embedding_dim > 0:
             self.embedding_appearance = Embedding(self.num_images, self.appearance_embedding_dim)
+        else:
+            self.embedding_appearance = None
         self.use_average_appearance_embedding = use_average_appearance_embedding
         self.use_transient_embedding = use_transient_embedding
         self.use_semantics = use_semantics
@@ -241,7 +243,7 @@ def get_outputs(
 
         # appearance
         embedded_appearance = None
-        if self.appearance_embedding_dim > 0:
+        if self.embedding_appearance is not None:
             if self.training:
                 embedded_appearance = self.embedding_appearance(camera_indices)
             else:
diff --git a/nerfstudio/fields/nerfw_field.py b/nerfstudio/fields/nerfw_field.py
--- a/nerfstudio/fields/nerfw_field.py
+++ b/nerfstudio/fields/nerfw_field.py
@@ -77,6 +77,8 @@ def __init__(
 
         if self.appearance_embedding_dim > 0:
             self.embedding_appearance = Embedding(self.num_images, self.appearance_embedding_dim)
+        else:
+            self.embedding_appearance = None
         self.embedding_transient = Embedding(self.num_images, self.transient_embedding_dim)
 
         self.mlp_base = MLP(
@@ -97,7 +99,7 @@ def __init__(
         self.mlp_head = MLP(
             in_dim=self.mlp_base.get_out_dim()
             + self.direction_encoding.get_out_dim()
-            + (self.embedding_appearance.get_out_dim() if self.appearance_embedding_dim > 0 else 0),
+            + (self.embedding_appearance.get_out_dim() if self.embedding_appearance is not None else 0),
             num_layers=head_mlp_num_layers,
             layer_width=head_mlp_layer_width,
             out_activation=nn.ReLU(),
@@ -136,7 +138,7 @@ def get_outputs(
             raise AttributeError("Camera indices are not provided.")
         camera_indices = ray_samples.camera_indices.squeeze().to(ray_samples.frustums.origins.device)
         mlp_in = [density_embedding, encoded_dir]
-        if self.appearance_embedding_dim > 0:
+        if self.embedding_appearance is not None:
             embedded_appearance = self.embedding_appearance(camera_indices)
             mlp_in.append(embedded_appearance)
         mlp_head_out = self.mlp_head(torch.cat(mlp_in, dim=-1))


Test patch: