internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: nerfstudio-project/nerfstudio

Issue:
Bridge server attempts to send over eval images
If the occupancy output is set, the server will crash after awhile. It does render some views correctly.

<img width="1141" alt="image" src="https://user-images.githubusercontent.com/3310961/179328841-f5b32124-97ba-41cf-a41c-b54d6de3a36f.png">



Discussion:
Was this "depth occupancy" with instant ngp? Also approx how many iterations until you encountered this?
It was any of the non-rgb outputs. It would happen after ~1k iters while navigating in the viewer. I tried to replicate last night but wasn't able to. Maybe it is somehow fixed, but lets keep this issue open for a bit to see if anyone else runs into it. Im worried that it is a race condition bug given that it sent a 800x800x1 shaped image which would never be requested by the viewer. The only time it would render that resolution would be for logging eval images to tb.
This seems to still be an issue. It occurs more often for me when running the most recent ngp code.
I've been able to replicate more consistently now (using a smaller batch size to slow down training). An eval image is sent over to the viewer after the eval step.

Resolving patch:
diff --git a/nerfactory/graphs/base.py b/nerfactory/graphs/base.py
--- a/nerfactory/graphs/base.py
+++ b/nerfactory/graphs/base.py
@@ -208,7 +208,6 @@ def get_outputs_for_camera_ray_bundle(self, camera_ray_bundle: RayBundle):
                 outputs_lists[output_name].append(output)
         for output_name, outputs_list in outputs_lists.items():
             outputs[output_name] = torch.cat(outputs_list).view(image_height, image_width, -1)
-        self.vis_outputs = outputs
         return outputs
 
     def get_outputs_for_camera(self, camera: Camera):
diff --git a/nerfactory/viewer/server/viewer_utils.py b/nerfactory/viewer/server/viewer_utils.py
--- a/nerfactory/viewer/server/viewer_utils.py
+++ b/nerfactory/viewer/server/viewer_utils.py
@@ -78,6 +78,7 @@ def __init__(self, state: "VisualizerState", graph: Graph, camera_ray_bundle: Ra
         self.graph = graph
         self.camera_ray_bundle = camera_ray_bundle
         self.exc = None
+        self.vis_outputs = None
 
     def run(self):
         """run function that renders out images given the current graph and ray bundles.
@@ -93,6 +94,7 @@ def run(self):
 
         if outputs:
             self.graph.process_outputs_as_images(outputs)
+            self.vis_outputs = outputs
 
         self.state.check_done_render = True
         self.state.check_interrupt_vis = False
@@ -356,7 +358,7 @@ def _render_image_in_viewer(self, graph: Graph) -> None:
             pass
 
         graph.train()
-        outputs = graph.vis_outputs
+        outputs = render_thread.vis_outputs
         if outputs is not None:
             self._send_output_to_viewer(outputs)
 


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
Spurious lines in metric logger
Some extra print statements get caught in the logger.
<img width="600" alt="image" src="https://user-images.githubusercontent.com/3310961/180073069-b5cdfddf-f011-4fdc-87d6-d5d9471fc92b.png">



Discussion:

Resolving patch:
diff --git a/nerfactory/utils/writer.py b/nerfactory/utils/writer.py
--- a/nerfactory/utils/writer.py
+++ b/nerfactory/utils/writer.py
@@ -322,6 +322,7 @@ def __init__(self, log_dir: str, stats_to_track: ListConfig, max_log_size: int =
         self.max_log_size = max_log_size
         self.keys = set()
         self.past_mssgs = ["", ""]
+        self.has_printed = False
 
     def write_image(self, name: str, image: TensorType["H", "W", "C"], step: int) -> None:
         if name in self.stats_to_track:
@@ -357,9 +358,10 @@ def _update_header(self, latest_map, new_key):
                     mssg += f"{name:<20} "
             self.past_mssgs[0] = mssg
             self.past_mssgs[1] = "-" * len(mssg)
-            if full_log_cond:
+            if full_log_cond or not self.has_printed:
                 print(mssg)
                 print("-" * len(mssg))
+                self.has_printed = True
 
     def _print_stats(self, latest_map, padding=" "):
         """helper to print out the stats in a readable format"""


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
Depth normalization inconsistent for packed vs. nonpacked samples
**Describe the bug**
When the raymarching samples are packed, the depth is calculated according to `sum_i w_i t_i`: https://github.com/nerfstudio-project/nerfstudio/blob/863fc77ab5f247ff3ce3c80f192173063529b036/nerfstudio/model_components/renderers.py#L236

When the raymarching samples are not packed, the depth is calculated with a normalization factor dividing by the total accumulation, `(sum_i w_i t_i) /  (sum_i w_i)`: https://github.com/nerfstudio-project/nerfstudio/blob/863fc77ab5f247ff3ce3c80f192173063529b036/nerfstudio/model_components/renderers.py#L238

**To Reproduce**
N/A

**Expected behavior**
For consistency, the calculation for packed samples should also divide by the total accumulation.

**Screenshots**
N/A

**Additional context**
If this is desired, I can implement the change.



Discussion:
Great catch! If you make a PR I can accept it!

Resolving patch:
diff --git a/nerfstudio/model_components/renderers.py b/nerfstudio/model_components/renderers.py
--- a/nerfstudio/model_components/renderers.py
+++ b/nerfstudio/model_components/renderers.py
@@ -234,6 +234,8 @@ def forward(
             if ray_indices is not None and num_rays is not None:
                 # Necessary for packed samples from volumetric ray sampler
                 depth = nerfacc.accumulate_along_rays(weights, ray_indices, steps, num_rays)
+                accumulation = nerfacc.accumulate_along_rays(weights, ray_indices, None, num_rays)
+                depth = depth / (accumulation + eps)
             else:
                 depth = torch.sum(weights * steps, dim=-2) / (torch.sum(weights, -2) + eps)
 


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
Relative data paths result in hidden output directories
Running `ns-train` with a relative path to the data results in an output directory that is hidden because the experiment name includes components of the relative path. For example, running: `ns-train nerfacto --data ../nerf_datasets/tum_fr1desk` results in an output directory `outputs/..-nerf_datasets-tum-fr1desk`.

**To Reproduce**
Steps to reproduce the behavior:
1. Run `ns-train nerfacto --data ../[path to data]`

**Expected behavior**
The expected behavior should probably be to remove any `"../"` from the data path when generating the experiment name.



Discussion:

Resolving patch:
diff --git a/nerfstudio/configs/base_config.py b/nerfstudio/configs/base_config.py
--- a/nerfstudio/configs/base_config.py
+++ b/nerfstudio/configs/base_config.py
@@ -241,7 +241,7 @@ def set_timestamp(self) -> None:
     def set_experiment_name(self) -> None:
         """Dynamically set the experiment name"""
         if self.experiment_name is None:
-            self.experiment_name = str(self.pipeline.datamanager.dataparser.data).replace("/", "-")
+            self.experiment_name = str(self.pipeline.datamanager.dataparser.data).replace("../", "").replace("/", "-")
 
     def get_base_dir(self) -> Path:
         """Retrieve the base directory to set relative paths"""


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
tanh instead of tan bug
Hi,
please change tanh (hyperbolic tan) to tan 

https://github.com/nerfstudio-project/nerfstudio/blob/1a24f3e58c544bc0211563e770d425426284256c/nerfstudio/data/dataparsers/instant_ngp_dataparser.py#L133

thanks


Discussion:

Resolving patch:
diff --git a/nerfstudio/data/dataparsers/instant_ngp_dataparser.py b/nerfstudio/data/dataparsers/instant_ngp_dataparser.py
--- a/nerfstudio/data/dataparsers/instant_ngp_dataparser.py
+++ b/nerfstudio/data/dataparsers/instant_ngp_dataparser.py
@@ -130,7 +130,7 @@ def get_focal_lengths(cls, meta: Dict) -> Tuple[float, float]:
         fl_x, fl_y = 0, 0
 
         def fov_to_focal_length(rad, res):
-            return 0.5 * res / np.tanh(0.5 * rad)
+            return 0.5 * res / np.tan(0.5 * rad)
 
         if "fl_x" in meta:
             fl_x = meta["fl_x"]


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
Render fails silently traj="interp" mode (with custom data)
**Describe the bug**
When using ns-render (in google colab) and with custom images, the render "finishes" (cell executes successfully without any error), however, there's no output. The same command works well with traj="spiral"

**To Reproduce**
Executing the following command fails to output any video/rendering:
`!ns-render --load-config $config_filename --traj=interp --output-path renders/output.mp4`

However, changing the trajectory mode to spiral produces a correct video:
`!ns-render --load-config $config_filename --traj=spiral --output-path outputs/renders/output.mp4`

**Additional context**
I've checked the documentation for some pre-requisites to run the interpolation mode, but I didn't find anything.

Thanks!



Discussion:
I just checked in https://github.com/nerfstudio-project/nerfstudio/blob/main/scripts/render.py (line 140) that the method is not implemented. However, the error does not get propagated in colab.
Yea, colab unfortunately fails to propagate the logs. I'm going to just remove this option from the script to avoid this confusion in the future.

Resolving patch:
diff --git a/scripts/render.py b/scripts/render.py
--- a/scripts/render.py
+++ b/scripts/render.py
@@ -103,7 +103,7 @@ class RenderTrajectory:
     # Name of the renderer output to use. rgb, depth, etc.
     rendered_output_name: str = "rgb"
     #  Trajectory to render.
-    traj: Literal["spiral", "interp", "filename"] = "spiral"
+    traj: Literal["spiral", "filename"] = "spiral"
     # Scaling factor to apply to the camera image resolution.
     downscale_factor: int = 1
     # Filename of the camera path to render.
@@ -133,11 +133,6 @@ def main(self) -> None:
             camera_start = pipeline.datamanager.eval_dataloader.get_camera(image_idx=0)
             # TODO(ethan): pass in the up direction of the camera
             camera_path = get_spiral_path(camera_start, steps=30, radius=0.1)
-        elif self.traj == "interp":
-            # cameras_a = pipeline.datamanager.eval_dataloader.get_camera(image_idx=0)
-            # cameras_b = pipeline.datamanager.eval_dataloader.get_camera(image_idx=10)
-            # camera_path = get_interpolated_camera_path(cameras, steps=30)
-            raise NotImplementedError("Interpolated camera path not implemented.")
         elif self.traj == "filename":
             with open(self.camera_path_filename, "r", encoding="utf-8") as f:
                 camera_path = json.load(f)


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
`ns-train semantic-nerfw` fails looking for images_4 png instead of jpg
**Describe the bug**
`ns-train semantic-nerfw` fails looking for image png instead of jpg

**To Reproduce**
Steps to reproduce the behavior:
1. Install nerfstudio.  (0.1.10, 0.1.11, latest main)
2. `ns-download-data friends` complains, so you must download from the GDrive link and extract into data/friends manually
3. `ns-train semantic-nerfw`
4. See error about PIL/Image.py -- but notice that it is looking for image png, rather than jpg which is present and specified in cameras.json

**Expected behavior**
Training should not fail.
`ns-train semantic-nerfw` fails looking for images_4 png instead of jpg
**Describe the bug**
`ns-train semantic-nerfw` fails looking for image png instead of jpg

**To Reproduce**
Steps to reproduce the behavior:
1. Install nerfstudio.  (0.1.10, 0.1.11, latest main)
2. `ns-download-data friends` complains, so you must download from the GDrive link and extract into data/friends manually
3. `ns-train semantic-nerfw`
4. See error about PIL/Image.py -- but notice that it is looking for image png, rather than jpg which is present and specified in cameras.json

**Expected behavior**
Training should not fail.


Discussion:

Resolving patch:
diff --git a/nerfstudio/data/dataparsers/friends_dataparser.py b/nerfstudio/data/dataparsers/friends_dataparser.py
--- a/nerfstudio/data/dataparsers/friends_dataparser.py
+++ b/nerfstudio/data/dataparsers/friends_dataparser.py
@@ -116,12 +116,11 @@ def _generate_dataparser_outputs(self, split="train"):  # pylint: disable=unused
 
         # --- semantics ---
         if self.config.include_semantics:
+            empty_path = Path()
+            replace_this_path = str(empty_path / images_folder / empty_path)
+            with_this_path = str(empty_path / segmentations_folder / "thing" / empty_path)
             filenames = [
-                Path(
-                    str(image_filename)
-                    .replace(f"/{images_folder}/", f"/{segmentations_folder}/thing/")
-                    .replace(".jpg", ".png")
-                )
+                Path(str(image_filename).replace(replace_this_path, with_this_path).replace(".jpg", ".png"))
                 for image_filename in image_filenames
             ]
             panoptic_classes = load_from_json(self.config.data / "panoptic_classes.json")


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
Training crashes if all cameras are deleted while in "Render View"
Steps to replicate:
1. Start train and open viewer
2. Add camera to scene
3. Switch to "RENDER VIEW"
4. Delete camera

Resulting error:
```
  File "/projects/pyrad/scripts/train.py", line 234, in main
    launch(
  File "/projects/pyrad/scripts/train.py", line 173, in launch
    main_func(local_rank=0, world_size=world_size, config=config)
  File "/projects/pyrad/scripts/train.py", line 88, in train_loop
    trainer.train()
  File "/projects/pyrad/nerfstudio/engine/trainer.py", line 166, in train
    self._update_viewer_state(step)
  File "/projects/pyrad/nerfstudio/utils/decorators.py", line 58, in wrapper
    ret = func(self, *args, **kwargs)
  File "/projects/pyrad/nerfstudio/engine/trainer.py", line 229, in _update_viewer_state
    self.viewer_state.update_scene(self, step, self.pipeline.model, num_rays_per_batch)
  File "/projects/pyrad/nerfstudio/utils/decorators.py", line 82, in wrapper
    ret = func(*args, **kwargs)
  File "/projects/pyrad/nerfstudio/viewer/server/viewer_utils.py", line 398, in update_scene
    self._render_image_in_viewer(camera_object, graph, is_training)
  File "/projects/pyrad/nerfstudio/utils/decorators.py", line 82, in wrapper
    ret = func(*args, **kwargs)
  File "/projects/pyrad/nerfstudio/utils/profiler.py", line 43, in wrapper
    ret = func(*args, **kwargs)
  File "/projects/pyrad/nerfstudio/viewer/server/viewer_utils.py", line 745, in _render_image_in_viewer
    intrinsics_matrix, camera_to_world_h = get_intrinsics_matrix_and_camera_to_world_h(
  File "/projects/pyrad/nerfstudio/viewer/server/utils.py", line 77, in get_intrinsics_matrix_and_camera_to_world_h
    focal_length = three_js_perspective_camera_focal_length(fov, image_height)
  File "/projects/pyrad/nerfstudio/viewer/server/utils.py", line 58, in three_js_perspective_camera_focal_length
    focal_length = pp_h / np.tan(fov * (np.pi / 180.0) / 2.0)
TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'
```


Discussion:

Resolving patch:
diff --git a/nerfstudio/viewer/server/utils.py b/nerfstudio/viewer/server/utils.py
--- a/nerfstudio/viewer/server/utils.py
+++ b/nerfstudio/viewer/server/utils.py
@@ -54,6 +54,9 @@ def three_js_perspective_camera_focal_length(fov: float, image_height: int):
         fov: the field of view of the camera in degrees.
         image_height: the height of the image in pixels.
     """
+    if fov is None:
+        print("Warning: fov is None, using default value")
+        return 50
     pp_h = image_height / 2.0
     focal_length = pp_h / np.tan(fov * (np.pi / 180.0) / 2.0)
     return focal_length


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
Include "--vis viewer" in sample "ns-train" command on the README
**Describe the bug**

On the [README](https://github.com/nerfstudio-project/nerfstudio), it implies the `ns-train` command will launch the web viewer by default. However when I tried it today it just launched WanB. I believe this is due to the `--vis` option being set to `wandb` by default, not `viewer` (although in the future, it'd be nice to be able to use both simultaneously.

Anyway, I got confused and I figured it might be worth updating the README to include the `--vis viewer` arg, so that the user can see the viewer while training runs. 

**To Reproduce**
Steps to reproduce the behavior: just following the first-time setup instructions on the README:
```bash
conda create --name nerfstudio -y python=3.8
conda activate nerfstudio
python -m pip install --upgrade pip
conda create --name nerfstudio -y python=3.8
conda activate nerfstudio
python -m pip install --upgrade pip
pip install nerfstudio
# I have a custom dataset, and I trained using TensoRF
ns-train tensorf --data data/engine_6_ds/
```

**Expected behavior**
When I go to https://viewer.nerf.studio/?websocket_url=ws://localhost:7007 after running `ns-train nerfacto --data data/nerfstudio/poster`, it should show the viewer is connected. 

**Screenshots**

<img width="1246" alt="Screen Shot 2022-12-31 at 7 51 34 PM" src="https://user-images.githubusercontent.com/35160126/210158399-bfd66d57-0dd9-413f-8542-a2bab66e5aca.png">


**Additional context**
- I just tried `nerfstudio` for the first time today. So I'm not if this could just be an issue with the latest version (I installed using `pip`).
- this was tested on Google Chrome

Thanks for creating this project! It is so helpful to have all these tools in one place.



Discussion:
I was having this issue in brave browser. it is working in chrome.
This looks like a tensorf-specific default config value.

cc @terrancewang does it make sense to set the default vis method for tensorf to `viewer`? Is it fast enough?
Oh yeah looks like the default for tensorf is set to wandb right now. tensorf is fast enough to work with the viewer on my machine and should be fine in general. I'll set it to the viewer, thanks for pointing this out!

Resolving patch:
diff --git a/nerfstudio/configs/method_configs.py b/nerfstudio/configs/method_configs.py
--- a/nerfstudio/configs/method_configs.py
+++ b/nerfstudio/configs/method_configs.py
@@ -248,6 +248,9 @@
 
 method_configs["tensorf"] = TrainerConfig(
     method_name="tensorf",
+    steps_per_eval_batch=500,
+    steps_per_save=2000,
+    max_num_iterations=30000,
     mixed_precision=False,
     pipeline=VanillaPipelineConfig(
         datamanager=VanillaDataManagerConfig(
@@ -265,6 +268,8 @@
             "scheduler": SchedulerConfig(lr_final=0.002, max_steps=30000),
         },
     },
+    viewer=ViewerConfig(num_rays_per_chunk=1 << 15),
+    vis="viewer",
 )
 
 method_configs["dnerf"] = TrainerConfig(


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
Error while doing ns-export poisson for depth-nerfacto trained model on polycam data
Hi all,

I have been trying to do ns-export poisson on depth-nerfacto trained model on polycam data and get the following error: 

In depth_nerfacto.py line 73 in get_outputs


outputs["directions_norm"] = ray_bundle.metadata["directions_norm"]  as the following:

TypeError: 'NoneType' object is not subscriptable

Reference: 

https://discord.com/channels/1025504971606724712/1073830375559806976/1074782026781175919


Can you please help. Thanks in advance.



Discussion:

Resolving patch:
diff --git a/nerfstudio/models/depth_nerfacto.py b/nerfstudio/models/depth_nerfacto.py
--- a/nerfstudio/models/depth_nerfacto.py
+++ b/nerfstudio/models/depth_nerfacto.py
@@ -70,7 +70,8 @@ def populate_modules(self):
 
     def get_outputs(self, ray_bundle: RayBundle):
         outputs = super().get_outputs(ray_bundle)
-        outputs["directions_norm"] = ray_bundle.metadata["directions_norm"]
+        if ray_bundle.metadata is not None and "directions_norm" in ray_bundle.metadata:
+            outputs["directions_norm"] = ray_bundle.metadata["directions_norm"]
         return outputs
 
     def get_metrics_dict(self, outputs, batch):


Test patch:
Repository: nerfstudio-project/nerfstudio

Issue:
 "verbose" param is misinterpreted as "crop_border_pixels" positionally and cause unexpected cropping of images.
**Describe the bug**
Erroneous positional assignment of _verbose_ parameter is causing unwanted cropping behavior due to being misinterpreted as _crop_border_pixels_.

https://github.com/nerfstudio-project/nerfstudio/blob/a91b92f89d401115b42e6ad295a8b850e02ee86f/nerfstudio/process_data/process_data_utils.py#L262

https://github.com/nerfstudio-project/nerfstudio/blob/a91b92f89d401115b42e6ad295a8b850e02ee86f/nerfstudio/process_data/process_data_utils.py#L153

**To Reproduce**
Steps to reproduce the behavior:
1. run `ns-process-data images` with --verbose flag
2. the copied images are resized with 2 pixels less.

**Additional context**
Someone should review all positional assigned parameters. 


Discussion:

Resolving patch:
diff --git a/nerfstudio/process_data/process_data_utils.py b/nerfstudio/process_data/process_data_utils.py
--- a/nerfstudio/process_data/process_data_utils.py
+++ b/nerfstudio/process_data/process_data_utils.py
@@ -259,7 +259,7 @@ def copy_images(data: Path, image_dir: Path, verbose) -> int:
             CONSOLE.log("[bold red]:skull: No usable images in the data folder.")
             sys.exit(1)
 
-        num_frames = len(copy_images_list(image_paths, image_dir, verbose))
+        num_frames = len(copy_images_list(image_paths=image_paths, image_dir=image_dir, verbose=verbose))
 
     return num_frames
 


Test patch: