Repository: nerfstudio-project/nerfstudio

Issue:
why are there weird red margins of my loaded images?
**Describe the bug**
When I load the images, they have red margins.

**To Reproduce**
Steps to reproduce the behavior:
I just install the latest version of nerfstudio (commitid 9b6010ea), and pip install -e . to install it.

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.
<img width="508" alt="image" src="https://github.com/nerfstudio-project/nerfstudio/assets/44845151/5ae36028-17e7-4683-b1a4-cec33f24dcf6">


**Additional context**
Add any other context about the problem here.



Discussion:
Hm, I'm seeing something similar:
![image](https://github.com/nerfstudio-project/nerfstudio/assets/6992947/981f260a-8dfa-4ea2-8e95-8106308df7de)

I believe those are the poses used for eval.
I use the instant-ngp data parser, so all of the images are used for eval by default? 
When I use nerfstudio data parser, only a few images have red margins.
@FrederikWarburg any idea what's going on?
Hi! 

I had a look at this. It seems like `InstantNGP(DataParser)` is using all frames for training and for eval. `def _generate_dataparser_outputs(self, split="train"):` takes split as input, but the variable is unused. Therefore, the viewer first plots all the training images without a border (aka all images), and then, on top of the training images, plots the eval images with a red border (aka also all images). 

So, I guess we could 
1) add train/eval split to the `InstantNGP(DataParser) `
2) maybe remove `InstantNGP(DataParser)` as it seems to be almost the same as `Nerfstudio(DataParser)`. I guess the main difference is `get_focal_lengths`, but could easily be added to `Nerfstudio(DataParser)`

In short, I don't think the problem is the recent commit, but rather that using `InstantNGP(DataParser)` reuses all training as eval images. Wouldn't mind doing any of these changes, so let me know what you think is the best way forward? 

Resolving patch:
diff --git a/nerfstudio/data/dataparsers/instant_ngp_dataparser.py b/nerfstudio/data/dataparsers/instant_ngp_dataparser.py
--- a/nerfstudio/data/dataparsers/instant_ngp_dataparser.py
+++ b/nerfstudio/data/dataparsers/instant_ngp_dataparser.py
@@ -18,7 +18,7 @@
 
 from dataclasses import dataclass, field
 from pathlib import Path
-from typing import Dict, Tuple, Type
+from typing import Dict, Tuple, Type, Literal
 
 import imageio
 import numpy as np
@@ -32,6 +32,12 @@
     DataparserOutputs,
 )
 from nerfstudio.data.scene_box import SceneBox
+from nerfstudio.data.utils.dataparsers_utils import (
+    get_train_eval_split_filename,
+    get_train_eval_split_fraction,
+    get_train_eval_split_interval,
+    get_train_eval_split_all,
+)
 from nerfstudio.utils.io import load_from_json
 from nerfstudio.utils.rich_utils import CONSOLE
 
@@ -46,6 +52,18 @@ class InstantNGPDataParserConfig(DataParserConfig):
     """Directory or explicit json file path specifying location of data."""
     scene_scale: float = 0.3333
     """How much to scale the scene."""
+    eval_mode: Literal["fraction", "filename", "interval", "all"] = "fraction"
+    """
+    The method to use for splitting the dataset into train and eval.
+    Fraction splits based on a percentage for train and the remaining for eval.
+    Filename splits based on filenames containing train/eval.
+    Interval uses every nth frame for eval.
+    All uses all the images for any split.
+    """
+    train_split_fraction: float = 0.9
+    """The percentage of the dataset to use for training. Only used when eval_mode is train-split-fraction."""
+    eval_interval: int = 8
+    """The interval between frames to use for eval. Only used when eval_mode is eval-interval."""
 
 
 @dataclass
@@ -94,12 +112,40 @@ def _generate_dataparser_outputs(self, split="train"):
         assert (
             len(image_filenames) != 0
         ), """
-        No image files found. 
+        No image files found.
         You should check the file_paths in the transforms.json file to make sure they are correct.
         """
         poses = np.array(poses).astype(np.float32)
         poses[:, :3, 3] *= self.config.scene_scale
 
+        # find train and eval indices based on the eval_mode specified
+        if self.config.eval_mode == "fraction":
+            i_train, i_eval = get_train_eval_split_fraction(image_filenames, self.config.train_split_fraction)
+        elif self.config.eval_mode == "filename":
+            i_train, i_eval = get_train_eval_split_filename(image_filenames)
+        elif self.config.eval_mode == "interval":
+            i_train, i_eval = get_train_eval_split_interval(image_filenames, self.config.eval_interval)
+        elif self.config.eval_mode == "all":
+            CONSOLE.log(
+                "[yellow] Be careful with '--eval-mode=all'. If using camera optimization, the cameras may diverge in the current implementation, giving unpredictable results."
+            )
+            i_train, i_eval = get_train_eval_split_all(image_filenames)
+        else:
+            raise ValueError(f"Unknown eval mode {self.config.eval_mode}")
+
+        if split == "train":
+            indices = i_train
+        elif split in ["val", "test"]:
+            indices = i_eval
+        else:
+            raise ValueError(f"Unknown dataparser split {split}")
+        # Choose image_filenames and poses based on split, but after auto orient and scaling the poses.
+        image_filenames = [image_filenames[i] for i in indices]
+        mask_filenames = [mask_filenames[i] for i in indices] if len(mask_filenames) > 0 else []
+
+        idx_tensor = torch.tensor(indices, dtype=torch.long)
+        poses = poses[idx_tensor]
+
         camera_to_world = torch.from_numpy(poses[:, :3])  # camera to world transform
 
         distortion_params = camera_utils.get_distortion_params(


Test patch: