SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Task Definitions We further define the old task as using the camera poses on the six camera trajectories of the global view mentioned above to evaluate the overall quality of the entire scene. While the new task is defined as evaluating the model with an object-centric camera view to represent the local quality of the altered region. In the sequential continual learning setting, we define  $n$  tasks ranging from task 1 to task  $n$ . Each task corresponds to an operation with an object-centric camera pose centering around the altered region. Which this pose is used in capturing new training data for model adaptation and evaluation. Therefore, to evaluate the forgetting behaviors, we will evaluate all the previous tasks after each fine-tuning. For example, in Table 3, the cell "training on task 4 and testing on task 1" under FT block represents the model after tuning on add, delete, move, and replace data; then evaluate on the add region.
Evaluation Metrics PSNR is a well-established rendering quality assessment criterion that measures the peak signal-to-noise ratio between original and reconstructed images. We calculate two quality metrics through the same average process as the local view with a pose from the global view to evaluate the model's memorization of unaltered areas after updating the model. In addition, we have drawn inspiration from the commonly used backward transfer scores employed to assess a continual learning system [24] and have accordingly designed similar metrics for evaluating the forgetfulness
level of our method in the continual learning setting. The forgetting level between the model fine-tuning step  $n$  and step  $m$ , while  $n \leq m$ , is intuitively measured by the backward transfer model. At each fine-tuning step, a new dataset captured from locally altered regions with corresponding camera poses specific to that tuning step is utilized for model adaptation. It is noteworthy that the pose of each step also serves as the evaluation pose. In such case, we first calculate the  $\mathrm{PSNR}_{m,n}$  by using the model tuned at step  $m$  to render outputs given poses from step  $n$ . Then, the definition of the Backward Transfer Metric (BTM) is shown in Eq. (6).
$$
\mathrm {B T M} = - \frac {1}{T - 1} \sum_ {i = 1} ^ {T - 1} \left(\mathrm {P S N R} _ {T, i} - \mathrm {P S N R} _ {i, i}\right) \quad \text {a n d} \quad \mathrm {F M} = \frac {1}{T} \sum_ {i = 1} ^ {T} \mathrm {P S N R} _ {T, i}. \tag {6}
$$
BTM quantifies the level of forgetting by measuring the average cumulative gap between the original model performance  $(\mathrm{PSNR}_{i,i})$  and the final performance  $(\mathrm{PSNR}_{T,i})$ . Furthermore, we introduce a Forgetting Metric (FM) that measures the overall performance upon completion of the entire sequence of operations, with  $T$  being the last step. The metric evaluates the average PSNR obtained by having the last model infer all old poses as Eq. (6).
We construct two kinds of distinct datasets, synthetic and real, to assess the performance of our algorithm and baselines, specifically focusing on continuous learning and the phenomenon of forgetting, thereby showcasing the algorithm's versatility in diverse environments.
Synthetic Dataset Synthetic datasets are created using the Blender simulator, employing predefined meshes with textures. We provide three synthetic datasets: Dataset Whiteroom, Dataset Kitchen and Dataset Rome. The first two datasets encompass indoor, room-level scenes of a living room (depicted in Figure 4) and a kitchen, respectively. Using Dataset Whiteroom as an example, for the pre-training stage on the original scene, we capture roughly 200-300 multi-view images from six pre-defined camera trajectories discussed in Section 4.2. The camera poses are designed to encompass the entire scene. To fine-tune the pre-trained model after each operation applied to the original scene with new training data, we collect about 10 multi-view images for each time of change, totaling 40 images for all four operations. Notably, these new images are captured around the modified area's center. Furthermore, we captured all images at  $960 \times 640$  resolution and partitioned the overall data into an  $80\%$  training set and a  $20\%$  validation set. The final dataset, Dataset Rome, encompasses a large-scale synthetic outdoor environment, featuring a reconstructed mesh of the Colosseum. More descriptions for Dataset Kitchen and Dataset Rome are provided in the supplementary file.
Real Dataset We capture two challenging real-world scenes, Dataset Lab (indoor) and Dataset Court (outdoor), featuring diverse objects and environments to demonstrate our algorithm's generalizability. For indoor scenes, we use an iPhone 14 Pro to capture both the RGB and LiDAR depth streams via a custom iOS application, with poses determined by ARKit. For outdoor scenes, RGB images are captured with a drone, and poses are computed using COLMAP [37, 38].
We assess the core ATOMIC function ADD. For initial training, we take 100-150 images; for finetuning, an extra 10-20 near changes; to assess adjustments and study task-forgetting, we capture another 10-20 and 20-30 images, respectively, in old and revised areas. We resize all images at a resolution of  $960 \times 720$  and  $960 \times 640$ , respectively. More descriptions for Dataset Lab and Dataset Court are available in the supplementary file.
5.2 Baselines for Comparison
As no prior work on our task or related tasks exists, we develop several baselines to evaluate the efficacy of our proposed CL-NeRF. In the supplementary file, we also provide a comparison with the generalizable NeRF.
Navie Fine-Tuning Our first baseline is a straightforward fine-tuning approach, denoted as "FT", in which we use new data to fine-tune the NeRF model. In the sequential operation setting, we fine-tune the model at step  $t - 1$  after applying the  $t$ -th operation to the scene.
Memory Replay (MR) Inspired by the prevalent memory replay approach in lifelong learning [6, 11], we randomly select old training images to augment the fine-tuning training set. Our empirical study
sets the replay data to new data ratio at  $50\% : 50\%$ . In the case of multiple operations appearing in a sequential form, we extend the data memory pool with the new data provided by previous stages. We maintain the same sampling ratio when randomly selecting old data to supplement the training set.
DyNeRF [17] We adapt our task to meet the requirements of dynamic NeRF, i.e., DyNeRF[17]. Following its setting, we assign a latent code for data from each operation; this latent code is a trainable vector  $z_{t}$ , which is jointly fed into the MLP along with the original input  $x$ .  $z_{t}$  is a 32-dimensional vector that will be optimized during training. Specifically, in our task, we first pre-train an MLP NeRF with latent code  $z_{o}$  for representing the original scene. At each operation step  $t$ , we initialize a new latent code  $z_{t}$  and use it to replace the latent code in the trained MLP model at step  $t - 1$ . Then, the whole model is fine-tuned on the new dataset. Furthermore, to prevent forgetting and align the input format of the original DyNeRF, we also employ the same sampling strategy as used in the Memory Replay Baseline. This baseline essentially combines the original DyNeRF [17] and Memory Replay, which sets a strong baseline. Note that the original DyNeRF [17] does not have a mechanism to avoid forgetting.
5.3 Main Results
Due to page limitations, we present the results of a single synthetic dataset Dataset Whiteroom and qualitative results for several others. Additional results can be found in the supplementary file.
Single-step Operations We first compare our method with all three baselines introduced above on single-step operation experiments. As shown in Table 2, our CL-NeRF delivers exceptional performance across all operation types, particularly in ADD and DELETE operations. Our method achieves the best rendering quality on altered regions, shown in the new columns in Table 2 while having the minimum forgetting problem among all the methods. It is important to mention that DyNeRF uses previous-stage old image data in training, while our approach only utilizes new images. We present more results and analysis in the supplementary file.
Figure 4: Visualization results of CL-NeRF (Ours) and all compared baselines on 4 single-step operations: add, delete, move, and replace. The ground truth images and inference images from the pre-trained model are used to highlight the rendering quality on new tasks and quality degradation caused by forgetting severity.
Figure 4 demonstrates the algorithm's efficacy in both new and old tasks. The new scene images are rendered from camera poses that reflect scene changes (i.e., object-centric camera pose trajectory), while the old scene images are rendered from camera poses where the scene remains unchanged. Our method captures significantly more details for the newly added object compared to the other three methods. Additionally, our method can completely remove the sofa, while the entire sofa or traces of the sofa remain visible in MR or DyNeRF. In comparison to "FT", our model retains more details of unchanged areas during the delete operation, such as the pillow patterns. In move and replace operations, our model also demonstrates the advantages of preserving unchanged areas, particularly
Figure 5: Visualization results after sequential operations. The images in the upper row are rendered after the model has been fine-tuned on four operations, while the images in the lower row test and demonstrate the forgetting issue on the old task.
Figure 6: Synthetic Dataset Rome.
Figure 7: Real Dataset Lab/Court.
in detailed representations. Figures 6 and 7 further attest to our algorithm's adaptability across diverse environments.
Metrics	FM↑	BTM↓
FT	20.908	11.367
MR	20.930	1.233
Ours	28.638	1.707
Sequential Operations Table 1, 2, 3 and Figure 5 present the results of sequential operations. Table 1 employs both BTM and FM to quantify the issue of forgetting, with specifics elucidated in Eq. 6. Table 2 denotes "SEQ" as the sequential operation, evaluating the final model's performance following a sequence of operations, whereas Table 3 examines performance at each individual stage. It can be observed that our proposed method exhibits superior performance in all evaluated tasks compared to the baseline methods. It performs on par with the "FT" base
line in Task 4. This is because both training and evaluation are conducted on Task 4, allowing the fine-tuning baseline to benefit more from the new data. Qualitative comparisons in Figure 5 also demonstrate the effectiveness of our method in delivering high-quality results.
Table 2: Rendering results of atomic/composite/sequential operation on Dataset Whiteroom. It showcases the PSNR on both new and old tasks following four single-step operations. The best results are highlighted in bold, while the second-best results are underlined.
Table 1: Forgetting metrics in sequential operations SEQ on Dataset Whiteroom.
Operations	ADD	DELETE	MOVE	REPLACE	SEQ
Old	New	Old	New	Old	New	Old	New	Old	New
FT	24.21	23.54	22.89	34.76	20.93	29.29	21.45	30.14	16.91	27.36
MR	31.97	19.40	31.40	17.21	32.39	25.66	32.48	25.15	25.95	24.45
DyNeRF	32.29	23.89	31.39	25.82	32.83	30.12	32.70	29.88	25.80	29.83
Ours	32.33	25.29	32.43	34.77	33.30	29.97	33.33	29.82	33.38	29.74
Table 3: Rendering result of the sequential operation on Dataset Whiteroom. Tasks 1-4 are defined to evaluate the model based on camera poses corresponding to four operations: add, delete, move, and replace, respectively. Models are evaluated after being tuned for previously completed tasks. The best results are highlighted in bold, while the second-best results are underlined.
Testing on	FT	MR	Ours
Task1	Task2	Task3	Task4	Task1	Task2	Task3	Task4	Task1	Task2	Task3	Task4
Training on	Task1	23.54	-	-	-	19.40	-	-	-	25.29	-	-	-
Task2	20.65	34.76	-	-	17.48	17.21	-	-	24.41	34.77	-	-
Task3	16.81	21.55	29.29	-	17.46	17.63	25.66	-	24.05	33.78	29.97	-
Task4	14.63	20.28	18.58	30.14	17.49	17.80	23.28	25.15	23.71	33.07	28.13	29.82
Expert Adaptor and Distillation Our model comprises two essential components: the lightweight expert adaptor and the conflict-aware knowledge-distillation learning objective. We conduct an ablation study on these components, with the results shown in Table 4. These results demonstrate that both modules contribute to improving performance. Without the expert adaptor, the model's performance drops significantly when evaluating new scenes, highlighting the effectiveness of the expert adaptor in enabling the model to adapt to new changes. In the absence of knowledge distillation (KD), the model performs poorly on old scenes, indicating that KD helps address the forgetting problem for the Continual Learning of NeRF.
Table 4: Ablation Study on Dataset Whiteroom. We study the importance of the expert adaptor and distillation strategy.
Operations	ADD	DELETE	MOVE	REPLACE	SEQ
Old	New	Old	New	Old	New	Old	New	Old	New
w/o Expert	31.24	23.42	30.36	27.48	31.23	30.86	31.33	30.00	29.39	24.30
w/o KD	29.81	24.10	29.27	34.57	23.90	29.92	23.47	28.97	18.63	28.19
w/o Lm	31.08	24.92	30.99	31.73	31.90	28.64	32.63	28.53	30.70	28.05
Ours	32.33	25.29	32.43	34.77	33.30	29.97	33.33	29.82	33.38	29.74
Mask Loss. In order to demonstrate the necessity of mask loss in resolving conflict, we conduct an experiment with mask loss  $L_{m}$  removed. As shown in Table 4, our findings suggest that mask loss  $L_{m}$  also contributes to high-quality view syntheses. Without the guidance provided by the mask loss, the model may struggle to correctly predict object boundaries and often produce inaccurate results.
We have presented a method CL-NeRF for efficiently adapting pre-trained NeRF to changing environments. CL-NeRF comprises two major components: a lightweight expert adaptor for modeling new changes in evolved scenes, and a conflict-aware knowledge distillation strategy for preserving memory in unaltered areas. These two designs collaborate, enabling our method to achieve superior results. To facilitate future research, we have also introduced a continual learning benchmark for NeRF with comprehensive evaluation metrics. To the best of our knowledge, this is the first of its kind. We hope our investigation and comprehensive benchmark inspire further research in this task, empowering NeRF to effectively model the ever-changing real-world scenes. More discussions about broader impacts, limitations including the scalability and other parameter-efficient tuning (PET) method are described in the supplementary file.
This work has been supported by Hong Kong Research Grant Council - Early Career Scheme (Grant No. 27209621), General Research Fund Scheme (Grant No. 17202422), and RGC Matching Fund Scheme (RMGS), and Tencent Research Fund.
[39] Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphics, 29(5):2732-2742, 2023.
[40] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5459-5469, 2022.
[41] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8248–8258, June 2022.
[42] Sebastian Thrun and Sebastian Thrun. Explanation-based neural network learning. Springer, 1996.
[43] Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In The European Conference on Computer Vision (ECCV), 2022.
[44] Tianhan Xu and Tatsuya Harada. Deforming radiance fields with cages. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII, pages 159-175. Springer, 2022.
[45] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547, 2017.
[46] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987-3995. PMLR, 2017.
[47] Chengwei Zheng, Wenbin Lin, and Feng Xu. Editablenerf: Editing topologically varying neural radiance fields by key points. arXiv preprint arXiv:2212.04247, 2022.