CL-NeRF: Continual Learning of Neural Radiance Fields for Evolving Scene Representation
Authors: Xiuzhe Wu
Peng Dai
Weipeng DENG
Handi Chen
Yang Wu
Yan-Pei Cao
Ying Shan
XIAOJUAN QI
Venue: NeurIPS 2023 poster (2023)
Keywords: Neural Radiance Field; Continual Learning; Scene Representation

Abstract: Existing methods for adapting Neural Radiance Fields (NeRFs) to scene changes require extensive data capture and model retraining, which is both time-consuming and labor-intensive. In this paper, we tackle the challenge of efficiently adapting NeRFs to real-world scene changes over time using a few new images while retaining the memory of unaltered areas, focusing on the continual learning aspect of NeRFs. To this end, we propose CL-NeRF, which consists of two key components: a lightweight expert adaptor for adapting to new changes and evolving scene representations and a conflict-aware knowledge distillation learning objective for memorizing unchanged parts. We also present a new benchmark for evaluating Continual Learning of NeRFs with comprehensive metrics. Our extensive experiments demonstrate that CL-NeRF can synthesize high-quality novel views of both changed and unchanged regions with  high training efficiency, surpassing existing methods in terms of reducing forgetting and adapting to changes. Code and benchmark will be made available.

CL-NeRF: Continual Learning of Neural Radiance Fields for Evolving Scene Representation
Xiuzhe Wu
Peng Dai Yan-Pei Cao
Weipeng Deng Ying Shan
Handi Chen Xiaojuan Qi
Yang Wut
Thhe University of Hong Kong FARC Lab, Tencent PCG
Tencent AT Lab
Abstract
Existing methods for adapting Neural Radiance Fields (NeRFs) to scene changes require extensive data capture and model retraining, which is both time-consuming and labor-intensive In this paper, we tackle the challenge of efficiently adapting NeRFs to real-world scene changes over time using a few new images while re- taining the memory of unaltered areas, focusing on the continual learning aspect NeRFs. To this end, we propose CL-NeRF, which consists of two key com- ponents: lightweight expert adaptor for adapting to new changes and evolving scene representations and conffict-aware knowledge distillation learning objective for memorizing unchanged parts. We also present new benchmark for evalu- ating Continual Learning NeRFs with comprehensive metrics. Our extensive experiments demonstrate that CL-NeRF can synthesize high-quality novel views of both changed and unchanged regions with high training efficiency, surpassing existing methods in terms of reducing forgetting and adapting changes. Code and benchmark will be made available.
Introduction
Neural Radiance Fields (NeRFs), powerful models capable of synthesizing high-fidelity novel views from multi-view images 26) 71. play crucial roles in various applications including AR/VR and autonomous driving simulators. However, real-world scenes constantly evolve over time, such as a chair being removed from a room. To handle such changes, it is essential to develop adaptable representations that can be continuously updated reflect these changes accurately with minimal
data and computational cost. However, existing NeRF methods 22 26 43 lot of new data and full retraining of the model to adapt even minor local changes, which is impractical and costly, especially for large-scale scenes e.g., city-scale scene.
41 40 often require
In this paper, we study novel efficient continual learning task for NeRF, where we aim update scene's 's NeRF using only a few new images (i.c., 10-20 images) that capture the scene changes over time while preserving the memory of the unchanged areas (see Figure I.). A simple approach is to fine-tune the NeRF with the new images, but this suffers from catastrophic forgetting of the unaltered areas (see Figure 1) One might argue that using explicit 3D representations,
9 22 27]. such as Plenoxels (9) could mitigate this problem since the explicit 3D representation might not change for the unchanged regions during fine-tume fine-tuning. Nonetheless,
initial experiments suggest that forgetting still occurs (refer to Figure D This because the ray-based volumetric rendering formula 26 22 27] propagates supervision signals all voxels along a ray, which may affect other regions that share voxels with the same ray. The most related work is dynamic NeRFs [17 16 30), which also model
dynamic changes using time-aware latent codes [17] scene flows 19 But it focuses changes
Equaa contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

uperations
Composite operations
Training
ML.P-based results
Vaxe-based results
Roplace
Pergertie
Rarliy
Fine
Nathing
Blokks
Figure 1: Catastrophic forgetting (left) and scene changes simulation (right). The left part illustrates the forgetting behavior MLP-based and Voxel- based NeRF during fine-tuning, while the right side simulates representative real-world change operations.
caused by continuous object motions and is designed to remember the observed regions at a specific timestamp, leading to forgetting of unobserved regions in large-scale scene.
To address this challenge, we introduce Continual Learning NeRF (CL-NeRF), which efficiently adapts a learned NeRF model to environmental changes using minimal new data while retaining its memory of unchanged areas. It consists of two core components: an expert adaptor for adapting to scene changes and conflict-aware distillation scheme for memorizing unchanged parts. The expert adaptor is lightweight network that learns to encode local changes from a few new images while keeping the original NeRF parameters fixed. It also predicts weight value called mask logit that indicates the likelihood of the rendering point being in the changed areas. It is used to fuse the original NeRF features and the expert adaptor for rendering. Further, the conflict-aware knowledge distillation scheme trains the model to preserve the original scene representation for unchanged areas by minimizing the discrepancy between the outputs of the original NeRF and the adapted model. The mask logit predicted by the expert network balances the distillation loss and rendering loss of new images, avoiding conflicting supervision signals.
Finally, we introduce a new continual learning benchmark for NeRF, consisting of synthetic and real-world scenes. Our benchmark encompasses four fundamental operations that represent prevalent real-world scene changes: add, remove, move, and replace. Furthermore, to facilitate evaluation, we develop comprehensive metrics assess the model's 's effectiveness in preserving unaltered areas and adapting to new changes in terms of accuracy and efficiency. Our extensive experiments, involving three baselines (i.e., fine-tuning, memory replay, and DyNeRF(17), demonstrate that model consistently outperforms all compared approaches on all evaluated scenarios while maintaining superior efficiency. Especially, our approach maintains high-quality results throughout series of operations in both old and new tasks as evidenced by our PSNR scores (29.74 and 33.38 in the new and old task, respectively.) Notably, our approach effectively mitigates the forgetting phenomenon when compared to direct fine-tuning (see Figure
In summary, primary contributions
1) To our best knowledge, we present the first study on Continual Learning of NeRF, aiming to enable a NeRF model adapt changes in real-world scenes. We also introduce novel benchmark for this task, which can serve as a valuable resource for future research.
2) We propose a new method CL-NeRF - consisting of two key components: a lightweight expert adaptor for adapting new changes and conffict-aware knowledge-disttllation objective for preserving unaltered scene representations.
3) We conduct extensive experiments and show that our model achieves high-quality view synthesis in both altered and unaltered regions, outperforming the baseline methods and mitigating the forgetting phenomenon.
2 Related Works
Neural Radiance Fields, Dynamic NeRF and NeRF Editing Neural Radiance Fields (NeRF) 26 have drawn lot of attention recently due to their high-fidelity rendering quality. To render novel-view images, previous methods encode the geometry and appearance into the weights of ML.s 26
or other explicit neural 3D representations [22 9. then render novel-view images through volume

---

rendering [12 based on the predicted radiance values of query points. However, most of these methods require per-scene optimization and are fragile to the scene changes [35]
To handle scene changes, the vanilla NeRF [26] is modified to model the dynamic objects the scene. This line of work usually takes videos containing dynamic objects inputs and can be divided into two lines: deformation field estimation [19 20 28 31
39 29 g and the incorporation of time-aware inputs [7 36 Dycheck [TO] offers critical assessment of recent developments in dynamic NeRFs. Given that the dynamic NeRFs are only trained on limited observations of the large-scale scene at each timestamp, thus the sigh-iidelity rendering that timestamp constrained to the observed regions. Different from dynamic NeRFs, we aim to render high-fidelity images of the entire scene at any given timestamp.
The recent progress made in enabling NeRFs to support geometry or appearance editing 23 15
47 has been quite significant recently. They focus editing objects within the scene, for example, changing the color [2)] editing the parts of objects (23] or making deformations [15 44). This
line of work usually handles editing following the user's instruction and limited existing objects within the scene while our goal is to adapt NeRFs to various real-world scene changes efficiently.
Continual Learning Continual Learning aims to learn different tasks progressively [42] 33). However, catastrophic forgetting is the key challenge, meaning the model will have performance degradation on previous tasks after training on new [Isk In our task, the Continual Learning of NeRF is defined as a model that should not have artifacts or quality degradation on unaltered areas after the model has adapted to the scene changes multiple times. (See Figure 2}. To tackle the forgetting problem in continual learning, some researchers propose memory replay method 5 6 [1]. which will store the previous training data and use them to augment the current training set while training the model on the new task. Besides, adding regularization constraints to the neural network to limit the parameter
Retrain
24h
24h
24h
Init
Add
Delete
Fine-tune
24h
0.2h
0.2h
Figure 2: Illustration of the Continual Learning of NeRF.
updated also proves to be an effective solution ] 1 21 46). Lastly, Adding separate module to handle different tasks also draw some attentions [18 32
3 Continual Learning of NeRF
Our study focuses on continual learning for scene-level NeRF, where the goal is to update pre-trained scene-level NeRF efficiently by utilizing a few new images while retaining the memory of unchanged areas. Figure 2 Continual Learning task of NeRF and subsequently deseribe our preliminary study which highlights the forgetting issue and motivates our method.
provides an illustration of the process. In this section, we will first formulate the
3.1 Problem Formulation
We represent a 3D scene as 5D radiance function, denoted by Fe (z. d) (c. in accordance with the vanilla NeRF (26). Specifically, of parameters, to predict the volume density a and color e according to each sample point a and its corresponding view direction d. NeRF has demonstrated high capacity for high-quality novel view synthesis. Our focus is on the Continual Learning of NeRF to adapt to changes while maintaining the unaltered components.
approach utilizes an MLP Fe. where 0 represents set
To simulate natural scene changes, we incorporate basic operations, denoted as Op, including adding, removing. moving, and replacing objects, which are common factors contributing to scene alterations. To emulate these scene changes over time, we simulate a series of operations [Op.. Op2: Op..l- representing time-varying changes that result in evolved scenes [Su, Sa,, .... Su,, respectively. Our simulation aims to capture the following characteristics of real-world scenes: 1) localized changes occurring within objects. 2) the impracticality of extensively recapturing scene-level data in large- scale environments, such as rooms or cities, due to time and cost constraints, and 3) the dynamic evolution of real-world scenes over time.
3

---

F9
(x
Frozen
Frozen
PE(x)
(x)
Frozen
Features
Fised feutures
Lightwiigtt expert adaptor
Backbone feotures
featuress
IH Mark logil
Gating module
Figure 3: An overall inference framework of CL-NeRF. The position encoding PE( is fed into both the original MLP backbone and the proposed lightweight expert adaptor module (the green dashed box), generating backbone features (3r) and expert features eo (2:). A scalar output by the expert utilized for feature fusion. Finally, the fused feature fō (æ: is passed to the volume density and color prediction heads for further volumetric rendering.
Given the changes in scenes over time, our continual learning task defined as follows: initially, we train model F6 with multi-view images of original scene S., Subsequently, we progressively update Fg using l new multi-view images that capture scene changes Su, after operation Op, at each time step i. It is worth noting that the training dataset is significantly smaller compared the original one (i.e., k < m).
3.2 Preliminary Study and Motivations
Preliminary Study A straightforward approach the continual learning task is to directly fine-tune an existing NeRF model with newly acquired data at each instance. We evaluate this baseline using a simple single-operation preliminary experiment, Specifically, as shown in Figure[]the room-level scene comprises several pieces of furmiture. We capture around 200-300 images for the scene to train the original model Fe. Then, we modify the scene with the ADD operation (i.c., placing several apples on the table) and take 10 images around the object. Finally, we use the new data to fine-tune Fg as a baseline.
Catastrophic Forgetting However, this simple strategy leads to unsatisfactory performance. While achieving acceptable quality for the updated region after the change, the model results in a catastrophic deterioration of the quality in unaltered regions. As illustrated in Figure] I after fine-tuning with the new data, artifacts appear on the bookshelf, which on the unchanged area located on the right-hand side of the table. The average PSNR dramatically drops from 32.71 to 24.21. We conjecture that NeRF stores geometry and appearance information in the weights of its MLP model in coupled and implicit manner. Consequently, applying changes locally without affecting other regions is difficult. This issue presents a significant obstacle in adapting pre-trained NeRF models to changing real- world scenes while preserving unaltered areas. The problem is particularly pronounced in large-scale scenes, making recapturing and full retraining impractical in many cases.
Comparisons with Traditional Continual Learning Task In practical settings, continually captur- ing images and retraining NeRF after every scene change ideal but unfeasible for large, dynamic scenes. Instead, we propose more realistic method: fine-tuning existing NeRFs with recent images consistently, which resembles traditional Continual Learning (CL) tasks. This technique, however, can induce artifacts and catastrophic forgetting in previously trained areas, as noted traditional CL tasks 13.. The divergence between CL in NeRF and traditional CL tasks lies in the inherent conflict between new and old data. While traditional CL tasks maintain stable one-to-one outptt correspondence, NeRF attributes colors and densities to specific points. Scene alterations can modify these associations, leading to conflicting supervisory signals. Standard CL solutions like Memory Replay (MR) prove insufficient in NeRF due to these inconsistencies, particularly in changed regions. To navigate these challenges, our study introduces novel method Section 4
4
Our Method
In this section, we introduce our novel approach, CL-NeRF, for adapting pre-trained NeRF to scene changes over time. The pipeline comprises two major components: a lightweight expert adaptor specifically designed to adapt to new scene changes and conffict-aware knowledge distillation

---

training objective to preserve the memory in unaltered areas (see Section As shown in Figure 3 the overall framework involves a pre-trained Fg for modeling the entire scene and a small set of new training images 7 that capture local scene changes. The lightweight trainable expert adaptor, represented as Fg. generates a feature for capturing scene changes which further fused with features from the original Fg using a masking mechanism. To prevent forgetting, we introduce a conflict-aware self-distillation mechanism (as illustrated in Figure 3) that distills knowledge from Fg, particularly in unaltered areas. Moreover, we propose a continual learning benchmark for NeRF, which includes various operational definitions and metries to assess the overall neural rendering performance in adapting changes and preserving unchanged content (see Section 4.2
4.1 CL-NeRF
Lightweight Expert Adaptor Our approach motivated by the observation that scene changes primarily occur in localized regions, fine-tuning all the model parameters resulting in forgetting in unaltered regions. To address the above issue, we introduce lightweight expert adaptor, denoted as Eg. which can be seamlessly integrated with the original NeRF model, represented by P'. as plug-and-play module. In addition to encoding information about newly modified areas of the scene, Eg also provides a rollback strategy that enables users to revert to any desired timestamp. By contrast, Fg is responsible for storing information about the overall original scene. These two components offer a comprehensive solution for modeling dynamic scenes with high fidelity.
To achieve our objectives, we define the input and output of Eg in the same manner as for Fo- Specifically, Eg takes the same input as the backbone Fo and generates two outputs: a mask logit m, and a feature vector fo. which subsequently integrated into the original Fo. The mask logit m takes on a gating role in the feature fusion module, whereas the feature output by Fg is fused with the output of Eg, as illustrated in Figure 3 This approach enables us to capture dynamic changes in the scene with Eg while preserving the overall structure of the original scene with Fg.
Concretely, each sampling point x is fed to the original network Fe. which generates feature fe(x) representing previous contents. Simultancously. Es predicts residual feature vector for x, represented as cg(x). The final fused feature fā (z) can be calculated as
(x fa(x) + m ex)
(I)
Then, fö (ar) is fed back into the remaining component of Fg to further predict volume-density a and view-dependent color e for neural rendering. As approaches O, the output features should remain as close as possible to the original features. When m approaches 1, the predicted features from Es will be most effective in predicting modifications made new areas, also such a design encourages the model not to change the origin scene.
Conffict-Aware deff-Distillation Mechanism The student model, responsible for adapting to new scenario, is initialized with prior knowledge and fine-tuned to incorporate relevant information from the previous scene. To prevent catastrophic forgetting. we designate the previously trained original model in the preceding scene as the teacher model. Through self-distillation we transfer scene-related information from the teacher model to the student model. To be specific, we randomly sample camera views and points in space and feed them into both the teacher model and the student model. The teacher model's prediction is used to supervise the student model.
However, the supervision signals provided by the images captured in new camera views can conflict with the supervision from the teacher model. To this end, we propose conflict-aware knowledge distillation. Specifically, for each sampling point, our expert adaptor has already predicted a mask logit. This allows us to utilize volume rendering in the same way as image prediction render a soft mask map M. This soft mask map can serve as a probability map indicating the likelihood of a pixel belonging to the changed area. A value close to signifies high probability of the pixel belonging to the conflict region, in which case the self- deff-disiillatin mechanism need not be executed. Conversely, when there is higher probability of the pixel belonging to the unaltered region, calculating an additional knowledge distillation loss to make the supervision signals more accurate. It can be formulated as
Lu = iī, (1 M) - i, -1-(111
(1 M)
(2)
where is and f, are the images predicted by the student model and the teacher model, respectively.
5

---

Training Objective Our whole pipeline is trained in self-supervised manner. The overall loss includes a photometric loss Cph. conflict-aware self dif-ddstillattio loss Lı and a mask loss C. is well noted that we introduce self-distillation training mechanism therefore we can have both the observed new images and teacher model to provide the supervisory signals. Hence we divide the inputs into two parts based on camera pose distributions, defined as new and old poses, respectively. To be specific, when the input points are sampled using new camera poses, we use Lph to generate supervision. Lpb measures the reconstruction error between the predicted image I, and newly captured ground-truth image I, as
Pph = | - 12
(3)
To further enhance the change awareness, also propose mask loss under new poses
Lm = MM - Mgell2,
where Mgr is the pscudo ground-truth mask. Mg is computed by the difference map between newly captured images and the predicted images from the teacher model using the same camera pose the newly captured images. Besides, when we want to compute Led, the uniformly sampled old camera views can be used as inputs to both the teacher model and the student model, and the definition is Finally, the overall training objective is
given in Equation 2
Call Wid
Led + Wpph Lph + Wm Lm-
(5)
More implementation details are described in the supplementary file.
4.2 Benchmarks
To our best knowledge, there is no existing dataset for our proposed continual learning task of NeRF and no proper evaluation metrics evaluate the forgetting behavior. Therefore, in this section, we will first propose a new dataset containing two synthesis scenes, one real-world scene, and an extra city-level scene. Then we propose metries specifically designed evaluate how serious the forgetting after fine-tuning is.
We categorize scene operations into three types: atomic, composite, and sequential. Atomic operations consist of addition and deletion. Composite operations include movement and substitution, which modify positional relationships replace existing objects with new ones. Sequen- tial operations apply a series of basic or composite operations over time. Different from composite operations' focus quick, specific region modifications, sequential operations aim to monitor long-term error accumulation in changing world scenarios.
Operations
Camera Views We evaluate the reconstruction quality of both local details and the overall scene using two testing indicators: local and global views consisting of a set of evaluation camera poses. Six camera trajectories are designed to cover the entire scene, with five capturing the four orientations of a room (local view) and the sixth simulating a person wandering around the room (global view). Additionally, we define an object-centric camera view to assess adaptation to new changes.
Task Definitions We further define the old task as using the camera poses on the six camera trajectories of the global view mentioned above to evaluate the overall quality of the entire scene. While the new task is defined as evaluating the model with an object-centric camera view to represent the local quality of the altered region. In the sequential continual learning setting, we define n tasks ranging from task to task n. Each task corresponds to an operation with an object-centric camera pose centering around the altered region. Which this pose is used in capturing new training data for model adaptation and evaluation. Therefore, to evaluate the forgetting behaviors, we will evaluate all the previous tasks after each fine-tuning, For example, in Table 3 the cell "training task 4 and testing on task 1" under FT block represents the model after tuning on add, delete, move, and replace data; then evaluate on the add region.
Metrics PSNR is well-established rendering quality assessment criterion that measures the peak signal-to-noiss ratio between original and reconstructed images. We calculate two quality metrics through the same average process as the local view with a pose from the global view to evaluate the model's sss memorization of unaltered areas after updating the model. In addition, we have drawn inspiration from the commonly used backward transfer scores employed to assess a continual learning system (24] and have accordingly designed similar metries for evaluating the forgetfulness
Evaluation

---

level of our method in the continual learning setting. The forgetting level between the model fine- tuning step n and step m, while n I m, is intuitively measured by the backward transfer model. At each fine-tuning step, a new dataset captured from locally altered regions with corresponding camera poses specific to that tuning step is utilized for model adaptation. It is noteworthy that the pose of each step also serves as the evaluation pose. In such case, we first calculate the PSNR m, by using the model tuned at step m to render outputs given poses from step n. Then, the definition of the Backward Transfer Metric (BTM) is shown in Eq. (
| - PSNRE,. BTM = T- (PSNRY, , and FM T -
1 T-
(6)
BTM quantifies the level of forgetting by measuring the average cumulative gap between the original model performance (PSNR,,) and the final performance (PSNRT;). Furthermore, we introduce Forgetting Metric (FM) that measures the overall performance upon completion of the entire sequence of operations, with T being the last step. The metric evaluates the average PSNR obtained by having the last model infer all old poses as Eq. (6)
Experiment
5.1 Datasets
We construct two kinds of distinct datasets, synthetic and real, to assess the performance of our algo- rithm and baselines, specifically focusing on continuous learning and the phenomenon of forgetting, thereby showcasing the algorithm's is versatility in diverse environments.
Synthetic datasets created using the Blender simulator, employing predefined meshes with textures. We provide three synthetic datasets: Dataset Whiteroom, Dataset Kitchen and Dataset Rome. The first two datasets encompass indoor, room-level scenes of living room (depicted in Figure 4 and a kitchen, respectively. Using Dataset Whiteroom as an example, for the pre-training stage on the original scene, we capture roughly 200-300 multi-view images from six pre-defined camera trajectories discussed in Section 4.2 The camera poses are designed to encompass the entire scene. To fine-tune the pre-trained model after each operation applied the original scene with new training data, we collect about 10 multi-view images for each time of change, totaling 40 images for all four operations. Notably, these new images are captured around the modified area's center. Furthermore, we captured all images at 960x64) resolution and partitioned the overall data into an 80% training set and a 20% validation set. The final dataset, Dataset Rome, encompasses large-scale synthetic outdoor environment, featuring reconstructed mesh of the Colosseum. More descriptions for Dataset Kitchen and Dataset Rome are provided in the supplementary file.
Synthetic Dataset
Real Dataset We capture two challenging real-world scenes, Dataset Lab (indoor) and Dataset Court (outdoor), featuring diverse objects and environments to demonstrate our algorithm's ' general- izability. For indoor scenes, we use an iPhone 14 Pro to capture both the RGB and LiDAR depth streams via a custom iOS application, with poses determined by ARKit. For outdoor scenes, RGB images captured with drone, and poses are computed using COLMAP 37 38
We assess the core ATOMIC function ADD. For initial training, we take 100-150 images; for finetuning, extra 10-20 near changes; to assess adjustments and study task-forgetting, we capture another 10-20 and 20-30 images, respectively, in old and revised areas. We resize all images at a resolution of 960 720 and 960x 640, respectively. More descriptions for Dataset Lab and Dataset Court are available in the supplementary file.
5.2 Baselines for Comparison
As no prior work on our task or related tasks exists, we develop several baselines to evaluate the efficacy of our proposed CL-NeRF. In the supplementary file, we also provide a comparison with the generalizable NeRF.
Navie Fine- Tane-TTnngg Our first baseline is straightforward fine-tuning approach, denoted "FT", in which we use new data to fine-tune the NeRF model. In the sequential operation setting, we fine-tune the model at step L 1 after applying the 1-th operation to the scene.
Memory Replay (MR) Inspired by the prevalent memory replay approach in lifelong learning 6 we randomly select old training images to augment the fine-tuning training set. Our empirical study
7

---

sets the replay data to new data ratio at 50%:50%. In the case of multiple operations appearing in sequential form, we extend the data memory pool with the new data provided by previous stages. We maintain the same sampling ratio when randomly selecting old data to supplement the training set.
DyNeRF [17] We adapt task meet the requirements of dynamic NeRF, i.e., DyNeRF(17.
Following its setting, we assign latent code for data from each operation; this latent code is trainable vector 2g, which jointly fed into the MLP along with the original input æ. 2t is a 32- dimensional vector that will be optimized during training. Specifically, in our task, we first pre-train an MLP NeRF with latent code 2, for representing the original scene. each operation step t, we initialize new latent code 2; and use it to replace the latent code in the trained MLP model at step 1. Then, the whole model fine-tuned align the imput format of the original DyNeRF, we also employ the same sampling strategy as used the Memory Replay Baseline. This baseline essentially combines the original DyNeRF [17] and Memory Replay, which sets strong baseline. Note that the original DyNeRF 17 does not have mechanism to avoid forgetting.
the new dataset. Furthermore, prevent forgetting and
5.3 Main Results
Due to page limitations, we present the results single synthetic dataset Dataset Whiteroom and qualitative results for several others. Additional results can be found in the supplementary file.
Single-step Operations We first compare our method with all three baselines introduced above on single-step operation experiments. As shown in Table 2 our CL-NeRF delivers exceptional performance across all operation types, particularly in ADD and DELETE operations. Our method achieves the best rendering quality on altered regions, shown in the new columns Table while having the minimum forgetting problem among all the methods. It is important to mention that DyNeRF uses previous-stage old image data training, while our approach only utilizes new images. We present more results and analysis in the supplementary file.
Add
Delete
Move
Replace (Books) Old
New
(Aplee)
New
(Sofa)
(Radio)
Old
MR
FT
Ours DyNeRF
Ground tuthh Pre-trained
Ground truth Pre-trained Ground truth
Pre-trained
Ground truth Pre-trained
Figure 4: Visualization results of CL-NeRF (Ours) and all compared baselines on 4 single-step operations: add, delete, move, and replace. The ground truth images and inference images from the pre-trained model are used to highlight the rendering quality on new tasks and quality degradation caused by forgetting severity.
Figure demonstrates the algorithm's 's efficacy in both new and old tasks. The new scene images are rendered from camera poses that reflect scene changes (i.e., object-centric camera pose trajectory). while the old scene images are rendered from camera poses where the scene remains unchanged. Our method captures significantly more details for the newly added object compared to the other three methods. Additionally, our method can completely remove the sofa, while the entire sofa or traces of the sofa remain visible in MR or DeRE In comparison to "FT", our model retains more details of unchanged areas during the delete operation, such the pillow patterns. In move and replace operations, model also demonstrates the advantages of preserving unchanged areas, particularly

---

MR FT
DyNeRF
Ours
Ground
New
Old
Pre-frained
Figure 5: Visualization results after sequential operations. The images in the upper rendered after the model has been fine-tuned on four operations, while the images in the lower row test and demonstrate the forgetting issue on the old task.
Add (Indsor)
Add (Outdeor)
New
(Greenbaz)
Old
New
(oog)
MI
NI
Our
Ground
Figure 6: Synthetic Dataset Rome.
Figure 7: Real Dalaset Lab/Court.
in detailed representations. Figures and further attest to our algorithm's 'ss adaptability across diverse
environments.
Sequential Operations Table I and Figure present the results of sequential operations. Table embloys both BTM and FM quantify the issue of forgetting. with specifics elucidated in Eq. 6 Table debotess "SEQ" as the sequential operation, evaluating the final model's per- formance following a sequence of operations, whereas Table GableeBexmmeen examines performance at each individual stage. can be observed that our proposed method exhibits supe- rior performance all evaluated tasks compared to the baseline methods. It performs on par with the "FT"" base- line Task This is because both training and evaluation are conducted on Task 4, allowing the fine-tuning bascline to benefit more from the new data. Qualitative comparisons in Figure 5 also demonstrate the effectiveness our method delivering high-quality results.
employs
Metries
FM
BTM
FT
20.908
11.367
MR
20.930
1.233
Ours
28.6338
1.707
Table I: Forgetting metrics in sequential operations SEQ on Dataset Whiteroom.
Table 2: Rendering results of atomicccompossiessquueiiaaa operation on Dataset Whiteroom. showcases the PSNR on both new and old tasks following four single-step operations. The best results are highlighted in bold, while the second-best results are underlined.
ADD
DELETE
MOVE
REPLACE
SEQ
Operations
Old
New
Old
New
Old
New
New
New
FT
24.21
23.54
22.89
34.76
20.93
29.29
21.45
30.14
16.91
27.36
MR
31.97
19.40
31.40
17.21
32.39
25.66
32.48
25.15
25.95
24.45
DyNeRF
32.29
23.89
31.39
25.82
32.83
30.12
32.70
29.88
25.80
29.83
Ours
32.33
25.29
32.43
34.77
33.30
29.97
33.33
29.82
33.38
29.74

---

Table 3; Rendering result of the sequential operation on Dataset Whiteroom. Tasks 1-4 are defined to evaluate the model based on camera poses corresponding to four operations: add, delete, move, and replace, respectively. Models are evaluated after being tuned for previously completed tasks. The best results are highlighted in bold, while the second- best results are underlined.
FT
MR
Ours
Testing on
Task]
Task2
Task3
Task4
Task!
Task2
Tank3
Task4
TankI
Task2
Task3
Task4
Taskl
23.54
19.40
25.29
Training
Task2
20.65
34.76
17.48
17.21
24.41
34.77
Task3
16.81
21.55
29.29
17.46
17.63
25.66
24.05
33.78
29.97
Tiased
14.63
20.28
18.58
30.14
17.49
17.80
23.28
25.15
23.71
33,07
28.13
29.82
5.4
Ablation Study
Expert Adaptor and Distillation Our model comprises essential components: the lightweight expert adaptor and the conflict-aware knowledge-distilaan learning objective. We conduct an ablation study on these components, with the results shown Table These results demonstrate that both modules contribute to improving performance. Without the expert adaptor, the model's performance drops significantly when evaluating new scenes, highlighting the effectiveness of the expert adaptor in enabling the model to adapt to new changes. In the absence of knowledge distillation (KD), the model performs poorly on old scenes. indicating that KD helps address the forgetting problem for the Continual Learning of NeRF.
Table 4: Ablation Study on Dataset Whiteroom. We study the importance the expert adaptor and distillation strategy.
ADD
DELETE
MOVE
REPLACE
SEQ
Operations
Old
Old
Old
New
Old
New
Old
New
w/o Expert
31.24
23.42
30.36
27.48
31.23
30.86
31.33
30.00
29.39
24.30
w/o KD
29.81
24.10
29.27
34.57
23.90
29.92
23.47
28.97
18.63
28.19
w/o Lon
31.08
24.92
30.99
31.73
31.90
28.64
32.63
28.53
30.70
28.05
Ours
32.33
25.29
32.43
34.77
33.30
29.97
33.33
29.82
33.38
29.74
Mask LOss. In order to demonstrate the necessity of mask loss in resolving conflict, we conduct an experiment with mask loss L.. removed. As shown Table 4 our findings suggest that mask loss I. also contributes to high-quality view syntheses. Without the guidance provided by the mask loss, the model may struggle to correctly predict object boundaries and often produce inaccurate results.
Conclusion
We have presented method CL-NeRF for efficiently adapting pre-trained NeRF changing envi- ronments. CL-NeRF comprises two major components: a lightweight expert adaptor for modeling new changes in evolved scenes, and conflict-aware knowledge distillation strategy for preserving memory in unaltered areas. These two designs collaborate, enabling our method to achieve superior results. To facilitate future research, we have also introduced continual learning benchmark for NeRF with comprehensive evaluation metrics. To the best of our knowledge, this is the first of its kind. We hope investigation and comprehensive benchmark inspire further research in this task, empowering NeRF to effectively model the ever-changing real-world scenes. More discussions about broader impacts, limitations including the scalability and other parameter-efficient tuning (PET) method are described in the supplementary file.
Acknowledgments
This work has been supported by Hong Kong Research Grant Council Early Career Scheme (Grant No. 27209621), General Research Fund Scheme (Grant No. 17202422), and RGC Matching Fund Scheme (RMGS), and Tencent Research Fund.
10

---

References
[1] Rahaf Aljundi. Francesca Babiloni, Mohamed Elhoseiny, Mareus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139-154, 2018.
[2] Jonathan Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the JEEF/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
[3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
[4] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. CVPR, 2023.
[5] Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00422 2018.
[6] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny. Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On tiny cpisodic memories in continual learning. arXiv preprint arXiv:1902 10486, 2019.
[7] Kangle Deng, Andrew Liu. Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. arXiv preprint arXiv:2107.022 107.02791. 2021.
[8] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers, 2022.
[9] Sara Fridovich-Keil Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. CVPR, 2022.
[10] Hang Gao, Ruilong Li. Shubham Tulsiani. Bryan Russell, and Angjoo Kanazawa. Dynamic novel-view synthesis: A reality check. In NeurlPS, 2022.
[11] Tyler L Hayes, Nathan D Cahill, and Christopher Kanan. Memory efficient experience replay for streaming leamiigg, In 2019 International Conference on Robotics and Automation (ICRA), pages 9769 9776. IEEE, 2019.
[12] James T Kajiya Brian P Von Herzen. Ray tracing volume densities. ACM SIGGRAPH computer graphics, 185))165-174, 1984.
[13] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barkkk---aiikkaaa, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences. 114(13):3521-3522 2017.
[14] Zhengfei Kuang, Fujun Luan, Sai Bi, Zhixin Shu, Gordon Wetzstein, and Kalyan Sunkavalli. Palettenerf: Palette-based appearance editing of neural radiance fields. arXiv preprint arXiv:2212. 22 10699, 2022.
[15] Verica Lazova, Vladimir Guzov, Kyle Olszewski, Sergey Tulyakov, and Gerard Pons-Moll. Control-nerf: Editable feature volumes for scene rendering and manipulation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4340-4350. 2023.
[16] Lingzhi Li, Zhen Shen, Li Shen, Ping Tan, et al. Streaming radiance fields for 3d video synthesis. In Advances in Neural Information Processing Systems.
[17] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5521-5531, 2022.
[18] Xilai Li. Yingbo Zhou, Tianfu Wu, Richard Socher. and Caiming Xiong. Learn to grow: continual structure learning framework for overcoming catastrophic forgetting. In International Conference on Machine Learning, pages 3925-3934, PMLR, 2019.
[19] Zhengqi Li. Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498 6508, 2021.

---

[20] Zhengqi Li. Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-base baagg--bsedddee rendering. arXiv preprint arXiv:2211.11082 2022.
[21] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 20012)22935- 2017.
[22] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. NewrlPS, 2020.
[23] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun- Yan Zhu, and Bryan Russell. Editing conditional radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5773 5783, 2021.
[24] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
[25] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pages 2018.
[26] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pages 405-424 4215.41..60 Springer, 2020.
[27] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding, ACM Trans. Graph., 41(4):102:1 July 2022.
[28] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[29] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Scitz. Hypernerf: A digees-ni representation for topologically varying neural radiance fields. ACM Trans. Graph., 40(6), dec 2021.
[30] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguee D-NeRF: Neural Radiance Fields for Dynamic Scenes. In Proceedings of the IEFE/CVF Conference Computer Vision and Partern Recognition, 2020.
[31] Albert Pumarola, Enric Corona, Gerard Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318-10327, 2021.
[32] Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell, Continual unsupervised representation learning. Advances in Neural Information Processing Systems. 32, 2019.
[33] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[34] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Ko- ray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.0m 04600000461...10 2016.
[35] Sara Sabour, Suhani Vora, Daniel Duckworth, Ivan Krasin, David J Fleet, and Andrea Tagliasacchi. Robustnerf: Ignoring distractors with robust losses. arXiv preprint arXiv:2302.0083 2023.
[36] Sara Fridovich-Keil and Giacomo Mcanti, Frederik Rahbak Warhurg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.
[37] Johannes L Schonberger and Jan-Michael Frahm. Streture-from-mooee revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104-4113, 2016.
[38] Johannes Schönberger, Enliang Zheng, Jan Michal Frahm, and Mare Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Computer Vision-ECC ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pages 501-518. Springer.
2016.
12

---

[39] Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphics, 2965)2732-2242,, 2023.
[40] Cheng Sun, Min Sun, and Hwann-TZae Tzong Chen. Direct voxel grid optimization: Super-fast convergence radiance fields reconstruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5459-5469, 2022.
[41] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8248 8258, June 2022.
[42] Sebastian Thrun and Sebastian Thrun. Explanation- based neural network learning. Springer, 1996.
[43] Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering, In The European Conference on Computer Vision (ECCV), 2022.
[44] Tianhan Xu and Tatsuya Harada. Deforming radiance fields with cages. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XXXIII, pages 159 175. Springer, 2022.
[45] Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint (rXiv:/708.015.. 2017.
[46] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference machine learning, pages 3987-3995. PMLR, 2017.
[47] Chengwei Zheng, Wenbin Lin, and Feng Xu. Editablenerf: Editing topologically varying neural radiance fields by key points, arXiv preprint arX1::2112.04247, 2022.
13


Reviews:
[{"review_id": "pzKB95iVlW", "reviewer": "Reviewer_QJef", "summary": "This work aims to tackle the challenge of efficiently adapting NeRFs to real-world scene changes in a continual learning setting. To achieve this, it develops two techniques, including an expert adaptor for adapting to new changes and a conflict-aware knowledge distillation scheme for memorizing unchanged parts. Experiments on self-collected datasets validate the superiority of the proposed method over other potential solutions for the continual learning of NeRF.", "questions": "I have listed my questions in the weakness section.", "limitations": "This work targets continual learning of NeRFs, which boosts the data efficiency of NeRF training and thus does not suffer obvious negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. As the first work targeting the continual learning of NeRF, this work could provide insights and references for the community;\n\n2. The proposed method can achieve notable improvements over other potential solutions for the continual learning of NeRF.", "weaknesses": "I have the following concerns about this work:\n\n1. The major concern is the limited technical contributions and novelty of the proposed framework. In particular, parameter-efficient tuning methods such as [1][2][3], which adapt parts of the pretrained model or newly added lightweight modules, have been widely adopted in the literature of continual learning and the proposed expert adaptor is an intuitive instantiation of these methods in the context of NeRF. And the conflict-aware knowledge distillation is more like a fine-grained memory replay thanks to the nature of 3D reconstruction tasks.\n\n[1] \"Dualprompt: Complementary prompting for rehearsal-free continual learning\", Z. Wang et al., ECCV 2022.\n\n[2] \"Learning to prompt for continual learning\", Z. Wang et al., CVPR 2022.\n\n[3] \"UFO: unified feature optimization\", T. Xi et al., ECCV 2022.\n\n2. It is not clear whether the proposed method can be generalized to other commonly adopted NeRF representations. For example, for voxel-based NeRFs like DVGO, the MLP part is small while most scene features are encoded in voxel embeddings. The authors are expected to discuss whether the proposed expert adaptor could also work in such scenarios.\n\n3. What are the differences among the old tasks? If they are all unaltered regions, is the only distinction the changed camera poses? In addition, experiments on more diverse scenes, containing diverse objects and environments, are highly desirable.\n\n4. One potential baseline is generalizable NeRF variants (e.g., IBRNet, MVSNeRF, and NeuRay), which can instantly render a new scene given a set of source views. The identified catastrophic forgetting issue can be considerably mitigated thanks to their cross-scene generalization capability even without any fine-tuning. The authors are expected to benchmark with this baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer QJef,\n\nThank you for appreciating our approach. We will address your comments below.\n\n**W1: Limited contribution and novelty, compared to [1,2,3].**\n\n1) Existing methods in continual learning primarily deal with image classification tasks that involve adding new classes incrementally. In such tasks, there is usually no clear conflict between the old and new tasks, and therefore, the main focus is on adapting to new tasks and avoiding forgetting. In contrast, our work focuses on studying this problem within the context of NeRF, where a scene's neural radiance field is continuously evolving. In addition to overcoming the forgetting issue and adapting to new content, our task requires the model to effectively handle conflicting supervisory signals that may arise due to the overlap of old and new scene areas.\n\n2) Our approach differs from methods like [1][2]. While these methods incorporate new trainable prompts for adapting to new tasks at the input level, we introduce new architecture designs that expand the network's capability to adapt to scene changes. Additionally, we have dedicated designs in place to handle conflicting supervisory signals. It is worth noting that the design of DyNeRF, in which we compared our approach, is similar to [1][2] in using trainable embeddings known as latent codes to capture time-specific information in the input. Our adaptor design demonstrates superiority over DyNeRF with or without Memory Replay (MR), as evidenced by the results shown in Table 2 and Figure 4 in the main paper, as well as Table 4 in the supplementary file.\n\n3) In comparison to the current MR method, our approach effectively addresses conflicts crucial to our problem that is overlooked by existing methods. By mitigating the adverse effects caused by conflicting supervisory signals, our design significantly improves performance. This is evident from the comparison in Table 2, where executing the DELETE operation in the old task yields a performance of 34.77, compared to 17.21 in the pure data MR scenario.\n\nIn summary, we believe that our proposed framework offers novel contributions to the field of NeRF, effectively addressing the challenges of adapting to scene changes and handling conflicting supervisory signals.\n\n**W2: Generalized to voxel-based NeRF (DVGO).**\n\nIn response to the concern about the generalizability of our proposed method to other commonly adopted NeRF representations, we are working on extending our approach to include voxel-based NeRFs like DVGO to demonstrate its adaptability. \n\nIn theory, it may also experience the forgetting phenomenon, as it first learns the old tasks and subsequently learns the new tasks with a different camera pose distribution. In such a scenario, our method could potentially be helpful in mitigating the negative impact. However, we regretfully cannot present the experimental results at this moment due to the considerable workload and time constraints for rebuttal. Nevertheless, we will promptly update and provide the experimental results during the discussion period.\n\nBesides, further experiments on diverse datasets in Table R1 highlight the general applicability of our proposed algorithm by demonstrating its robust performance in various environments.\n\n**W3: What are the differences among the old tasks? Contain more diverse scenes?**\n\n1) We aim to clarify our approach for evaluating the old task in the following manner: Initially, we design a camera pose set to ensure comprehensive coverage of the given scene. This set assists in training an original NeRF model to represent the entire scene accurately. Following an operation or a series of operations, we can identify the unchanged areas by sampling camera poses that do not capture the alterations. To illustrate, during the execution of an ADD operation (such as adding an apple), we eliminate camera poses from the set that captures the apple. Finally, we evaluate the performance of the old task using the remaining poses from before the operation is carried out. If they are all unaltered regions, then the old task is to assess the rendering quality of the pre-trained NeRF.\n\n2) We conduct experiments on two more challenging scenes (real-world indoor and outdoor scenes) containing various objects and environments to demonstrate the generalization ability of our algorithm. For the indoor scene, we capture images using a camera with LiDAR (specifically, an iPhone 14 Pro) and employ the Record3D App with ARKit to ensure accurate camera poses. On the other hand, the outdoor scene is captured using a DJI drone, focusing on a large-scale subject (the audience seats on a basketball court), with COLMAP utilized to calculate the camera poses. The results of these experiments are presented in Table R1 and Figure R1. We observe that Fine-tuning (FT) exhibits severe forgetting in old tasks, while Memory Replay (MR) underperforms in new tasks, particularly in qualitative results. These findings effectively showcase the robust performance of our methods across varying environments, further demonstrating the versatility and adaptability of our proposed algorithm.\n\n**W4: Compare with generalizable NeRF.**\n\nTo address this concern, we conduct comparative analyses with a generalizable NeRF called IBRNet, as detailed in Table R2 and Figure R2. Our tests, conducted with 10 new images, reveal unsatisfactory performance and serious artifacts in IBRNet's performance (denoted as MR1). Additionally, we conduct further testing on IBRNet, employing a more selective MR method involving manual removal of conflict data (denoted as MR1*). However, the obtained results remain unsatisfactory. These can be attributed to its heavy reliance on neighboring frames. To further evaluate IBRNet, we test it in different settings by varying the number of images (denoted as MR2, MR3). Although increasing the number of images improves IBRNet's performance, it still falls short compared to our model.\n\n"}, {"review_id": "aTgwQEmH0x", "reviewer": "Reviewer_fqk8", "summary": "The authors propose CL-NeRF, which tries to solve the problem of rendering scenes that evolve over time using a few images of the altered scene while retaining information about the unaltered regions. The proposed method contains two key components - 1. an expert adapter, to adapt to new regions, and 2. a conflict-aware knowledge distillation to preserve the original scene representation in the unchanged regions. The paper also introduces a new benchmark to evaluate NeRF-based methods on a continually evolving scene, particularly focusing on a few key operations including ADD, REMOVE, MOVE, and DELETE. The method claims strong adaptation to scene changes, requiring minimal images captured in the changed area while still ensuring high rendering quality in the unchanged regions. ", "questions": "- In the sequential setup, are new experts added to the network from the previous task? i.e at task-1 there are no experts, task-2: 1 expert is added, task-3: 2 experts are added, etc. \n- Is it necessary for the captured images in the later tasks to be local? i.e. only capturing the altered regions? What would happen if the captured images are more global and covered the entire scene?\n- In the current benchmark, every operation only manages to alter one area. It would be interesting to evaluate multiple altered regions and how the current method would scale in such cases. ", "limitations": "I don't see any negative social impact on their work. The authors have discussed limitations in the supplementary.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The proposed task is novel, i.e. given sufficient images at time t=0 to train a NeRF, we want to adapt the radiance field (at time t=t > 0) to new changes in the scene with as few images in the altered regions. \n- The paper is easy to read and follow. All the components in the proposed method are well-motivated. \n- The authors introduce a new continual learning benchmark for NeRFs, containing 3 scenes with 4 single-step operations and a sequence of them. The method is compared against sufficient baselines, outperforming them by a sufficient margin. ", "weaknesses": "- There are not many significant weaknesses, however, some components of the method e.g. knowledge distillation to prevent forgetting have been seen in some form (without being conflict aware) in [1]. This does not decrease the novelty of the current work, but it would be interesting to evaluate these methods in the proposed benchmark. \n- It would also be interesting to see how this method would compare against a Generalizable NeRF baseline that uses the memory replay of the previous task (perhaps a bit more selective to not cover the altered regions) and the new training images captured in the altered region. \n\n1. MEIL-NeRF - Memory Efficient Increment Learning of Neural Radiance Fields. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer fqk8,\n\nThank you for appreciating our approach. We will address your comments below.\n\n**W1: Evaluate [1] in the proposed benchmark.**\n\nThank you for the insightful advice. Unfortunately, due to the limited rebuttal time and the unavailability of MEIL-NeRF's [1] code, it is challenging for us to reproduce their work and provide a comparison in this response. Nevertheless, we appreciate the suggestion and will include a comparison result in the final version of our paper. It is worth noting that MEIL-NeRF [1] is designed for novel synthesis tasks involving static scene stream data, whereas our approach addresses the efficient adaptation of NeRFs to dynamic real-world scene changes using minimal new images. Furthermore, our method emphasizes preserving unaltered areas, adapting to new scenes, and resolving conflicts due to scene dynamics through continual learning, which are not addressed in [1].\n\n**W2: Compare against a Generalizable NeRF.**\n\nTo address this concern, we conduct comparative analyses with a generalizable NeRF called IBRNet, as detailed in Table R2 and Figure R2. Our tests, conducted with 10 new images, reveal unsatisfactory performance and serious artifacts in IBRNet's performance (denoted as MR1). Additionally, we conduct further testing on IBRNet, employing a more selective MR method involving manual removal of conflict data (denoted as MR1*). However, the obtained results remain unsatisfactory. These can be attributed to its heavy reliance on neighboring frames. To further evaluate IBRNet, we test it in different settings by varying the number of images (denoted as MR2, MR3). Although increasing the number of images improves IBRNet's performance, it still falls short compared to our model.\n\n**Q1: Are new experts added to the network from the previous task?**\n\nWe would like to clarify that we employ only one expert throughout our process. For task-1, no experts are used. At task-2, one expert is added. For task-3 and subsequent tasks, we continue with just one expert, utilizing the original model combined with the expert from the previous task to distill knowledge into the current expert. This approach allows our method to efficiently adapt to new tasks while preserving knowledge from previous tasks without the need to add multiple experts.\n\nIn our current exploration, this strategy is sufficient to achieve reasonable performance (see Table 1-3 in the main paper) and outperform baseline methods without increasing model complexities. We believe that adding one expert at a time may further boost performance, albeit at the cost of increased model complexity. We will study the trade-off between model complexities (adaptor numbers) and performance in the final version of our paper.\n\n**Q2: Is it necessary for the captured images in the later tasks to be local?**\n\nYes, we opt for a local view approach in our training and evaluation for several reasons:\n\n1) Changes within a scene typically occur in localized areas. Capturing images of altered regions helps reduce the number of images used for adapting to new changes. \n\n2) This setting allows for a more accurate assessment of performance gains and progress, as the PSNR is calculated at the pixel level.\n\nOur approach can also adapt to images that cover global views. This is because when the captured new images contain global views, the predicted mask logit makes the adaptor primarily learn the difference, and our proposed conflict-aware knowledge distillation mechanism ensures that altered areas contained in our previously captured images do not damage the training. This enhances the robustness of our approach to scenes with global views, as these may have more conflict regions.\n\n**Q3: How the current method would scale in multiple altered regions?**\n\nTo showcase the effectiveness of our method, we have developed a benchmark where 10 objects are simultaneously added and distributed across various locations within the room. The results, presented in Table R3 and Figure R3, illustrate that our method performs admirably even in challenging scenarios, achieving 32.31 and 32.15 for the old and new tasks, respectively.\n\n\n"}, {"review_id": "PQMHp5pcoe", "reviewer": "Reviewer_f2t1", "summary": "This paper proposes a challenge of how to reduce the data and time cost of retraining NeRF when the scene changes. To this end, the paper proposes two key components: a trainable network for the changing part of the scene, and a conflict-aware network for the unchanged part of the scene. With these two components, CL-NeRF achieves high training efficiency for updating the changing part of the scene while preserving the unchanged part well. This leads to the proposal of a new benchmark based on this challenge.", "questions": "* It is possible that the poor performance of DyNeRF in scene representation is due to the use of MR methods. If this is the case, how will the performance of DyNeRF be when using only the images of the unchanged parts of the scene and the images of the changing parts separately?\n* The experiment only includes a few scenes, all of which are synthetic. There are no experiments with real scenes. It would be beneficial to include experiments with more scenes in future studies.", "limitations": "The authors have addressed limitations in the supplementary material.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* When there are changes in the scene, such as adding or removing objects, CL-NeRF can update the previously modeled results with very few photos and remember the unchanged parts.\n* By using a lightweight expert adapter to predict masks, CL-NeRF can perceive which parts have been changed and only modify the features related to the location of the scene change. The self-distillation mechanism that is aware of conflicts can reduce conflicts in the supervised signals.\n* A new dataset has been established to evaluate under the condition of scene changes.", "weaknesses": "* The number of scenes in the experiment is not large and all of them are synthetic. However, this is understandable due to limitations in time and equipment. It is highly recommended to evaluate the method on more real scenes.\n* Training DyNeRF using Memory Replay (MR) methods and simultaneously using photos with both changed and unchanged scenes may lead to ambiguity, which may lead to unfair comparisons. A better MR method may be partitioning the data based on whether the region has changed or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer f2t1,\n\nThank you for appreciating our approach. We will address your comments below.\n\n**W1: Evaluate the method on more real scenes.**\n\nWe conduct experiments on two more challenging scenes (real-world indoor and outdoor scenes) containing various objects and environments to demonstrate the generalization ability of our algorithm. \n\nFor the indoor scene, we capture images using a camera with LiDAR (specifically, an iPhone 14 Pro) and employ the Record3D App with ARKit to ensure accurate camera poses. On the other hand, the outdoor scene is captured using a DJI drone, focusing on a large-scale subject (the audience seats on a basketball court), with COLMAP utilized to calculate the camera poses. The results of these experiments are presented in Table R1 and Figure R1. We observe that Fine-tuning (FT) exhibits severe forgetting in old tasks, while Memory Replay (MR) underperforms in new tasks, particularly in qualitative results. These findings effectively showcase the robust performance of our methods across varying environments, further demonstrating the versatility and adaptability of our proposed algorithm.\n\n**W2: Training DyNeRF using a better MR method.**\n\n1) It is important to note that the original DyNeRF does not include a memory replay (MR) operation as our method does. DyNeRF performs poorly without any MR, as discussed in section B.4 of the supplementary materials. Therefore, in Table 2 of the main paper, we compare DyNeRF with a robust MR method that retains all previous-stage data.\n\n2) Following your suggestion, we also explore more accurate MR methods on DyNeRF by manually excluding the conflict views, with results presented in Table R4. MR1 represents the original MR method from our main paper, while MR2 excludes frames with conflict regions in the old task, which have been manually removed. This manual selection effectively avoids conflict, enhancing new task performance but reducing the original image count, thus weakening old task performance.\n\n3) Our method demonstrates superior performance compared to DyNeRF with various MR strategies. Importantly, our approach does not rely on using any previously captured images for training, whereas DyNeRF with the augmented MR method requires storing old images from all previous stages. This distinction highlights the efficiency and effectiveness of our proposed method in handling changing scenes without the need for extensive storage and manual intervention. \n\n**Q1: How will the performance of DyNeRF be when using data without conflict?**\n\nPlease refer to the question **W2: training DyNeRF using a better MR method**.\n\n**Q2: It would be beneficial to include experiments with more scenes.**\n\nPlease refer to the question **W1: evaluate the method on more real scenes**."}, {"review_id": "Kvnc2PHENG", "reviewer": "Reviewer_tKPz", "summary": "This paper tackles the task of continuing learning of NeRF, which aims to adapting NeRFs to real-world scene changes over time using a few new images. To prevent the forgetting problem during adapting, the authors propose CL-NeRF. The CL-NeRF consists of two key components: an expert adaptor for adapting to scene changes and a conflict-aware distillation scheme for memorising unchanged parts. The expert adaptor learns to encode local changes  and can be learned from just a few new images. The conflict-aware distillation scheme is designed to preserve the original scene representation for unchanged areas by via student-teacher knowledge distillation. Moreover, A new continual learning benchmark for NeRF is introduced to evaluating the proposed method.", "questions": "1.\tWhat is the performance gap between the proposed method and the scenario when new data is abundant? The latter scenario is upper bound for the proposed task.\n2.\tHow many testing data are used for each operation? Take the ‘ADD’ operation for example, we can add different objects to different scenes. Is the result based on just one or multiple scenarios?", "limitations": "The limitations and potential negative societal impact are resolved in the supplementary material.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper tackles a practical task, which aims to adapt NeRFs to real-world scene changes over time with minimal data.\n2. The paper is well written and easy to follow.\n3. New datasets and evaluation metrics are proposed for the introduced task.\n", "weaknesses": "1.\tThe experimental results are not very well represented. For example, there is no text explanation for Table 1.  What does the ‘SEQ’ represent in Table 2 since there are also results for sequential operation in Table 3. The quantitative results are shown without clear text illustration.\n2. Follow the previous comment, the qualitative results in Figure 4 are not very intuitive.  For example, the new and old scenes are very different after the adding operation in the first column. From my understanding, it should be only adding the apple to the old scene?  Similar for the Move and Replace operation, the new and old scenes seem very different. \n3.\tThe predicted mask plays an important role in detecting changing area, it would be better to show some qualitative or quantitative results on how well the mask logits are predicted.\n4. The scale of the network will increase when more operation are added since the expert adaptor is scene-specific.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer tKPz,\n\nThank you for appreciating our approach. We will address your comments below.\n\n**W1: Experimental results are not very well represented.**\n\n1) We regret the omission of the analysis for Table 1 and appreciate your comments. Table 1 uses the Backward Transfer Metric (BTM) and Forgetting Metric (FM), which are both employed to quantify the forgetting issue. BTM quantifies the level of forgetting by measuring the cumulative gap between the original model performance ($PSNR_{i,i}$) and the final performance ($PSNR_{T,i}$), whereas FM directly measures actual performance after T operations. Details are explained in Section 4.2 (evaluation metrics) in our paper.\n\n2) SEQ stands for sequential operation, as defined in Section 4.2. In both Tables 2 and 3, 'SEQ' refers to the same operation. However, we assess this operation from two different perspectives. In Table 2, we evaluate the performance (i.e., PSNR) of the final model after a series of operations, while Table 3 assesses the performance at each stage. This distinction provides a more comprehensive understanding of our method's performance throughout the entire process.\n\n**W2: Qualitative results in Figure 4 are not very intuitive.**\n\nYes, only adding an apple to the old scene. The difference in qualitative results comes from the different camera views. The new scene images are rendered from camera poses that reflect scene changes (i.e., object-centric camera pose trajectory), while the old scene images are rendered from camera poses where the scene remains unchanged. They belong to different areas; thus, we render them to depict the effective adaptation of new scenes and the preservation of unchanged areas. We sincerely apologize for any confusion and will clarify this point in the final version of our paper.\n\n**W3: How well the mask logits are predicted?**\n\nWe have included a visualization of the predicted mask in Figure R4 to demonstrate its quality. A pixel value approaching 1 (white) indicates a high probability that the pixel belongs to an altered region, whereas a lower value suggests an unaltered region. This visualization offers a qualitative insight into the effectiveness of our mask logits in detecting changes within the scene.\n\n**W4: The scale of the network will increase when more operation are added.**\n\nWe would like to clarify that we employ only one expert throughout our process. For task-1, no experts are used. At task-2, one expert is added. For task-3 and subsequent tasks, we continue with just one expert, utilizing the original model combined with the expert from the previous task to distill knowledge into the current expert. This approach allows our method to efficiently adapt to new tasks while preserving knowledge from previous tasks without the need to add multiple experts.\n\nIn our current exploration, this strategy is sufficient to achieve reasonable performance (see Table 1-3 in the main paper) and outperform baseline methods without increasing model complexities. We believe that adding one expert at a time may further boost performance, albeit at the cost of increased model complexity. We will study the trade-off between model complexities (adaptor numbers) and performance in the final version of our paper.\n\n**Q1: What is the performance gap when new data is abundant?**\n\nTo demonstrate the upper bound of our proposed task, Figure R5 shows the relationship between performance and the number of images, specifically for the ADD operation on the Whitehouse dataset. The figure reveals that while increasing the number of training images does enhance performance, the improvement is relatively minor.\n\n**Q2: How many testing data for each operation? Add different objects to different scenes? Is the result based on just one or multiple scenarios?**\n\n1) For synthetic datasets such as Whitehouse, Kitchen, and Rome, the testing data comprises approximately 20 images. \n2) In our study, we primarily focus on basic operations involving a single object. Additionally, we have conducted tests with multiple objects added simultaneously, and the results are presented in Table R3 and Figure R3.\n3) Results in each table correspond to each specific scenario, however we encompass the tests across several synthetic and real-world datasets."}, {"review_id": "mg0qqE3m9b", "reviewer": "Reviewer_Fafa", "summary": "The paper introduces CL-NeRF, an approach for efficiently adapting Neural Radiance Fields (NeRFs) to real-world scene changes over time. CL-NeRF focuses on continual learning and requires only a few new images to adapt to changes while retaining the memory of unaltered areas. It consists of two main components: a lightweight expert adaptor for adapting to new changes and evolving scene representations, and a conflict-aware knowledge distillation learning objective for memorizing unchanged parts. The authors also propose a new benchmark with comprehensive metrics to evaluate Continual Learning of NeRFs.", "questions": "1. Why do we need to assume that the training data is not available? The training images themselves are already memorized by NeRFs.\n2. Why not just compare the image before change and after change?", "limitations": "The problem setting is problematic.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The inclusion of a benchmark is a valuable contribution as it provides a standardized evaluation framework for assessing the performance of continual learning methods applied to NeRFs. \n2. The provision of rendered data sets generated by CL-NeRF is beneficial for the research community.", "weaknesses": "The paper's weakness lies in the assumption that continual learning is necessary for NeRFs and the relevance of this assumption to the problem at hand. NeRFs have distinct characteristics that differentiate them from traditional machine learning problems. Firstly, NeRFs do not typically require a vast amount of training data, as they focus on scene-specific recon rather than generalization (i.e., the plain NeRFs studied in the paper, not SRN or pixelNeRF). One scene -- one NeRF.\nAdditionally, NeRFs do not necessarily face the issue of inaccessible training data, as the training data can be obtained by rendering images of the scene. Therefore, the need for continual learning, which assumes the unavailability of training data, may not be applicable to NeRFs.\n\nFurthermore, the paper does not adequately address a simple baseline approach __for the scene change challenge__. By rendering images for the new given poses, it becomes straightforward to identify areas where the scene changes occur. This straightforward method eliminates the need for complex continual learning techniques in this specific scenario.\n\n\n---\n\nAfter rebuttal:\n\nI have updated my score to be borderline accept. The authors' response regarding large-scale tuning of a trained NeRF model has convinced me of the value of their proposed method. While the experiments do not directly validate performance in the large-scale scenario, the current results demonstrate promising capabilities that could benefit future research in this direction. All other reviewers value the contribution of the continual learning nerf method in this paper.\n\nHowever, I still have some comments about the writing of the paper to avoid potentially misleading readers on the concept of continual learning:\n\n1. The mentioning of \"catastrophic forgetting\" may require more explanation. Most readers likely associate this problem with the paper \"Overcoming catastrophic forgetting in neural networks,\" [1] which highlighted it as a serious challenge for sequential learning tasks. But NeRF does not assume sequential learning, since all previous data is stored. The authors could emphasize that even with full data retention, fine-tuning can still be expensive without a continual learning scheme.\n\n2. It may be helpful to include some discussion of efficient network tuning methods as motivation, like the well-known LoRA scheme. Although this paper uses MLPs, parameter-efficient tuning techniques for other networks like transformers are highly relevant to the problem setting.\n\n[1] Kirkpatrick, James, et al. \"Overcoming catastrophic forgetting in neural networks.\" Proceedings of the national academy of sciences 114.13 (2017): 3521-3526.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Fafa,\n\nThank you for appreciating our approach. We will address your comments below.\n\n**W1: The assumption that continual learning is necessary for NeRFs and the relevance of this assumption to the problem.**\n\nThank you for the comment. We understand your concerns regarding continual learning for NeRFs. However, we believe that there are potential scenarios where continual learning is urgently needed:\n\n1) For large-scale scene applications, like city-level scene reconstruction/novel view synthesis, capturing images can be cumbersome and require expensive devices such as drones to capture the entire city. Thus, a setting that requires fewer images for training would alleviate the burden of data capture.\n\n2) As changes may occur frequently, it would be time-consuming to gather training data and retrain the model after each change. For example, if there are ten scene changes, capturing the entire scene ten times is prohibitively expensive. The burden is further heightened by the need for camera calibration and pose estimation, especially when dealing with a large number of images.\n\n3) Our investigation may also be useful for resource-constrained platforms, such as smartphones, where storage resources are an important consideration. This issue is bypassed by our continual learning formulation, as we only need to retain the few images that reflect scene changes.\n\nMoreover, the primary challenge that our study addresses lies in adapting to changes without encountering conflicting supervisory signals, rather than whether we can use information from a pre-trained NeRF or the amount of training data available. Even when the data are available, our designs, especially the adaptor and conflict-aware mechanism, are still useful.\n\nWe specifically address the challenge of learning new content with conflict in our paper. By using our method, we can adapt to changes and maintain the integrity of scene representations, enabling us to adjust to scene changes with minimal effort (i.e., 10-20 training images and 10-20 minute training time).\n\n**W2: Does not adequately address a simple baseline.**\n\nIn fact, we have employed the straightforward method you mentioned, as detailed in Section 4.1, to identify and generate the pseudo mask for supervising the training of the mask logit. However, the main objective of our continual learning task is to learn a new neural radiance field from the new data that captures scene changes.\n\nSimply identifying areas where scene changes occur does not address the problem of adapting the neural radiance field to the new scene content while maintaining unaltered scene representation. Our proposed continual learning techniques aim to provide a more effective solution to this challenge by leveraging the newly captured images and minimizing the forgetting of previously learned content.\n\n**Q1: Why do we need to assume that the training data is not available?**\n\nPlease refer to the question **W1: The assumption that continual learning is necessary for NeRFs and the relevance of this assumption to the problem**.\n\n**Q2: Why not just compare the image before change and after change?**\n\nPlease refer to the question **W2: Does not adequately address a simple baseline**."}]

Author rebuttal:
Dear Reviewers and ACs:

Thank you so much for your time and efforts in assessing our paper. Hope our rebuttal has addressed your concerns. We are happy to discuss with you further if you still have other concerns. Thanks for helping improve our paper.

Best regards, Paper 1565 Authors

Meta-review: Reviewers agree that the paper considers a practically significant problem and presents novel solutions. All reviewers agree for acceptance.
Decision: Accept (poster)