SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiuzhe Wu
Yang Wu†
Peng Dai\*
Yan-Pei Cao
Weipeng Deng\*
Ying Shan†
Handi Chen
Xiaojuan Qi
$^{\diamond}$ The University of Hong Kong
†Tencent AI Lab  $\ddagger$ ARC Lab, Tencent PCG
Existing methods for adapting Neural Radiance Fields (NeRFs) to scene changes require extensive data capture and model retraining, which is both time-consuming and labor-intensive. In this paper, we tackle the challenge of efficiently adapting NeRFs to real-world scene changes over time using a few new images while retaining the memory of unaltered areas, focusing on the continual learning aspect of NeRFs. To this end, we propose CL-NeRF, which consists of two key components: a lightweight expert adaptor for adapting to new changes and evolving scene representations and a conflict-aware knowledge distillation learning objective for memorizing unchanged parts. We also present a new benchmark for evaluating Continual Learning of NeRFs with comprehensive metrics. Our extensive experiments demonstrate that CL-NeRF can synthesize high-quality novel views of both changed and unchanged regions with high training efficiency, surpassing existing methods in terms of reducing forgetting and adapting to changes. Code and benchmark will be made available.
Neural Radiance Fields (NeRFs), powerful models capable of synthesizing high-fidelity novel views from multi-view images [2, 26, 7], play crucial roles in various applications including AR/VR and autonomous driving simulators. However, real-world scenes constantly evolve over time, such as a chair being removed from a room. To handle such changes, it is essential to develop adaptable representations that can be continuously updated to reflect these changes accurately with minimal data and computational cost. However, existing NeRF methods [2, 26, 7, 43, 41, 40] often require a lot of new data and full retraining of the model to adapt to even minor local changes, which is impractical and costly, especially for large-scale scenes e.g., city-scale scene.
In this paper, we study a novel efficient continual learning task for NeRF, where we aim to update a scene's NeRF using only a few new images (i.e., 10-20 images) that capture the scene changes over time while preserving the memory of the unchanged areas (see Figure 1). A simple approach is to fine-tune the NeRF with the new images, but this suffers from catastrophic forgetting of the unaltered areas (see Figure 1). One might argue that using explicit 3D representations, [9, 22, 27], such as Plenoxels [9], could mitigate this problem since the explicit 3D representation might not change for the unchanged regions during fine-tuning. Nonetheless, our initial experiments suggest that forgetting still occurs (refer to Figure 1). This is because the ray-based volumetric rendering formula [2, 26, 9, 22, 27] propagates supervision signals to all voxels along a ray, which may affect other regions that share voxels with the same ray. The most related work is dynamic NeRFs [17, 16, 30], which also model dynamic changes using time-aware latent codes [17] or scene flows [19]. But it focuses on changes
Figure 1: Catastrophic forgetting (left) and scene changes simulation (right). The left part illustrates the forgetting behavior of MLP-based and Voxel-based NeRF during fine-tuning, while the right side simulates representative real-world change operations.
caused by continuous object motions and is designed to remember the observed regions at a specific timestamp, leading to forgetting of unobserved regions in a large-scale scene.
To address this challenge, we introduce Continual Learning NeRF (CL-NeRF), which efficiently adapts a learned NeRF model to environmental changes using minimal new data while retaining its memory of unchanged areas. It consists of two core components: an expert adaptor for adapting to scene changes and a conflict-aware distillation scheme for memorizing unchanged parts. The expert adaptor is a lightweight network that learns to encode local changes from a few new images while keeping the original NeRF parameters fixed. It also predicts a weight value called mask logit that indicates the likelihood of the rendering point being in the changed areas. It is used to fuse the original NeRF features and the expert adaptor for rendering. Further, the conflict-aware knowledge distillation scheme trains the model to preserve the original scene representation for unchanged areas by minimizing the discrepancy between the outputs of the original NeRF and the adapted model. The mask logit predicted by the expert network balances the distillation loss and rendering loss of new images, avoiding conflicting supervision signals.
Finally, we introduce a new continual learning benchmark for NeRF, consisting of synthetic and real-world scenes. Our benchmark encompasses four fundamental operations that represent prevalent real-world scene changes: add, remove, move, and replace. Furthermore, to facilitate evaluation, we develop comprehensive metrics to assess the model's effectiveness in preserving unaltered areas and adapting to new changes in terms of accuracy and efficiency. Our extensive experiments, involving three baselines (i.e., fine-tuning, memory replay, and DyNeRF[17]), demonstrate that our model consistently outperforms all compared approaches on all evaluated scenarios while maintaining superior efficiency. Especially, our approach maintains high-quality results throughout a series of operations in both old and new tasks as evidenced by our PSNR scores (29.74 and 33.38 in the new and old task, respectively.) Notably, our approach effectively mitigates the forgetting phenomenon when compared to direct fine-tuning (see Figure 1).
1) To our best knowledge, we present the first study on Continual Learning of NeRF, aiming to enable a NeRF model to adapt to changes in real-world scenes. We also introduce a novel benchmark for this task, which can serve as a valuable resource for future research.
2) We propose a new method - CL-NeRF - consisting of two key components: a lightweight expert adaptor for adapting to new changes and a conflict-aware knowledge-distillation objective for preserving unaltered scene representations.
3) We conduct extensive experiments and show that our model achieves high-quality view synthesis in both altered and unaltered regions, outperforming the baseline methods and mitigating the forgetting phenomenon.
Neural Radiance Fields, Dynamic NeRF and NeRF Editing Neural Radiance Fields (NeRF) [26] have drawn a lot of attention recently due to their high-fidelity rendering quality. To render novel-view images, previous methods encode the geometry and appearance into the weights of MLPs [26, 2, 3] or other explicit neural 3D representations [22, 9], then render novel-view images through volume
rendering [12] based on the predicted radiance values of query points. However, most of these methods require per-scene optimization and are fragile to the scene changes [35].
To handle scene changes, the vanilla NeRF [26] is modified to model the dynamic objects in the scene. This line of work usually takes videos containing dynamic objects as inputs and can be divided into two lines: deformation field estimation [19, 20, 28, 31, 39, 29, 8] and the incorporation of time-aware inputs [17, 4, 36]. Dycheck [10] offers a critical assessment of recent developments in dynamic NeRFs. Given that the dynamic NeRFs are only trained on limited observations of the large-scale scene at each timestamp, thus the high-fidelity rendering at that timestamp is constrained to the observed regions. Different from dynamic NeRFs, we aim to render high-fidelity images of the entire scene at any given timestamp.
The recent progress made in enabling NeRFs to support geometry or appearance editing [23, 14, 15, 47] has been quite significant recently. They focus on editing objects within the scene, for example, changing the color [23, 14], editing the parts of objects [23] or making deformations [15, 44]. This line of work usually handles editing following the user's instruction and is limited to existing objects within the scene while our goal is to adapt NeRFs to various real-world scene changes efficiently.
Figure 2: Illustration of the Continual Learning of NeRF.
Continual Learning Continual Learning aims to learn different tasks progressively[42, 33]. However, catastrophic forgetting is the key challenge, meaning the model will have performance degradation on previous tasks after training on new task[13]. In our task, the Continual Learning of NeRF is defined as a
model that should not have artifacts or quality degradation on unaltered areas after the model has adapted to the scene changes multiple times. (See Figure 2). To tackle the forgetting problem in continual learning, some researchers propose memory replay method[5, 6, 11], which will store the previous training data and use them to augment the current training set while training the model on the new task. Besides, adding regularization constraints to the neural network to limit the parameter updated also proves to be an effective solution[1, 13, 21, 46]. Lastly, Adding separate module to handle different tasks also draw some attentions[18, 32, 34, 45, 25].
3 Continual Learning of NeRF
Our study focuses on continual learning for scene-level NeRF, where the goal is to update a pre-trained scene-level NeRF efficiently by utilizing a few new images while retaining the memory of unchanged areas. Figure 2 provides an illustration of the process. In this section, we will first formulate the Continual Learning task of NeRF and subsequently describe our preliminary study which highlights the forgetting issue and motivates our method.
We represent a 3D scene as a 5D radiance function, denoted by  $F_{\theta}:(x,d)\to (c,\sigma)$ , in accordance with the vanilla NeRF [26]. Specifically, our approach utilizes an MLP  $F_{\theta}$ , where  $\theta$  represents a set of parameters, to predict the volume density  $\sigma$  and color  $c$  according to each sample point  $x$  and its corresponding view direction  $d$ . NeRF has demonstrated a high capacity for high-quality novel view synthesis. Our focus is on the Continual Learning of NeRF to adapt to changes while maintaining the unaltered components.
To simulate natural scene changes, we incorporate basic operations, denoted as  $O_{p}$ , including adding, removing, moving, and replacing objects, which are common factors contributing to scene alterations. To emulate these scene changes over time, we simulate a series of operations  $[O_{p_1}, O_{p_2}, \dots, O_{p_n}]$ , representing time-varying changes that result in evolved scenes  $[S_{u_1}, S_{u_2}, \dots, S_{u_n}]$ , respectively. Our simulation aims to capture the following characteristics of real-world scenes: 1) localized changes occurring within objects, 2) the impracticality of extensively recapturing scene-level data in large-scale environments, such as rooms or cities, due to time and cost constraints, and 3) the dynamic evolution of real-world scenes over time.
Figure 3: An overall inference framework of CL-NeRF. The position encoding  $\mathrm{PE}(x)$  is fed into both the original MLP backbone and the proposed lightweight expert adaptor module (the green dashed box), generating backbone features  $f_{\theta}(x)$  and expert features  $e_{\theta}(x)$ . A scalar output  $m$  by the expert is utilized for feature fusion. Finally, the fused feature  $f_{\theta}^{*}(x)$  is passed to the volume density and color prediction heads for further volumetric rendering.
Given the changes in scenes over time, our continual learning task is defined as follows: initially, we train a model  $F_{\theta}$  with  $m$  multi-view images of an original scene  $S_{o}$ . Subsequently, we progressively update  $F_{\theta}$  using  $k$  new multi-view images that capture scene changes  $S_{u_i}$  after operation  $O_{p_i}$  at each time step  $i$ . It is worth noting that the new training dataset is significantly smaller compared to the original one (i.e.,  $k \ll m$ ).
3.2 Preliminary Study and Motivations
Preliminary Study A straightforward approach to the continual learning task is to directly fine-tune an existing NeRF model with newly acquired data at each instance. We evaluate this baseline using a simple single-operation preliminary experiment. Specifically, as shown in Figure 1 the room-level scene comprises several pieces of furniture. We capture around 200-300 images for the scene to train the original model  $F_{\theta}$ . Then, we modify the scene with the ADD operation (i.e., placing several apples on the table) and take 10 images around the object. Finally, we use the new data to fine-tune  $F_{\theta}$  as a baseline.
Catastrophic Forgetting However, this simple strategy leads to unsatisfactory performance. While achieving acceptable quality for the updated region after the change, the model results in a catastrophic deterioration of the quality in unaltered regions. As illustrated in Figure 1, after fine-tuning with the new data, artifacts appear on the bookshelf, which is on the unchanged area located on the right-hand side of the table. The average PSNR dramatically drops from 32.71 to 24.21. We conjecture that NeRF stores geometry and appearance information in the weights of its MLP model in a coupled and implicit manner. Consequently, applying changes locally without affecting other regions is difficult. This issue presents a significant obstacle in adapting pre-trained NeRF models to changing real-world scenes while preserving unaltered areas. The problem is particularly pronounced in large-scale scenes, making recapturing and full retraining impractical in many cases.
Comparisons with Traditional Continual Learning Task In practical settings, continually capturing images and retraining a NeRF after every scene change is ideal but unfeasible for large, dynamic scenes. Instead, we propose a more realistic method: fine-tuning existing NeRFs with recent images consistently, which resembles traditional Continual Learning (CL) tasks. This technique, however, can induce artifacts and catastrophic forgetting in previously trained areas, as noted in traditional CL tasks [13]. The divergence between CL in NeRF and traditional CL tasks lies in the inherent conflict between new and old data. While traditional CL tasks maintain a stable one-to-one input-output correspondence, NeRF attributes colors and densities to specific points. Scene alterations can modify these associations, leading to conflicting supervisory signals. Standard CL solutions like Memory Replay (MR) prove insufficient in NeRF due to these inconsistencies, particularly in changed regions. To navigate these challenges, our study introduces a novel method in Section 4.
4 Our Method
In this section, we introduce our novel approach, CL-NeRF, for adapting a pre-trained NeRF to scene changes over time. The pipeline comprises two major components: a lightweight expert adaptor specifically designed to adapt to new scene changes and a conflict-aware knowledge distillation
training objective to preserve the memory in unaltered areas (see Section 4.1). As shown in Figure 3, the overall framework involves a pre-trained  $F_{\theta}$  for modeling the entire scene and a small set of new training images  $I$  that capture local scene changes. The lightweight trainable expert adaptor, represented as  $E_{\theta}$ , generates a feature for capturing scene changes which is further fused with features from the original  $F_{\theta}$  using a masking mechanism. To prevent forgetting, we introduce a conflict-aware self-distillation mechanism (as illustrated in Figure 3) that distills knowledge from  $F_{\theta}$ , particularly in unaltered areas. Moreover, we propose a continual learning benchmark for NeRF, which includes various operational definitions and metrics to assess the overall neural rendering performance in adapting to changes and preserving unchanged content (see Section 4.2).
4.1 CL-NeRF
Lightweight Expert Adaptor Our approach is motivated by the observation that scene changes primarily occur in localized regions, fine-tuning all the model parameters resulting in forgetting in unaltered regions. To address the above issue, we introduce a lightweight expert adaptor, denoted as  $E_{\theta}$ , which can be seamlessly integrated with the original NeRF model, represented by  $F_{\theta}$ , as a plug-and-play module. In addition to encoding information about newly modified areas of the scene,  $E_{\theta}$  also provides a rollback strategy that enables users to revert to any desired timestamp. By contrast,  $F_{\theta}$  is responsible for storing information about the overall original scene. These two components offer a comprehensive solution for modeling dynamic scenes with high fidelity.
To achieve our objectives, we define the input and output of  $E_{\theta}$  in the same manner as for  $F_{\theta}$ . Specifically,  $E_{\theta}$  takes the same input as the backbone  $F_{\theta}$  and generates two outputs: a mask logit  $m$ , and a feature vector  $f_{o}$ , which is subsequently integrated into the original  $F_{\theta}$ . The mask logit  $m$  takes on a gating role in the feature fusion module, whereas the feature output by  $F_{\theta}$  is fused with the output of  $E_{\theta}$ , as illustrated in Figure 3. This approach enables us to capture dynamic changes in the scene with  $E_{\theta}$  while preserving the overall structure of the original scene with  $F_{\theta}$ .
Concretely, each sampling point  $x$  is fed to the original network  $F_{\theta}$ , which generates a feature  $f_{\theta}(x)$  representing previous contents. Simultaneously,  $E_{\theta}$  predicts a residual feature vector for  $x$ , represented as  $e_{\theta}(x)$ . The final fused feature  $f_{\theta}^{*}(x)$  can be calculated as
$$
f _ {\theta} ^ {*} (x) = f _ {\theta} (x) + m \cdot e _ {\theta} (x). \tag {1}
$$
Then,  $f_{\theta}^{*}(x)$  is fed back into the remaining component of  $F_{\theta}$  to further predict volume-density  $\sigma$  and view-dependent color  $c$  for neural rendering. As  $m$  approaches 0, the output features should remain as close as possible to the original features. When  $m$  approaches 1, the predicted features from  $E_{\theta}$  will be most effective in predicting modifications made to new areas, also such a design encourages the model not to change the origin scene.
Conflict-Aware Self-Distillation Mechanism The student model, responsible for adapting to a new scenario, is initialized with prior knowledge and fine-tuned to incorporate relevant information from the previous scene. To prevent catastrophic forgetting, we designate the previously trained original model in the preceding scene as the teacher model. Through self-distillation, we transfer scene-related information from the teacher model to the student model. To be specific, we randomly sample camera views and points in space and feed them into both the teacher model and the student model. The teacher model's prediction is used to supervise the student model.
However, the supervision signals provided by the images captured in new camera views can conflict with the supervision from the teacher model. To this end, we propose conflict-aware knowledge distillation. Specifically, for each sampling point, our expert adaptor has already predicted a mask logit. This allows us to utilize volume rendering in the same way as image prediction to render a soft mask map  $\hat{M}$ . This soft mask map can serve as a probability map indicating the likelihood of a pixel belonging to the changed area. A value close to 1 signifies a high probability of the pixel belonging to the conflict region, in which case the self-distillation mechanism need not be executed. Conversely, when there is a higher probability of the pixel belonging to the unaltered region, calculating an additional knowledge distillation loss to make the supervision signals more accurate. It can be formulated as
$$
\mathcal {L} _ {\mathrm {k d}} = \left\| \hat {I} _ {s} \cdot \left(1 - \hat {M}\right) - \hat {I} _ {t} \cdot \left(1 - \hat {M}\right) \right\| _ {2}, \tag {2}
$$
where  $\hat{I}_s$  and  $\hat{I}_t$  are the images predicted by the student model and the teacher model, respectively.
**Training Objective** Our whole pipeline is trained in a self-supervised manner. The overall loss includes a photometric loss  $\mathcal{L}_{\mathrm{ph}}$ , a conflict-aware self-distillation loss  $\mathcal{L}_{\mathrm{kd}}$  and a mask loss  $\mathcal{L}_m$ . It is well noted that we introduce a self-distillation training mechanism therefore we can have both the observed new images and a teacher model to provide the supervisory signals. Hence we divide the inputs into two parts based on camera pose distributions, defined as new and old poses, respectively. To be specific, when the input points are sampled using new camera poses, we use  $\mathcal{L}_{\mathrm{ph}}$  to generate supervision.  $\mathcal{L}_{\mathrm{ph}}$  measures the reconstruction error between the predicted image  $\hat{I}_s$  and newly captured ground-truth image  $I_n$  as
$$
\mathcal {L} _ {\mathrm {p h}} = \left\| \hat {I} _ {s} - I _ {n} \right\| _ {2}. \tag {3}
$$
To further enhance the change awareness, we also propose a mask loss under new poses as
$$
\mathcal {L} _ {\mathrm {m}} = \left\| \hat {M} - \hat {M} _ {g t} \right\| _ {2}, \tag {4}
$$
where  $\hat{M}_{\mathrm{gt}}$  is the pseudo ground-truth mask.  $\hat{M}_{\mathrm{gt}}$  is computed by the difference map between newly captured images and the predicted images from the teacher model using the same camera pose as the newly captured images. Besides, when we want to compute  $\mathcal{L}_{\mathrm{kd}}$ , the uniformly sampled old camera views can be used as inputs to both the teacher model and the student model, and the definition is given in Equation 2. Finally, the overall training objective is
$$
\mathcal {L} _ {\text {a l l}} = w _ {k d} \cdot \mathcal {L} _ {\mathrm {k d}} + w _ {p h} \cdot \mathcal {L} _ {\mathrm {p h}} + w _ {m} \cdot \mathcal {L} _ {\mathrm {m}}. \tag {5}
$$
More implementation details are described in the supplementary file.
4.2 Benchmarks
To our best knowledge, there is no existing dataset for our proposed continual learning task of NeRF and no proper evaluation metrics to evaluate the forgetting behavior. Therefore, in this section, we will first propose a new dataset containing two synthesis scenes, one real-world scene, and an extra city-level scene. Then we propose metrics specifically designed to evaluate how serious the forgetting after fine-tuning is.
Operations We categorize scene operations into three types: atomic, composite, and sequential. Atomic operations consist of addition and deletion. Composite operations include movement and substitution, which modify positional relationships or replace existing objects with new ones. Sequential operations apply a series of basic or composite operations over time. Different from composite operations' focus on quick, specific region modifications, sequential operations aim to monitor long-term error accumulation in changing real-world scenarios.
Camera Views We evaluate the reconstruction quality of both local details and the overall scene using two testing indicators: local and global views consisting of a set of evaluation camera poses. Six camera trajectories are designed to cover the entire scene, with five capturing the four orientations of a room (local view) and the sixth simulating a person wandering around the room (global view). Additionally, we define an object-centric camera view to assess adaptation to new changes.