Meta-Continual Learning of Neural Fields

Authors: Seungyoon Woo, Junhyeog Yun, Gunhee Kim
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Meta-Learning; Continual Learning; 3D Vision; Neural Radiance Fields;

Abstract:
Neural Fields (NF) have gained prominence as a versatile framework for complex data representation. This work unveils a new problem setting termed Meta-Continual Learning of Neural Fields (MCL-NF) and introduces a novel strategy that employs a modular architecture combined with optimization-based meta-learning. Focused on overcoming the limitations of existing methods for continual learning of neural fields, such as catastrophic forgetting and slow convergence, our strategy achieves high-quality reconstruction with significantly improved learning speed. We further introduce Fisher Information Maximization loss for neural radiance fields (FIM-NeRF), which maximizes information gains at the sample level to enhance learning generalization, with proved convergence guarantee and generalization bound. We perform extensive evaluations across image, audio, video reconstruction, and view synthesis tasks on six diverse datasets, demonstrating our method’s superiority in reconstruction quality and speed over existing MCL and CL-NF approaches. Notably, our approach attains rapid adaptation of neural fields for city-scale NeRF rendering with reduced parameter requirement.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730529250128

thread_type:
official_review

<thread_object>
creator: reviewer_381t

<content>
summary:
In this paper, the authors propose a new problem setting and strategy to introduce meta-continual learning in neural fields. They conduct extensive experiments to demonstrate that their approach outperforms baseline methods.

strengths:
1.	Experiments on various datasets are conducted to validate the effectiveness of the method.
2.	The method is general and could be applied to many different applications.

weaknesses:
1.	The implementation and comparison are conducted based on methods published in 2021 and 2022. Only SwitchNeRF was published in ICLR 2023, which was also long ago. It is kindly suggested to conduct experiments on more recent methods to evaluate the proposed approach. I would like to see some comparative experiments with [a].
2.	I would expect more visualization in the experiments, for example, comparison of view synthesis on MatrixCity dataset.
3.	The experiments should include some commonly used evaluation metrics, such as SSIM.

[a] GF-NeRF: Global-guided Focal Neural Radiance Field for Large-scale Scene Rendering, WACV 2025

questions:
See Weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715292176

thread_type:
official_review

<thread_object>
creator: reviewer_r5bd

<content>
summary:
The paper introduces Meta-Continual Learning of Neural Fields (MCL-NF), a framework merging modular neural architectures with meta-learning strategies to support continuous and adaptive learning of neural fields. The proposed approach aims to address the challenges in traditional continual learning for neural fields, such as catastrophic forgetting and slow convergence. The authors implemented Fisher Information Maximization loss (FIM loss) to optimize learning by emphasizing more informative samples, achieving improved generalization and learning speed. The paper’s extensive experiments across various tasks including image, audio, video and NeRF-based view synthesis, highlight the performance advantages of MCL-NF over existing methods. They offer an efficient and scalable solution suitable for large-scale and resource-constrained environments, advancing the capabilities of neural fields in sequential learning settings.

strengths:
1. MCL-NF’s combination of modular architecture and optimization based meta-learning is innovative and meets the adaptability needs of neural fields. 
2. The method mitigates catastrophic forgetting through modularization and shared initialization, without the need for experience replay that can be resource-intensive. The modular approach within meta-continual learning sounds pretty compelling.
3. The FIM loss implementation is theoretically sound, supported by convergence guarantees, and introduces a detailed weighting mechanism for learning stability and efficiency. 
4. The paper provides thorough experimental results, showcasing the model’s performance on various datasets, supporting the robustness of MCL-NF. The concept of evaluating the method in different modalities definitely deserves recognition. 
5. This work helps understand the broader scope of how meta-learning techniques could be integrated successfully with frameworks for continual learning, particularly in the context of neural fields. This can help further research to consider similar synergies between learning paradigms, which may result in even more powerful and generalizable machine learning models.
6. The paper is well-organized and written in a clear, academic tone. It is suitably embedded in the meta-continual learning world  with relevant citations. The authors show their deep understanding of the topic by presenting the setting, highlighting its motivation and organizing the experiments in a structured way.

weaknesses:
1. The setting, at least in the image domain, seems artificial - dividing the images into four patches (regardless of the size) seems to raise some concerns. It would be desirable to present an ablation study on it.
2. From a continual learning perspective, this setting is highly confusing. It should be explicitly described how the data is selected, processed and how the model is evaluated.
3. The paper lacks more recent methods in the main table. And even if MAML is a well-known method, it is quite outdated, with lots of recent enhancements such as La-MAML [1] for continual learning.  Moreover, the comparison is mostly made against methods in a different setting with only two methods in MCL (and MAML was published in 2017).
4. And even in different settings (not MCL), the papers are well-established, but some of them nowhere near the current state-of-the-art. There is no elaboration how this approach differs from recently presented settings/methods such as [2]. And even if they are mentioned, such as ANML they are not compared against.
5. Experimental results show the performance with a number of continual tasks set to only four, which raises concerns regarding the method’s scalability.
6. These experimental results are largely dependent on PSNR as the main metric of reconstruction quality. Additional metrics, such as Structural Similarity Index (SSIM) for image and video tasks, or perceptual evaluation metrics for audio, would provide a more comprehensive validation of MCL-NF’s performance.

Minor issues:
Line 155 - Redundant closing bracket

questions:
1. Could additional strategies beyond Fisher Information, such as curriculum learning, improve MCL-NF's adaptability and reduce reliance on specific meta-learning techniques?
2. Should the paper also use different techniques of splitting the image than cropping into patches? The idea behind the method is based on that, and perhaps applying Fourier Transformations would be beneficial.
3. Could alternative loss functions achieve similar sample prioritization for continual learning? If so, what criteria would make FIM the preferred choice?
4. As far as I understand, we treat each image separately (batch size 1). How do we find the appropriate number of coordinates?
5. What insights can this setting bring to the continual learning community?
6. In the methods compared without modularization, do these approaches utilize the full MLP network capacity to match the parameter count of the modularized versions?
7. The Fisher Information Maximization loss introduces a new hyperparameter. Could you provide more details on how this hyperparameter was selected and discuss its sensitivity and impact on the model's performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730730216315

thread_type:
official_review

<thread_object>
creator: reviewer_6fgr

<content>
summary:
The problem of continual learning in neural fields is an important topic. This work focuses on addressing the issues of catastrophic forgetting and slow convergence in existing approaches by proposing a new paradigm called MCL-NF. It introduces the Fisher Information Maximization loss. The experiments demonstrate that the proposed method can achieve satisfactory results.

strengths:
1.	This paper is well-written.
2.	This work combines meta-learning and continual learning paradigms into neural field training to address the challenges of catastrophic forgetting and slow convergence, which is interesting.
3.	This paper demonstrates the advantages of their method in experiments, which alleviates the issue of slow convergence to some extent.

weaknesses:
1.	In the experimental section, the limited scenarios are my main concern, which makes me worry about the limitations of the method.
2.	The paper uses a large number of quantitative metrics but lacks qualitative comparisons, especially for neural radiance fields.
3.	In neural fields, some incremental learning methods [1] might also need to be discussed, but they are not mentioned in the article.

[1] Zhang, L., Li, M., Chen, C., & Xu, J. (2023). IL-NeRF: Incremental Learning for Neural Radiance Fields with Camera Pose Alignment. arXiv preprint arXiv:2312.05748.

4.  The paper is recommended to discuss the differences between continuous radiance fields and traditional continual learning. You know, issues like catastrophic forgetting and slow convergence have long been a focus of continual learning researchers.

questions:
Please kindly refer to the above weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730800836072

thread_type:
official_review

<thread_object>
creator: reviewer_pv54

<content>
summary:
Briefly, this paper presents a strategy to continually and rapidly learn neural fields in a meta-learning manner. The paper further introduces a fisher information maximization loss for neural radiance fields. The proposed method advances in leading to no performance degradation incurred by forgetting during test-time by synergizing modular architecture with meta-learning.

strengths:
+ The paper is well written and easy to follow. 
+ Extensive and comprehensive experiments demonstrate the effectiveness of the proposed method.

weaknesses:
+ The technical novelty of the proposed method seems to be marginal since the authors directly employ the existing techniques. For instance, Fisher Information against catastrophic forgetting has already been proposed by Kirkpatrick et al., 2016. The authors do discuss the relation to this method and claim that their proposed FIM loss operates at the sample level rather than the parameter level. However, it is difficult to identify the advantage of the proposed FIM loss over the parameter-level approaches in CL (Chaudhry et al., 2018; Konishi et al., 2023) without fair experimental comparison and detailed discussion.

+ More detailed discussions and analyses in Table 1 are required to demonstrate the contribution of the proposed method. 

+ More recent state-of-the-art methods should be included for comparison to demonstrate the superiority of the proposed method. For instance, missing some SOTA methods, e.g., (Chung et al., 2022 and Po et al., 2023), for comparison on the MatrixCity dataset in Table 1.

+ More ablation studies in the main paper are required to demonstrate the contribution of the main component (i.e., the FIM loss) of the proposed method in the main paper. The authors only provide some results about the mod and MIM  with two different hidden dimensions in Appendix Table 2. 

Reference:
Po et al., Instant Continual Learning of Neural Radiance Fields, In ICCVW 2023.

questions:
Please refer to the Weaknesses section.

There is a wrong citation in either Line 322 (EWC Chaudhry et al., 2018) or Line 402 (EWC (Kirkpatrick et al., 2016) ). Actually, Chaudhry et al., 2018 denote their method as EWC++.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732703651823

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their helpful feedback. We appreciate that they acknowledge our effective methodology for MCL-NF (pV54, 6fGr, R5BD,  381t) and our clear writing (pV54, 6fGr, R5BD). Most importantly, they find our framework provide a meaningful contribution to the research community and relevant applications (R5BD,  381t).

We have uploaded the revised paper, with all modified components highlighted in magenta. Additionally, we have addressed the questions raised by each reviewer in our responses. We will incorporate the feedback and make the official release of the code publicly accessible.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a novel algorithm for neural fields, focusing on continual learning and meta-learning. Specifically, the weight updates in a continual manner are estimated based on a Fisher Information Matrix-derived criterion. All reviewers are positive about the value of the paper, and there was significant discussion and improvements during the rebuttal phase. In the rebuttal phase there were questions regarding comparisons to more recent baselines, and also comparisons with more realistic data settings, to which the authors answered to the reviewers convincingly. All in all, this paper is a good contribution to the literature.