Meta-Continual Learning of Neural Fields
Authors: Seungyoon Woo
Junhyeog Yun
Gunhee Kim
Venue: ICLR 2025 Poster (2025)
Keywords: Meta-Learning; Continual Learning; 3D Vision; Neural Radiance Fields;
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: This work is the first to combine meta-learning and continual learning for neural fields. Our approach improves performance with few adaptation steps for not only image, audio, video reconstruction but also city-scale NeRF rendering.

Abstract: Neural Fields (NF) have gained prominence as a versatile framework for complex data representation. This work unveils a new problem setting termed Meta-Continual Learning of Neural Fields (MCL-NF) and introduces a novel strategy that employs a modular architecture combined with optimization-based meta-learning. Focused on overcoming the limitations of existing methods for continual learning of neural fields, such as catastrophic forgetting and slow convergence, our strategy achieves high-quality reconstruction with significantly improved learning speed. We further introduce Fisher Information Maximization loss for neural radiance fields (FIM-NeRF), which maximizes information gains at the sample level to enhance learning generalization, with proved convergence guarantee and generalization bound. We perform extensive evaluations across image, audio, video reconstruction, and view synthesis tasks on six diverse datasets, demonstrating our method’s superiority in reconstruction quality and speed over existing MCL and CL-NF approaches. Notably, our approach attains rapid adaptation of neural fields for city-scale NeRF rendering with reduced parameter requirement.

Published as a conference paper at ICLR 2025
META-CONTINUA LEARNING OF NEURAL FIELDS
Seungyoon Woo, Junhyeog Yun, Gunhee Kim
Scoul National University
seungycen. woolvision. anu.ac..e kr, (Junhyeog, gunhee Jaanaheeeeae 0snu ac, kr
ABSTRACT
Neural Fields (NF) have gained prominence as versatile framework for plex data representation. This work unveils a new problem setting termed Meta- Continual Learning of Neural Fields (MCL-NF) and introduces novel strat- egy that employs a modular architecture combined with optimization-aase learning. Focused on overcoming the limitations of existing methods for continual learning of neural fields. such as catastrophic forgetting and slow convergence, our strategy achieves high-quality reconstruction with significantly improved learning speed. We further introduce Fisher Information Maximization loss for neural radi- ance fields (FIM-NeRF), which maximizes information gains at the sample level to enhance learning generalization, with proved convergence guarantee and gen- eralization bound. We perform extensive evaluations across image, audio, video reconstruction, and view synthesis tasks six diverse datasets, demonstrating our method's s superiority in reconstruction quality and speed over existing MCL and CL-NF approaches. Notably, our approach attains rapid adaptation of neural fields for city-scale NeRF rendering with reduced parameter requirement. Code is available htttttt:///gttti/t con/ seungyoon- wtto--ooooooM--L
meta-
INTRODUCTION
Neural fields have recently emerged as versatile framework for representing complex data with neural networks (Park al. 2019 Mildenhall et al. 2020 Yariv al.] 2021; Yu et al.) (022). In this framework, data point (e.g.. an image, a video, or a 3D scene) is regarded as a field, and neural network trained to map a coordinate to the corresponding field value (e.g., pixel value, signed distance, radiance). Following the success of neural fields, multiple attempts have been made to learn neural fields continual learning (CL) settings (Yan et al., 202I Guo et al.] 2023 Wang eet al. 2023 Chung al. 2022). The primary assumption of these approaches, which we collectively refer to as continual learning of neural fields (CL-NF), is that the information of a field is not fully accessible once but instead sequentially obtained. Naively training neural network on such non-stationary data stream would result in catastrophic forgetting (Robins 1995; Kirkpatrick et al., 2016), and thus additional techniques introduced to mitigate the forgetting.
Meanwhile, one of the significant challenges of neural fields is the extensive training time. Rapid training of neural fields is crucial for practical applications. and it has motivated the emergence of meta-learning approaches for neural fields (ML-NF) (Tancik et al. 2021 Chen & Wang 2022]
Kim et al. 2023 Unlike conventional neural field training, which fits a neural network to a single data point, the meta- weea-learning approaches consist of multiple data points split into meta-traie meta-training and meta-test sets. By optimizing the learning process using the meta-training set, they can significantly reduce the training time of neural fields for new tasks.
In this work, we explore novel problem setting that aims to combine the best of both worlds: Learning of Neural Fields (MCL-NF). Within this setting, meta-learn how to learn neural fields not only continually but also rapidly. Although there are several meta-continual learning (MCL) approaches in image classification domains (Javed & White 2019 Beaulieu al.)
2020 Banayeeanzade et al. 2021), to the best of our knowledge, ours is the first work that explores MCL the context of neural fields.
MCL-NF is powerful in drone and satellite applications, where large-scale environments are cap- tured by resource-constrained edge devices and the memory limitation hinder full exploitation of

---

Published as a conference paper at ICLR 2025
captured data. MCL-NF can leverage neural fields' ability to compress complex 3D environments into compact representations and continual learning's 's capacity to seamlessly transition between dif- result, MCL-NF offers fast learning,
ferent levels of detail based on viewpoint and resources; scalability, real-time rendering, and adaptability. This powerful combination facilitates fast gener- ation of highly detailed virtual models for applications like urban planning, construction, environ- mental monitoring, and disaster response. We experiment our MCL-NF on real-worid scenes in city-scale such MatrixCity (Li al. 2023) to shed a potential light on large-scale renderings.
As first solution to MCL-NF so far, we explore a strategy that combines modular architecture with optimization based meta-leaming. Existing CL-NF approaches mostly rely on replaying previous data to prevent forgetting. However, ceplay-based approaches suffer from consistently increasing replay costs, which can problematic in the MCL.-NF setting where rapid adaptation large-scale rendering is desired. Modularization is major branch of CL. that circumvents this problem by as- signing a separate module for each task (Rusu et al.) 2016; Aljundi et al. (017). Moreover, modular architectures have been popularly studied in the neural field literature, especially for NeRF (Milden- hall et al. 2020j, to improve the efficiency of handling large-scale scenes (Reiser et al. 2021 Turki
et al. 2022 Mi & Xu 2023). By learning a good initialization for the modules with an optimization- based meta- weeannn such as MAML (Finn et al.) 2017), our approach can rapidly learn new tasks while maintaining existing knowledge. Previously, modularization techniques employed a fixed number of sub-modules for processing, despite the potential advantages of a continual learning that allows separating modules in a task-wise manner. By sharing initializations between sub-modules, we can increase the number of sub-modules whenever stakeholders require. This continual approach to modularization can dynamically adapt to evolving requirements without being constrained by predetermined, finite number of modules.
We also introduce a Fisher Information Maximization loss for Neural Radiance Fields (FIM-NeRF) as a novel approach that incorporates Fisher Information directly into the NeRF training process. FIM-NeRF weights individual samples in the loss function based on their Fisher Information con- tribution, effectively prioritizing more informative regions of the scene during training. FIM-NeRF maximizes the mutual information between the model parameters and the observed data, potentially leading to more efficient learning and better generalization. We also present the theorems for proving the convergence guarantee and generalization bound of this new loss function. This sample-level ap- plication of Fisher Information distinguishes FIM-NeRF from previous parameter-leve approaches in CL (Chaudhry et al.) 20188] Konishi et al.] 2023), offering a new perspective on how to leverage information theory in 3D scene representation and rendering.
Our approach, synergizing modular architecture with meta-learning, leads to no performance degra- dation incurred by forgetting during test-time. Our framework is versatile across various NF modal- ities. We carry out extensive empirical evaluation across the image, audio, video reconstruction. and view synthesis tasks on six diverse datasets, including three 2D image datasets (CelebA (Liu et al. 2015). FFHQ (Karras et al., 2019). and ImageNette (Deng et al. 2009)), one video dataset (VoxCeleb2 (Chung et al.] 2018)). one audio dataset (LibriSpeech (Panayotov et al. 2015)), and one NeRF dataset (Matrix (Li et al. )023)). Our method surpasses both previous MCL and CL-NF approaches in reconstruction quality and speed. Ultimately, our unified approach efficiently addresses city-scale NeRF challenges on MatrixCity (Li et al. 2023) using a smaller parameter size than existing single neural field methods.
2 RELATED WORKS
Continual Learning of Neural Fields (CL-NF), Neural fields learn function fo Rd + R that maps a coordinate E Rdd to field quantity E Re using a neural network to represent complex data point. Continual learning trains a model on a series of tasks, Ti, T2,.... ..... Tna each associated with its own data. In video examples, each frame can be task T. and in NeRF con- texts, each grid separated by its 3D coordinates is a distinct task. Every task T, consists of samples {(x), uf)), (xf, vu),.... ... (x= y." ))")), each of which is made up of the coordinates. A significant chal- lenge here is catastrophic forgetting (Robins) 1995] Kirkpatrick et al.] 2016];; when the model, after training on a new task T loses the knowledge acquired from the prior tasks.
Among several branches to handle forgetting, replaying previous samples has been a popular ap- proach due their simplicity and effectiveness. For the CL. of neural fields, CNM (Yan et al. 2021)
2

---

Published as a conference paper at ICLR 2025
utilizes experience replay to reconstruct a 3D surface from streaming depth inputs. Later, Guo et al. 2023); ; Chhnn et al. (2022): Po et al. (2023) leverage knowledge distillation to generate pseudo- samples of past expenence using neural network as memory storage. However, replay methods have scalability issues and increase computation in large-scale environments. Our proposed method is far from this limitation while solving challenges that arise in the CL. problems.
Meta-Learning of Neural Fields (ML-NFF, In the field of NF, several works leverage meta-learning for rapid learning and generality to unseen data instances and data-type agnostic features. MetaSDF (Sitzmann et al. 2020) learns a weight initialization for fitting neural representations to signed dis- tance fields. LearnedInit (Tancik et al., 202T) and GradNCP (Tack et al. 2023) exploit bilevel optimizations to encode meta-imitialization prior to expanding their horizons to general NFs. Mod- ulating purl-parts of the architecture such as TransINR (Chen & Wang 2022) and Generalizable IPC (Kim et al. 2023) can increase the efficiency of NF training. While they show promising results, they have to yet satisfy some applications like drone or satellite imagery (Po et al. 2023), which require sequential data processing with limited memory storage and immediate updates. Our ap- proach introduces optimization-bass weight initializations CL settings, and effectively meets such demands of NFs while promoting quicker convergence and enhanced generality.
Meta- Cetta-Contiuall Learning (MCL). Optimization based meta-learning (Finn et al., 2017]: Antoniou et al. 2019| Nichol et al. 2018 Fallah et al. 2020; Flennerhag al., (020) tries to find a favorable starting point of model's parameters to enhance the SGD optimization. However, a straightforward use of meta-learning makes all parameters vulnerable to SGD updates, which can lead to excessive adaptability and forgetting. Online meta-learning (Javed & White| (2019) is introduced as an MCL strategy to tackle this; in its inner loop, only the top layers are updated while fixing the other lay- ers, which are updated in turn in the outer loop. ANML (Beaulieu et al.] 2020) seeks to balance stability and plasticity in meta-learning; rather than freezing the lower part of the encoder, it uses a neuromodulatory network that remains static during the inner loop. However, they inherit some challenges from meta-learning like the need to compute second-order gradients through the entire inner loop, leading to high computational demands and potential gradient issues.
Modularization in Mixture-of-Experts. Modularization can be viewed as a special case of the Mixture-of-Expert (MoE) framework (Jacobs et al. 1991), which has seen significant advance- ments in recent years particularly in the realm of large language models. For continual learning, Progressive networks (Rusu et al. 2016) and ExpertGate (Aljundi et al., (017) are typical examples of this one-tast mee-task-pee task-per-exxert modularization strategy. Interestingly, Chen et al. (2023); [Shen et al.) 2023 Pan et al. (2024) optimize the distribution of data among experts by maximizing the mu- tual information (MIM) between the input and the expert selection. For instance, Pan et al. (2024) propose an MIM-based expert routing to improve the performance and interpretability of language tasks. In the context of neural fields, this has led to novel NeRF techniques for scene decomposition and representation (Rebain et al. 202T: Tancik et al., 2022). They leverage information -thooretcc principles to guide the division of 3D space among different modules, effectively creating modular representation of complex scenes.
Extending the concept of information -uided modularization to the meta-continual learning. we use Fisher Information Maximization (FIM) into our loss function, drawing parallels to the MIM approaches in MoF, but in the sample-specific level. This allows us to prioritize informative samples during training, potentially leading to more efficient learning and better generalization in the CL-NF.
3 PROBLEM STATEMENT
3.1 CONTINUAL LEARNING OF NEURAL FIELDS (CL-NF)
the continual learning of NF, the model fo updates its parameters / based on the sequential task- specific data. They are a series of context sets C₁ each comprising coordinates (x*, y²) in R', R for j = 1,..., m. A context set can be defined as a subset of data relevant to a particular task within sequence of tasks to be learned. While learning from a context set C, the model cannot access the previous context set C,-1. adhering to the constraint of CL. Each context set functions as distinct task T/ such as each frame in video domain (Wang al. 2023) each 3D grid in a NeRF problem (Chung et al., 2022) Po et al. (023) without explicit geometric supervision. The model fo, updates its parameters based on the sequence of context C., producing an output fö, for
3

---

Published as a conference paper at ICLR 2025
6
Aiber Mfperatoeteees
Figure 1: Illustration of the transition from traditional MSE loss to FIM-Loss in neural network. highlights how the FIM used to calculate sample-specific weights. These weights are then incorporated into the final loss function, allowing the model to prioritize more informative samples.
Tú, which then becomes the starting point for Te+1. The loss function for each T, defined as Mean Squared Error (MSE): C; = 1 Ey=1fo, (fo, (x) v/) - =/)² where fo, (x), yt) predicts the signal value for coordinates (e/.v.) and s/ the true value. The objective is to minimize the mean loss across all tasks up to t: Liotal = Ek=1 (f,.), which aims to reduce the cumulative loss and thereby mitigating catastrophic forgetting across all previously seen tasks.
3.2
MCL-NF: CL-NF MEETS META-LEARNING
Continual learning in the NF domain often requires a large number of convergence steps (Tack et al.,
2023
Yan al. 2 Chung al. 2022) and lacks generality across tasks (Guo al. [023] Wang et al. 2023). A prevalent solution may be to adopt optimization-
meea-nearning, which not only addresses the challenges of convergence and generality but also brings forth the benefits of well-defined task variants and modality-agnoste features (Nichol et al. 2018 Zintgraf et al. 2019
Antoniou al. 2019)
We structure our approach into episodes E2.,8...... E,), each consisting of two stages: task-specife adaptation and meta-update. The task adaptation stage, akin to upon mentioned CL-NF, comprises sequences of signals forming context sets. For any given T, in this stage, the model fa, updates over iterations, starting from the end state of the previous T,-1, denoted as fer where & typically spans hundreds of thousands of iterations in prior works (Yan et al., 2022 Chung et al., (022). The meta-update stage optimizes the initial model configuration fo, of each episode, based the final output for- Here, the parameter 8' is optimized using
(z プ 1=1
st)² $
8' - 0o Vo.
(fo,
(1)
1=1
where the initialization 0' is optimized by multiple meta-train episodes and becomes starting point for the meta-test set, thereby enabling rapid adaptation to new tasks within meta-test episodes.
Meta-learning enables neural networks to assimilate priors for swift adaptation, thereby facilitating rapid learning and efficient memory usage. However, the mere conjunction of continual learning and meta-learning, like previous methodologies (Javed & White 2019: Beaulieu al. 2020), does not
4

---

Published as a conference paper at ICLR 2025
Algorithm Modularized MCL-NF with Fisher Information Maximization Loss
Require: (1) A sequence of tasks {71, ......
along with context sets {C1, .z...., Learning rates Thaner. Nouter. and (3) Fisher Information weight X.
. Te)
....,, Ct). (2)
1: Shared Initialization:
2: Initialize a shared set of parameters Ssharod-
for 1 to do
4: Initialize module fe, with Oshared for task Tí
5: end for
6: for i 1 & do
7: Select m coordinates context set C, for task T.
8: Initialize Fisher Information Matrix F; for task T:
9:
Inner Loop (Task-speeiifc Learning):
10:
for each training sample (z, y) in context set C2 do
11:
Compute output = fe, v).
12:
Compute Fisher Information weight: w0,)) = 1 + tr(g(0,)) ((,)),
13:
where g(+;) Vo, log l(ss())
14:
Compute FIM Loss: FFIM f1 x. y) = w(f)) : =sl?.
15:
Update 8, for task T;:
0(6) = 0000 Tinner Σjn OCFIM (ffi of v). s)))s)))) 5)/00
16:
Update Fisher Information Matrix F,, end for
17:
18:
if new task arrives then
19:
Expand module fa, by adding new parameters new initialized with shared-
20:
end if
21:
Outer Loop (Metallearning):
for each validation sample (Ieals Yeat) in query set Qual do
23: Compute FIM Loss LFIM Teats Yeat) as in steps 10-12.
Update 0g for meta initialization: 0¹ = 0: Mouter Vo, LrMM (f (foge) (Deats Yeat), seal).
25: end for
26: end for
27: Inference:
28: for each coordinate (z, y) in the input space do
29: Determine subset of modules where (a, y) belongs to: Gz(z).
Obtain the final output from relevant modules: s(x.y) ΣT G((x)) fo; (2.1).
31: end for
inherently ensure the mitigation of forgetting. Therefore, we propose a simple yet effective method to address these challenges.
APPROACH
4.1 SPATIAL AND TEMPORAL MODULARIZATION
We adopt MAML (Finn et al. 2017) as an optimization-bast meta-learning algorithm, known for its efficiency with small convergence steps. Then, our method utilizes modularization within the MCL framework of NFs. The modular approach, inspired by the recent successful large-scale NeRFs such as MegaNeRF (Turki et al. (022) and SwitchNeRF (Mi & Xu 2023), not only en- sures efficient memory usage with quicker computations but also mitigates catastrophic forgetting by keeping task-specifie parameters separate. We can harness the intrinsic spatial and temporal correlations of coordinates, based on that subsequent frames in video or views NeRF are highly overlapped. Thus, we enhance module adaptability by sharing initialization of each module, key factor in large-scale radiance fields. Moreover, our shared initialization provides the flexibility to adjust the number of modules, independent of the number of tasks. which is vital for managing varying complexities within large datasets. Unlike the traditional MAML approaches that struggle with forgetting over extensive iterations in test-time, our strategy enables super-resolution image

---

Published as a conference paper at ICLR 2025
hancements and detailed city-scale NeRF reconstructions while overcoming the spectral bias Tack et al. 2023) often seen in NFs through effective test-time optimization.
4.2 THE FISHER INFORMATION MAXIMIZATION Loss
For large-scale, resource-constrained continual learning, efficient knowledge re-use is crucial. How- ever, standard modularization struggles with sharing knowledge across overlapping tasks such as repeating building appearances in city-scale 3D modeling. These limitations call for an addi- tional method to complement modularization, facilitating efficient knowledge transfer, optimizing resources, and maintaining performance across tasks.
Hence, approach takes advantage of sample re-weighting to replace popular rehearsal methods that may struggle when storage is limited and rapid processing is required. This approach dynam- ically adjusts the input importance based on information content, and prioritizes learning from the most relevant aspects of current inputs. Consequently, it can achieve similar knowledge retention and transfer rehearsal methods, but without memory and computational overheads.
As modularization is viewed as a mixture of single-task experts, mutual information maximization can be a useful vehicle. Our solution to perform sample re-weighting exploiting Fisher Infor- mation Matrix (FIM), which captures task-relevant information within model parameters without explicit storage of past experiences. It provides computationally efficient proxy for mutual infor- mation through a local quadratic approximation of KL-divergence between parameter distributions.
FIM calculation from readily available gradient information facilitates efficient optimization, mak- ing it particularly suitable for our large-scale, resource-limites settings.
We propose the Fisher Information Maximization loss (FIM loss) to replace Eq.(T) by appending only weight term:
(0) s9)2 }
0' - V/V
(2)
and (f) is a weight derived from the Fisher Information:
w(()) = + Ag( FF())
(3)
where (0) Volog (f: v()) is the score function, F(0) is the Fisher information matrix, and À > 0 is hyperparameter. The weight (0) comprises base weight of 1, hyperparameter controlling Fisher Information influence, the gradient (:) representing sample sensitivity, and the inverse Fisher Information Matrix F(0)-1 for parameter importance normalization. E3. pri- oritizes the samples causing big changes in important model parameters. Larger gradients indicate more informative samples, while F(0)- adjusts for varying parameter importance. This weighting focuses the samples most likely to improve model performance, balancing knowledge preservation with information acquisition, which crucial for meta-continual learning.
4.3 THE RELATION WITH INFORMATION GAIN MAXIMIZATION
We now establish the connection between FIM loss and the principle of information gain max- imization. Based on the theorem in (Kunstner et al. 2019) that links KL-divergence to Fisher In- formation,
show that the FIM loss approximates mutual information maximization between model parameters 0 and data D. The theorem and its detailed proof is presented in Appendix 1
Let pD((0) and p(D Ad) be two probability distributions parameterized by 0 and 0 A8, spectively. The Kullback-Leibiee (KL) divergence between these distributions can be approximated (Belghazi al. 2018 Veyrat-Charvillon Standaert 2009):
KLD(D)))) 40) ~ F())AO
(4)
where F(0) is the Fisher Information Matrix. As F(0) provides local quadratic approximation of -iedieerg,, by using F(0) in our sample-level loss function, we implicitly measure and priori- tize information content, optimizing mutual information without explicit calculation.

---

Published as a conference paper at ICLR 2025
Comparison with EWC. EWC (Chaudhry et al. 2018) also Fisher Information against catas- trophic forgetting. It works at the parameter level. penalizing changes to important previous task parameters. On the other hand, our FIM loss operates the sample level, dynamically weighting samples based on the Fisher Information contribution. This enables finer learning control, crucial for neural fields' spatial and temporal information locality. By prioritizing informative data regions or timepoints, it enhances rapid adaptation to new tasks in continual learning of complex 3D envi- ronments, while preserving previous knowledge.
4.3.1 CONVERGENCE ANALYSIS AND GENERALIZATION BOUND FIM-SGD
In Appendix A.2 we present the theorem and its derivation about the convergence of our Fisher Information Maximization via Stochastic Gradient Descent (FIM-SGD). This analysis guarantees the convergence of our FIM-SGD algorithm to the optimal parameters, with rate that depends on the condition number of the problem and the variance of the stochastic gradients. This convergence property can further explain the effectiveness of our algorithm.
A generalization bound provides a probabilistic limit on the generalization error, which measures how well an algorithm performs on unseen data compared to training data (Pérez Louis (020) Cao & Gu 2020). We present the theorem and its derivation
generalization bound for our FIM- loss Appendix AA.3 shows that our FIM-Loss maintains good generalization properties with the bound scaling similarly to standard VC-dimension based bounds.
EXPERIMENTS
We evaluate the performance of our approach across image, video, audio, and NeRF datasets with high diversity and minimal redundancy. Moreover, we investigate the compute and memory usage of our approach in large-scale environments. Due to space limitation, include additional details of experiments in Appendix, which covers ablation study, visualizations, and computation analysis. They provide a more comprehensive view of our method's performance and characteristics.
5.1 EXPERIMENTAL SETUP
We set the number of continual tasks to four, following prior works (Mi & Xu 2023 Cho al., 2022). This translates to four frames in the video domain and four 3D grids in the NeRF domain. We measure increased performance over multiple iterations from meta-initialization, focusing on the order of computations (steps). We run all experiments three times, reporting average performance until 500K outer steps, following (Tancik al. 202T).
Baselines. LearnedInit (Tancik et al.) 2021) is an ML-NF approach, which can be regarded as an upper-bound baseline (Meta (UB) since leverages batch processing while bypassing contin- ual learning challenges. Other ML-NF methods such as (Chen & Wang 2022 Kim et al.) (023) employ transformer-basee hypemetworks. which are not applicable our problem setting. We compare with OML (Javed & White 2019) as representative MCL baseline, and MAML (Finn al. 2017) combined with CL setup, which denoted by 'MAML+CL'. We select OML because it supports both classification and regression tasks, aligning with neural fields' regression output. We also include EWC (Kirppatrick et al.) 2016) and ER (Rolnick et al. 2019), which represent typ- ical regularization-basee and sircaaral-basce CL approaches in comparisons, well as MER (Riemer et al., 2019), which exemplifies advanced CL baseline with bi-level optimization. Through- out the experiments, "Ours (mod)' and 'Ours (MIM)' refer to the proposed MCL-NF method with modularization without or with MIM, respectively.
5.2 RESULTS OF IMAGE RECONSTRUCTION
We first experiment 2D image reconstruction using CelebA (Liu et al. 2015). FFHQ Karras et al., 2019), and ImageNette (Deng et al.) 2009) datasets. We process 2D image by dividing it into multiple patches; for instance, a 180 x 180 image is split into four 180 x 45 patches. The coordinate- based MLP consists of five layers with Sine activations, configured with d 128, din for image signal input, and d = for RGB outputs. We measure the reconstruction accuracy for the entire images in terms of PSNR.
7

---

Published as a conference paper at ICLR 2025
15
15
PENTR
30
18
32
84
125
258
512 1024 2018
8
18
32
84
125
250
512 1024 2048
Test time optimizattoon stops per task
Test time optimization steps task
(a) CelehA
(b) FPHQ
40
PSNR
16
32
64
128
256
512
1024 2010 6096
16 32
66
128
250
512
1024 4068 6096
Tist Gime optiniration slapa Lunk
Thi--lime optimization slaps pur Lonk
(c) ImageNette
(d) VoxCeleb2
60
PSSRR
PSNNR:
100
30
16
256
2048-666
Test-time time optimiration steps took
Thet-time ogtimization stepe per task
(e) LibriSpeechl
LibriSpeech3
32.5
30.0
30.0
23NR
PSNR
27.5
27.5
25.0
25.0
22.5
22.5
0
36
32
64
328
250
512
1024
36
32
64
120
256
532
1026
Test-thme optimization stage per tasis
Test Test-tim opelmtration stogs tanis
(g) MatrixCity-Br
(h) MatrixCity-Bii
MOL
MAMLACL
OML
Ours (mod)
Ours (MIM)
Figure 2: The PSNR comparison between various meta-learning methods over the adaptation steps. Our method demonstrates consistent improvement in PSNR the number of steps increases, out- performing traditional MAML (Finn et al. 2017) and OML (Javed & White 2019), particularly in longer adaptation sequences in all modalities and datasets.
Highly- Sttucturr Images. The CelebA (Liu et al. 2015) has been actively used in meta- meta-continual learning research. The consistent positioning of faces the center of images provides strong prior, ensuring a level of uniformity across the dataset. For CelebA, we use 178 x 178 resolution images with 1-pixel aardppgdiing for 180 x 180 input resolution, adhering to established practices.
High-Resolution Images. The FFHQ (Karras et al., 2019) consists of 512 x 512 high-resolution images. evaluates whether even with limited memory meta-learning can reconstruct the data.
High-Diversity Images. The ImageNette (Deng et al. 2009) with a 160 x 160 resolution shows its high diversity, containing various classes, objects, and backgrounds. It can demonstrate the model's robustness to reconstruct images with less redundancy in patches in just few learning steps.
Results. Fig 2a 2c show the results three image benchmarks. The consistently superior perfor- mance demonstrates the robustness of our method, with each dataset highlighting unique benefits. The PSNR values of our method are stably high across all test-time optimization steps in CelebA and FFHQ, suggesting that method maintain its great quality despite computational constraints. Furthermore, results on ImageNette show gradual improvement, implying our method's 's capability to handle diversity and complexity effectively.
8

---

Published as a conference paper at ICLR 2025
5.3 RESULTS OF VIDEO RECONSTRUCTION
We use the VoxCeleb2 dataset (Chung et al. 2018) for 3-dimensional video processing. Each video is dissected into consecutive frames with a size of 112 x 112, extracted at consistent time intervals to form sequence. Following prior works, figured with d = 256, din- = 3 for (a, y, t) coordinates, where t represents the temporal dimension, and dous for RGB outputs. Four sub-modules are divided along temporal axis, meaning that each sub-module configured with ( d = 128, din = dout 3) is allocated to each of the frame sequence. The goal of the test to accurately reconstruct these sequential frames, thereby achieving coherent and continuous representation of the video. The temporal aspect of videos is essential for models to understand not only static features but also their sequence evolving over time.
utilize a five-layer MLP with Sine activations, con-
Results. As shown in Fig 2d the consistent enhancement in PSNR at each step confirms that our method can capture and reconstruct the temporal dynamics of video sequences. This trend validates method's 's efficacy in dealing with sequential data, where temporal correlations are crucial.
5.4 RESULTS OF AUDIO RECONSTRUCTION
Our framework is further extended to audio experiments with the LibriSpeech-clcan dataset (Panay- otov et al. )2015). We train our framework on randomly cropped segments of audio data. In previous works (Kim et al. 2023 Tack al. 2023), the coordinate-based MLP is equipped with five layers using Sine activations (d = 256, diz
for the (x. coordinates, and dout = 1), amounting the parameter size of (p 197, 120). We transition from a singular MLP to a more modular architecture. We divided the model into four sub-modules, each having a reduced dimensionality of (d 128) and cumulatively totaling (p 196, 704) parameters. For evaluation, we reconstruct audio segments for accurate and coherent playback. Test samples are meticulously trimmed into seg- ments of three seconds. By breaking down long audio sequences into smaller, manageable segments for continual learning, models can meta-learn the characteristics of speech more effectively.
Results. In Fig 2f our method consistently increases its reconstruction quality whereas the OML loses its ability up to 64 steps and its baseline MAML suffers from the same issue within the CL setting. Moreover, our methods show comparable or even higher performance (in some time-steps) than Learnedlnit, the upper bound (UB) of meta-learning. Our model can understand and gener- ate speech by capturing the relations between phonemes and the subtleties within various speech attributes, leading more natural and accurate speech processing.
5.5 RESULTS OF VIEW SYNTHESIS
We evaluate approach to view synthesis in the context of large-scale city rendering in the Matrix- City dataset (Li ct al. 2023). Each city scene is processed into a set of multi-view images captured from various viewpoints to ensure comprehensive coverage of the urban environment. The resolu- tion of 1920 x 1080 pixels is resized into 480 x 270 due to its memory constraint. We partition the city blocks into meta-training and meta-test sets, allocating 70% of the total blocks to meta-training and the remaining 30% to meta-testing. Following standard NeRF practices, we employ ten-layer MLP with ReLU activations, configured with d 1024, dnn 5 for (x, y, z, 0, Φ) coordinates, where 6 and ó represent viewing angles, and dout 4 for RGB and density outputs. We reduce the original hidden dimension by a half, configured with d 512, as the computation of second-order derivatives in longer sequences significantly increases memory requirements. Four sub-modules are divided along spatial axes, meaning that each sub-module configured with 256, dnn 5. dout 4. Following MegaNeRF (Turki al. (202)) aim not for the SOTA performance but for the practicality and efficiency of our method in city-scale volumetric rendering.
Results. Both Ours (mod) and Ours (MIM) show higher quality results compared to the MAML+CL. The MAML+CL approach exhibits a symptom of forgetting during inner adaptation, as reflected in the performance metrics. Throughout most of the experimental periods, Ours (MIM) at- tains better performance than Ours (mod). Additionally, Ours (MIM) displays a more stable increase in performance over time, characteristic that is necessary for test-time optimization scenarios.

---

Published as a conference paper at ICLR 2025
Table 1: Performance comparison in terms of PSNR (with the best and second best) at three key points: Step 1 (initial performance), Best (step), and Step 4096 (final performance). Methods are categorized into offline learning (OL), Continual learning (CL), continual meta-learning (CML), Meta-offline learning (MOL), and Meta-continual learning (MCL).
Modality
Image
Datasct
ColobA
ImageNette
FFHQ
Metric (PSNR)
Step
Best (step)
Step 4096
Step
Best (step)
Step 4096
Stop
Best (step)
Step 4096
OL
OL
11.78
38.15 (4096)
38.15
12.16
38.75 (4096)
38.75
12.78
32.98 (4096)
32,98
CL
12.41
15,60 (256)
15:57
13.14
16.76 (64)
16.49
13.74
16.35 (128)
15.93
FR
12.53
43.90 (4096)
43,90
13:23
45.74 (4096)
45.74
14.87
35.30 (4096)
35.30
EWC
12.59
20.08 (256)
18.19
13.23
19.77 (256)
17.30
13.96
21.29 (512)
19.25
CML
MER
21.97 (256)
20.27
12:28
21.31 (256)
19.28
23.09 (256)
20.61
MOL
MOL
29.21
45.28 (4096)
45.28
24.35
45.68 (4096)
45.68
26.71
35,21 (4096)
35,21
MAML+CL
29.38
33.22 (64)
28.06
24.63
29.21 (64)
25.03
26.94
29.91 (64)
23.88
MCL
OML
23.06
(16)
28.65 (32)
22.23
Ours
29.29
45.40 (4096)
45.40
25.18
44.83 (4096)
44.83
27.20
35.89 (4099)
35.89
Modality
Audio
Video
Dataset
LibriSpoochl
LibriSpooch3
VoxCeleb2
Metric (PSNK)
Step1
(step)
Step
Step
Best (step)
Step
Step
Best (step)
Step 4096
OL
OL
29.63
48.44 (4096)
48.4
29.63
38.61 (4096)
38.61
12.93
36.10 (4096)
36.10
CL
31.07
32.83 (256)
32.77
30.85
32.31 (32)
31.34
15.00
19.80 (32)
18.18
CL
ER
30.86
(4096)
49.83
30.42
38.49 (4096)
38.49
14.08
40.39 (4096)
40.39
EWC
30.98
32.92 (64)
32.43
30.75
32.32 (64)
30.91
15.13
20.82 (256)
19.04
CML
MER
29.27
34.86 (2048)
34,85
29.39
33, (1024)
33.07
12.78
21.80 (256)
20.49
MOL
MOL
40.74
55.76 (4096)
55.76
35.83
41.97 (4096)
41.97
28.21
41.75 (4096)
41.75
MAML+CL
41.30
43.40
37.63
36.47
37.47
28.98
28.45
30.05
25.04
MCL
OMT.
43.46
44.40 (16)
41.62
40.04
41.40 (8)
35.11
28. 14
29.16 (16)
21.59
Ours
42.09
67.58 (4096)
67.58
38.84
51.09 (4096)
51.09
29.84
47.80 (4096)
47,80
Modality
NeRF
MatrixCity-B5
MatrixCity-
Metric (PSNR)
Step
(step)
Step 1024
Step
Best (step)
Step 1024
ER
14.318
30.214 (3488)
26.339
6,956
28.869 (3984)
26.235
CL
EWC
17.241
28.811 (3904)
24.781
10.442
28.457 (3856)
26.25
MOL
22,901
30.569 (1024)
30.569
21.961
26.837 (1024)
26.837
MAML+CL
22.722
25.792 (1024)
25.792
22.253
25.218 (1024)
25.218
OMI.
23.082
26.5 (1024)
26.5
22.305
24.83 (1024)
24.83
Ours (mod)
23.885
32.712 (1024)
32,712
23.217
30.407 (1024)
30.407
Ours (MIM)
24.223
32.804 (1024)
32.804
23.341
30.761 (1024)
30.761
6 CONCLUSION
We introduced MCL-NF, a framework combining modular architecture and optimization- meta-learning for neural fields. Our approach, featuring FIM-Loss, effectively addresses catas- trophic forgetting and slow convergence continual learning. Experiments across diverse datasets show superior performance in reconstruction quality and learning speed, especially in resource- constrained environments. MCL-NF demonstrates potential for scalable, efficient neural field appli- cations in complex, dynamic data streams.
Limitations. Despite its advantages, MCL-NF faces challenges: potential performance degradation over very long, dissimilar task sequences; increased memory usage with task number growth; and possible limitations in handling abrupt, significant data distribution changes. These areas present opportunities for future research to enhance the framework's real-works applicability.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENTS
This work was supported by Samsung Advanced Institute of Technology and the Institute of Infor- mation & Communications Technology Planning & Evaluation (ITTP) grants funded by the Korea government (MSIT) (No. RS-2022-11220156 Fundamental research on continual meta-learning for quality enhancement of casual videos and their 3D metaverse transformation; No. RS-2019- 11191082, SW StarLab; No.
Artificial Intelligence Graduate School Program (Seoul National University)). and by the Institute of Information & Communications Technology Planning & Evaluation Info)--nfrrmation Technology Research Center (TTRC) grant funded by the Korea government (MSIT) (No. RS-2024-004376333. Gunhee Kim is the corresponding author.
REFERENCES
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. In CVPR, pp. 7120-7129. IEEE Computer Society, 2017.
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. In ICLR, 2019.
Mohammadamin Banayceanzade, Rasoul Mirzaiezadeh, Hosein Hasani, and Mahdich Soleymani. Generative vs. discriminative: Rethinking the meta-continual learning. In Marc' Aurelio Ran- zato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), NeurIPS, pp. 21592-21604, 2021.
D. A. Barajas-Solano and A. M. Tartakovsky. Approximate bayesian model inversion for pdes with heterogeneous and state-dependent coefficients. Journal of Computational Physics, 395:247-262, 2019. doi: jop.016f..ep.2219..66
Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O. Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn. In Giuseppe De Giacomo, Alejandro Catalá, Bistra Dilkina, Michela Milano, Senén Barro, Alberto Bugarín, and Jérôme Lang (eds.), ECAI 2020 24th European Conference Artificial Intelligence, volume 325 of Frontiers in Artificial Intelli- gence and Applications, pp. 992-1001. IOS Press, 2020.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, R. Devon Hjelm, and Aaron C. Courville. Mutual information neural estimation. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 530-539, PMLR, 2018
Yuan Cao and Quanquan Gu. Generalization error bounds of gradient descent for learning over- parameterized deep relu networks. In The Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Confer- ence, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 3349-3356, AAAI Press, 2020.
Arslan Chaudhry, Puneet Kumar Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Rie- mannian walk for incremental learning: Understanding forgetting and intransigence. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings. Part XI, volume 11215 of Lecture Notes in Computer Science, pp. 572- Springer, 2018.
Y. Chen, D. S. Menasché, and D. Towsley. To collaborate or not in distributed statistical estimation with resource constraints? 2021 55th Annual Conference on Information Sciences and Systems (CISS), 2021. doi: 10. 1001100//s0s008877.. 200.:97777.221.9022200
Yinbo Chen and Xiaolong Wang. Transformers meta-learners for implicit neural representations. In ECCV, 2022.
Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao Erik G. Learned- Miller, and Chuang Gan. Mod-squad: Designing mixtures of experts modular multi-task learners In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 1 1828-118377 IEFF, 2023.

---

Published as a conference paper at ICLR 2025
Junwoo Cho, Seungtae Nam. Daniel Rho, Jong Hwan Ko, and Eunbyung Park. Streamable neural fields. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), ECCV, pp. 595-612. Springer, 2022.
J. S. Chung, A. Nagrani, and A. Zisserman. SPEECH, 2018.
Voxceleb2: Deep speaker recognition. In INTER-
Jaeyoung Chung, Kanggeon Lee, Sungyong Baik, and Kyoung Mu Lee. Meil-nerf: Memory- efficient incremental learning of neural radiance fields, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR. pp. 248-255, IEEE Computer Society, 2009.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of gradient- based model-agnostie meta-learning algorithms. In Proceedings of the International Conference on Artificial Intelligence and Statistics, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning, 2017.
Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-leamnig with warped gradient descent. In ICLR, 2020.
P. Gong and J. Ye. Linear convergence of variance-reduced stochastic gradient without strong con- vexity. 2014. doi: 10.48550/arxiv. 1406. 1002.
Mengqi Guo, Chen Li, Hanlin Chen, and Gim Hee Lee. Incremental neural implicit representation with uncertainty- muilrtainty-filteree knowledge distillation, 2023.
Gunshi Gupta, Karmesh Yadav, and Liam Paull. Look-ahead meta learning for continual learning. In Hugo Larochelle, Marc" Aurelio Ranzato, Raia Hadsell, Maria-Florins Balcan, and Hsuan- - Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020. NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Robert A. Jacobs, Michael Jordan, Steven J. Nowlan, and Geoffrey Hinton, Adaptive mixtures of local experts. Neural Comput., 3(1):99-87, 1991.
Khurram Javed and Martha White. Meta-Meerning representations for continual learning. In NeurlPS, 2019.
S. Jiang. Relative performance of fisher information in interval estimation. 2021. doi: 10.48550/ arxiv.2107.0462
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pp. 4401 Computer Vision Foundation / IEEE, 2019.
Chiheon Kim, Doyup Lee, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Generalizable implicit neural representations via instance pattern composers. In CVPR, 2023.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 2016.
Tatsuya Konishi, Mori Kurokawa, Chihiro Ono, Zixuan Ke, Gyuhak Kim, and Bing Liu. Parameter- level soft-masking for continual learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt. Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Ma- chine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceed- ings of Machine Learning Research, pp. 17492-17505. PMLR, 2023.
Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approxi- mation for natural gradient descent. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Informa- tion Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 4158 4169, 2019.
12

---

Published as a conference paper at ICLR 2025
Yixuan Li, Lihan Jiang, Linning Xu, Yuanbo Xiangli, Zhenzhi Wang, Dahua Lin, and Bo Dai. Matrixcity: A large-scale city dataset for city-scale neural rendering and beyond. In ICCV, pp- 3182-3192. IEEE, 2023.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning. 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, December 2015.
U. Luxburg and B. Schölkopf. Statistical learning theory: models, concepts, and results. Hand- book of the History of Logic, pp. 651-706, 2011. doi: 10.1016//978-00444-5259533 7.50666-..
Zhenxing Mi and Dan Xu. Switch-nerf: Learning scene decomposition with mixture of experts for large-scale neural radiance fields. In ICLR, 2023.
B. Mildenhall, P. Srinivasan, M. T., J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms, 2018.
Bowen Pan, Yikang Shen, Haokun Liu, Mayank Mishra, Gaoyuan Zhang, Aude Oliva, Colin Raffel, and Rameswar Panda. Dense training. sparse inference: Rethinking training of mixture-of-expprts language models. CoRR, abs/2404.05567, 2024.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 2200-52.0. IEEE, 2015.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
Guillermo Valle Pérez and Ard A. Louis. Generalization bounds for deep learning. CoRR, abs/2012.04115,, 2020.
Ryan Po, Zhengyang Dong, Alexander W. Bergman, and Gordon Wetzstein. Instant continual learn- ing of neural radiance fields, 2023.
Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf: Decomposed radiance fields. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 14153-14161. Computer Vision Foundation / IEEE, 2021.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In ICCV, pp. 14315-14325. IEEE, 2021.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning learn without forgetting by maximizing transfer and minimizing inter- ference. In 7th International Conference on Learning Representations, ICLR 2019, New Or- leans. LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL htttttt///ppeeeeiieeee n Forum? Ai B...d-BigShhAttt
Antony W. Rix, John G. Beerends, Michael P. Hollier, and Andries P. Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2001, 7-11 May, 2001, Salt Palace Convention Center, Salt Lake City, Utah, USA, Proceedings, pp. 749-752. IEEE, 2001.
Anthony V. Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connect. Sci., 1995.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Greg Wayne. Experience replay for continual learning. In NeurlPS, 2019.
13

---

Published as a conference paper at ICLR 2025
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, 00s//666.046777 2016.
Yikang Shen, Zheyu Zhang, Tianyou Cao, Shawn Tan, Zhenfang Chen, and Chuang Gan. Module- former: Learning modular large language models from uncurated data. CoRR, abs/2306.04640, 2023.
Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. NewrIPS, 2020.
Jaehyeon Son, Soochan Lee, and Gunhee Kim. When meta-learning meets online and continual learning: A survey. CoRR, abs/231 1,02441, 2023, doi: 10448550/RXIV.2211 1..02241.. URL hittps://do wrw/o0.......................... 48550/ a rxi
A. Spantini, A. Solonen, T. Cui, L. Martin, L. Tenorio, and Y. Marzouk. Optimal low-rank approximations of bayesian linear inverse problems. SIAM Journal on Scientific Computing, 37: A2451-A2487, 2015. doi: 10.1 1111131119999 409/7709
Jihoon Tack, Subin Kim, Sihyun Yu, Jaeho Lee, Jinwoo Shin, and Jonathan Richard Schwarz. Learn- ing large-scale neural fields via context pruned meta-learning. In NeurlPS, 2023.
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representa- tions. In CVPR, 2021.
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben P. Mildenhall, Pratul P. Srini- vasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 8238-8248. IEEE, 2022.
Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan, Mega-nerf: Scalable construction of large-scale nerfs for virtual fly- throughs. In CVPR, pp. 12912-12921. IEFF, 2022.
Nicolas Jeyrat-Charvilne and François-Xaviee Standaert. Mutual information analysis: How, when and why? In Christophe Clavier and Kris Gaj (eds.), Cryptographic Hardware and Embed- ded Systems CHES 2009, I 11th International Workshop, Lausanne, Switzerland, September 6-9, 2009, Proceedings, volume 5747 of Lecture Notes in Computer Science, pp. 429-443. Springer, 2009.
Johannes von Oswald, Dominic Zhao, Seijin Kobayashi, Simon Schug, Massimo Caccia, Nicolas Zucchet, and João Sacramento. Learning where learn: Gradient sparsity in meta and con- tinual learning. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp- 9250-5263, 2021.
Shengze Wang, Alexey Supikov, Joshua Ratcliff, Henry Fuchs, and Ronald Azuma. Inv: Towards streaming incremental neural videos, 2023.
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE Trans. Image Process., 13(4):600-612, 2004.
A. Wilson and D. Murphey. Local e-optimality conditions for trajectory design to estimate pa- rameters in nonlinear systems. 2014 American Control Conference, 2014. doi: 10.1 1011 6858649.
Yichen Wu, Long-Kai Huang, Renzhen Wang. Deyu Meng, and Ying Wei. Meta continual learn- ing revisited: Implicitly enhancing online hessian approximation via variance reduction. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Aus- tria, May 7-11, 2024. OpenReview.net, 2024. URL htttp:://opereeiice net net/forum? Hd=TpD2aG1h00
14

---

Published as a conference paper at ICLR 2025
Z. Yan, Y. Tian, X. Shi, P. Guo, P. Wang, and H. Zha. Continual neural mapping: Learning an implicit scene representation from sequential observations. In ICCV, 2021.
X. Yang and J. Honorio. Information theoretic lower bounds for feed-forward fully-connected deep networks. 2020. doi: 10:48550/axiv.2200.077666.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In NeurlPS, 2021.
Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Ex- ploring monocular geometric cues for neural implicit surface reconstruction. NeurIPS, 2022.
S. Zhu, T. Gu, and X. Liu. Aims:average information matrix splitting. 2016. doi: 10.48550/arxiv. 1605.07646.
Luisa M. Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta- meta-learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 7693-7702. PMLR, 2019.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DETAILED PROOF OF FISHER INFORMATION MAXIMIZATION
Theorem (Fisher Information and KL-Divergence). Let ((D(0) and p( 40) be two probabil- ity distributions parameterized by 0 and 0 - AO respectively. The Beilback-Leiblee (KL) divergence between these distributions can be approximated as:
KLIn(DI0)lD(De++ 20) 2 F("
(5)
where F(0) is the Fisher Information Matrix.
Proof. We begin with the definition of KI--divergence Zhu et al. (2016):
p(D
KT(p(D10))<((10 + A0) = DD log
(6)
p(D + A0)
Expand log pD10 + A0) using Taylor series around
Jiang (201 :
( + A0) log p(D))) + A6" V log p D(0) + マ A A HgA0
where Hg is the Hessian of (D) Substituting this into the KL-divergence:
AO Vo ((lo(()))))))))ooo)))))))))())) _ AA''HHAAl
KLI(DD0)) A0) -DDDp(D)0)
The first term vanishes VVV, log p(D(0)) 0 Jiang 2021
The negative expectation of the
Hessian the Fisher Information Matrix:
F(O) - ED-epp-pCDD Ha]
Therefore, we arrive the desired approximation al. 2015):
Barajas- Solano & Tartakovsky (2019); Spantini
KLIp(D10\lo(000+a0))) 1 AOTF(O))
(10)
A.2 CONVERGENCE ANALYSIS
We now analyze the convergence properties of our Fisher Information Maximization Stochastic Gradient Descent (FIM-SGD) algorithm.
Theorem (Convergence of FIM-SGD). Under the following assumptions:
1. The expected loss function L()) Lipschitz continuous gradients Gong Ye (2014)
(0; D)] is p-strongly convex and has L-
2. The stochastic gradients are unbiased estimates of the true gradient: (f v (()) Gong & Ye (2014).
3. The variance of the stochastic gradients is bounded: Gong & Ye (2014).
- 02
4. The Fisher Information Matrix F(8) is positive definite and its eigervalues are bounded between Amin and Xmaz Wilson Murphey (2014).
16

---

Published as a conference paper at ICLR 2025
The FIM-SGD algorithm with constant learning rate n satisfying 0 expectation the optimal parameter 0
2μ Amcca
converges in
na² X2 X 2Aminf nL X
E| E| - 1011 2m + 17 12 maa 10-1 +
(11)
where is the number of iterations.
Proof. Let V = 110:- 0- be our Lyapunov function. The update rule for FIM-SGD is:
12+1 = 0. - F),()) -/((()) (VFFIM(0;
(12)
where D, is the mini-batch at iteration l. We analyze the expected decrease in V, Wilson & Murphey 2014):
Vi++ [ = E[ -V 07
(13)
((B) Vi 2n((0 F r' " (FFIM((;:
(14)
Using the μ-strong convexity of [()), we have:
(0. - 00--0))VV 2 ul10; - 1
(15)
And from the bounded eigenvalues of F(0):
(0. - ( Aminju] 02 1-
(16)
For the last term, using the L-Lipschitz continuuty of VL(0) and the bounded variance of stochastic gradients Gong & (214))
((O,) " "VCFIM(0; < ( 1 + 0²)
(17)
Combining these inequalities:
E [0:] < (1 21/mmint + 222 max) + 22 max
(18)
Taking the total expectation and applying this recursively:
n0 X2 max
E S (1 - 22/Amin/t + 22 Vo + 2)min 1 maz
(19)
which completes the proof.
A.3 GENERALIZATION BOUND
We now derive a generalization bound for our Fisher Information Maximization Loss. Theorem (Generalization Bound for FIM Loss), Let H be hypothesis class with VC-dimension and LFIM be our Fisher Information Maximization Loss. Then, with probability at least - 6, for all h E H:
(FIM(()) - (h)) so ( + log(1/8)
(20)
N
where Leras (h) is the true expected Fisher Information Maximization Loss and L (h) is the empirical Fisher Information Maximization Loss.
17

---

Published as a conference paper at ICLR 2025
Proof. We begin by noting that our Fisher weights are bounded:
1 S wy w;(0) S Afmaz
(21)
where Paaz is the maximum eigenvalue of (0) over all 0. Let W 1 Ammaz.
We can now apply the standard VC-dimension based generalization bound to our weighted loss function Yang & Honorio (2020); Luxburg & Schölkopf (20TT). For any >> 0:
Ne 32W2
P
sup (h) (FIM(h)) >e) 48(H,, 2N) exp New
(22)
where S(H. is the shattering coefficient H on sample of size 2/.. Using the fact Luxburg & Schölkopf [001] that S(H, 2N) - (2N) for hypothesis class with VC-dimension d, and setting the right-hand side to ö, we get:
(log(N/d) + (og(1
W
(23)
N
Noting that W is constant (with respect N), arrive at the stated bound Chen et al. (2021
B QUANTITATIVE RESULTS
The results in indicaae that our Meta- Peta-Cootinual Learning method, specifically Ours (MIM), gen- erally demonstrates better initial performance and faster convergence across all tested modalities. When compared to Offline Learning (OL) and Continual Learning (CL) approaches, our method shows consistently strong start, suggesting effective initialization and continual adaptation. In terms of achieving the highest PSNR (Best Step), Ours (MIM) often performs at least as well Meta-Offline Learning (MOL). with significantly fewer optimization steps. Furthermore, Ours (MIM) maintains competitive quality at later steps, indicating stability during the learning process. CL methods, the other hand, require much more optimization steps reach comparable perfor- mance levels, and even then, often fall short of Ours (MIM) in terms of reconstruction quality. This trend is consistent across image, audio, video, and NeRF datasets in resource-constraiied environ- ments where rapid training is essential. Overall, the findings suggest that Meta-Contine Learning can offer robust alternative to more resource- resource-intensive methods, without sacrificing reconstruction quality.
We implement SSIM (Structural Similarity Index Measure) (Wang al.) 2004) for the image and video domains, and PESQ (Perceptual Evaluation of Speech Quality) (Rix et al. 2001) for the audio domain. In the table, we focus on meta-continual learning specifically designed for learning to continually learn neural radiance fields. We excluded the methods that are categorized Continual Bi-level optimization (Gupta et al., 2020) yon Oswald et al. 2021: Wu et al. 2024) based on (Son) et al. 2023), since their settings differ from ours. MCL aims create initialization or strategy that quickly adapts to new tasks while retaining prior knowledge. In contrast, the latter focuses on adapting the meta-learner itself
time. Instead, we plan experiment with multiple MAML. variants eL et al. 2017 Antoniou et al. 2019 Nichol et al. 2018). The SSIM values show that our approach (Ours) maintains strong visual similarity, achieving higher scores at Step 4096 across all image and video datasets. Similarly, PESQ for the audio domain reveals that our method provides consistent perceptual quality compared others, especially for longer adaptation sequences (Step 4096). Notably, our approach outperforms baseline models in terms of both PSNR and perceptual quality metrics, highlighting its robustness in maintaining quality during sequential learning.
C ABLATION STUDY
Our ablation study compared four variants of our method: modular (mod) and Mutual Information Maximization (MIM) approaches, each with hidden dimensions of 256 and 512 Results show that
18

---

Published as a conference paper at ICLR 2025
Modality
Image
Video
Dataset
CelebA
ImageNette
VoxCeleb2
Metrie (SSIM)
Step
Bost (step)
Step 4096
Step
Best (step)
Step 4096
Step
Best (step)
Step 4096
OL
OL
0.329
0.969 (4096)
0.969
0.248
0.98 (4096)
0.98
0.347
0.967 (4096)
0.967
CL
0.313
0.519 (64)
0.463
0,248
0.526 (128)
0.477
0.365
0.611 (64)
0.493
CL
ER
0.316
0.547 (64)
0.504
0.273
0.555 (128)
0.489
0.438
0.629 (64)
0.508
EWC
0.321
0.63 (256)
0.503
0.251
0.617 (256)
0.475
0.369
0.648 (256)
0.526
CML
MER
0.332
0.711 (256)
0.633
0.251
0.674 (256)
0.581
0.348
0.68 (256)
0.603
MOL
MOL
0.82
0.995 (4096)
0.995
0.729
0.997 (4096)
0.997
0.807
0.991 (4096)
0.991
MAML+CL
0.822
0.92 (64)
0.816
0.742
0.884 (32)
0.789
0.815
0.888 (32)
0.773
MCL
OML
0.796
0.884 (32)
0.65
0.696
0.81 (16)
0.553
0.796
0.855 (16)
0.599
Ours
0.817
0.995 (4096)
0.995
0.751
0.996 (4096)
0.996
0.857
0.996 (4096)
0.996
Modaliit
Audio
LibriSpeech1
Metric (PESQ) 1
Step 1
Best (snap)
Step 4096
OL
1.03
1.82 (2018)
1.82
CL
1.02
1.12 (2018)
1,12
1.02
1.13 (4096)
EWC
1.02
1.12 (2018)
1,12
CML
1.03
MOL
1.33
2,48
MAML
1.37
1.62 (32)
1.59
MCL
OML
1.34
1.62 (512)
1.59
Ours
1.3
3.73 (4096)
3.73
Table 2: Evaluation results for various methods across multiple modalities (Image, Video, Audio) and datasets. Metrics such as SSIM (for Image and Video) and PESQ (for Audio) are reported for different steps of optimization. Best results are highlighted in bold.
MIM consistently outperforms the modular approach across all steps, regardless of dimension size, indicating its significant contribution the method's effectiveness. Increasing the hidden dimension from 256 to 512 generally improves performance, especially in later steps, with the improvement more pronounced in the modular approach. Notably. the MIM variant with 256 dimensions of- ten outperforms the modular variant with 512 dimensions, suggesting MIM achieves better results with fewer parameters. All variants demonstrate consistent improvement over steps, showcasing the method's continual learning capability. The performance gap between variants widens as the number steps increases, highlighting the benefits MIM and larger hidden dimensions in longer sequences. These findings underscore the importance of both the MIM component and appropriate dimensionality in our method's ' overall performance.
To demonstrate the scalability and effectiveness of the MIM method over longer sequences, expanded the number of tasks to and 10 The metrics were evaluated at steps doubling from to 1024. Overall, "Ours (MIM)" consistently outperforms "Ours (mod)" in terms of PSNR, especially at later steps. This suggests that the Fisher Information Maximization (MIM) strategy contributes improved reconstruction quality during extended optimization. This trend observed in both the 5-task and 10-task scenarios.
D IESOUUTION-RDDUCE SEQUENTIAL OPTIMIZATION
In this experimental setup, we focus on dividing images into lower resolutions. Specifically, an n x image is split into four smaller sub-images of resolution (n//2 02/ x (n/ where each sub-image maintains the same height and width as the original image but with half the resolution. These smaller sub-ine sub-images are then optimized sequentially, enabling faster convergence by processing less complex data in each optimization step. Importantly, the total image area remains unchanged, and only the resolution is reduced while preserving essential content and structure for training.
19

---

Published as a conference paper at ICLR 2025
Modality
Image
Video
Dataset
CelebA
ImageNette
VoxCeleb2
Metrie (PSNR)
Step
Bost (step)
Step 4096
Step
Best (step)
Step 4096
Step
Best (step)
Step 4096
OL
OL
11.84
38.2 (4096)
38.2
12.19
38.87 (4096)
38.87
13
37.83 (4096)
37.83
CL
17.63
26.3 (4096)
26.3
16.7
26.67 (2048)
26.67
17.53
33.77 (1024)
33.39
CL
ER
17.63
26.5 (1024)
26.47
16.7
26.69 (2048)
26.66
17:33
33.79 (1024)
33.44
EWC
18.21
28.12 (4096)
28. 12
17.01
26.88 (1024)
26.75
17.96
34,18 (4096)
34.18
CML
MER
12.21
26.62 (4096)
26.62
12.56
26.62 (2048)
26.61
13.03
33,94 (4096)
33.94
MOL
MOL
29.28
46.36 36 (4096)
46.36
24.52
48.55 (4096)
48.55
28.9
44.21 (4096)
44.21
MAML+CL
33,12
34.12
32.6
27.6
28.24
26.83
33.65
36.3 (16)
34,09
MCL
OML
32,55
33.69 (4)
33.14
26.96
27.53 (2)
26.85
34.05
36.67 (16)
35.19
Ours
24.22
48.94 (4096)
48.94
20.62
46.81 (4096)
46.81
24.81
42.31 (4096)
42.31
Modality
Image
Video
Dataset
CelebA
ImageNette
VoxCeleb2
Metric (SSIM) f
Step
Best (step)
Step 4096
Step
Best (step)
Step 4096
Step
Best (step)
Step 4096
OL
OL
0.329
0.969 (4096)
0.969
0.248
0.98 (4096)
0.98
0.347
0.967 (4096)
0.967
CL
0.469
0.915 (4096)
0.915
0.344
0.892 (4096)
0.892
0.438
0.93 (1024)
0.918
CL
ER
0.469
0.916 (4096)
0.916
0,344
0.893 (2048)
0.892
0.438
0.93 (1024)
0.919
EWC
0.483
(4096)
0.926
0.357
0.892 (2048)
0.891
0.462
0.935 (2048)
0.932
CML
MER
0.343
0.914 (4096)
0.914
0.258
0.882 (4096)
0.882
0.356
0.93 (4096)
0.93
MOL
MOL
0.82
0.995 (4096)
0.995
0.729
0.997 (4096)
0.997
0.807
0.991 (4096)
0.991
MAML+CL
0.912
0.933 (64)
0.902
0.852
0,878 (32)
0.826
0.913
0.95 (16)
0.912
MCL
OML
0.892
0.925
0.915
0.836
0.822
0.913
(16)
Ours
0.669
0.996 (4096)
0.996
0.532
0.996 (4096)
0.996
0.659
0.984 (4096)
0.984
Table 3: This table presents the results of the experiment on resolution-reduced sequential opti- mization for image, and video modalities. The table compares the performance of various methods, including "Ours," in terms of image and video reconstruction quality at different optimization steps (Step 1, Best Step, and Step 4096).
Steps
1
4
16
32
64
128
256
$12
Ours (mod; 256)
23.885
24.205
24.334
25.436
26.332
27.342
28.663
30.089
31.124
32,088
Ours (MIM; 256)
24.223
24.64
24.655
25.471
26.387
27.362
29.031
30.281
31.186
32.437
Ours (mod; 512)
24.02
23.906
24.502
25.445
26.29
27.94
29,354
30.231
31.403
32.486
Ours (MIM; 512)
24.124
24.176
24.813
25.555
26.399
27.859
29.115
30.301
31.529
32.63
Table 4: Performance comparison of our methods across different model configurations (rows) and optimization steps (columns). In each row, 'mod' refers to modularization, and "MIM' modular- ization with mutual information maximization. The number next to "mod' or 'MIM" indicates the hidden dimension and the ray batch size for each iteration.
The experimental results demonstrate that our resolution-reduced sequential optimization approach consistently outperforms other methods across image and video modalities. For the image datasets (CelebA, ImageNette), our method ("Ours")) achieves the highest SSIM scores of 0.995 at step 4096, surpassing upper bound approaches such as MOL. Similarly, for video data (VoxCeleb2), "Ours" yields the best SSIM score of 0.996 at step 4096, again outperforming competitors. These results highlight the effectiveness abling faster convergence and improved reconstruction quality during extended optimization steps. The advantage of our method becomes more apparent in later steps, where it consistently delivers better results in terms of SSIM, confirming the benefits of the proposed optimization strategy.
reducing image resolution while maintaining essential content, en-
20

---

Published as a conference paper at ICLR 2025
of Tasks
Method
Step
Slep
Step
Step
Step
Stop 32
Step 61
Step 128
Step 256
Step 512
Step 1021
Ours (modd
24,072
24.155
24.484
24,096
25.072
25.677
26,045
28,075
29.603
30.836
31.687
Ours (MIM)
24.143
25.61
31,005
31,822
Ours (mod)
23.415
23.374
23.18
24.001
24.295
24.703
26.037
27.02
28:35
29.76
31.009
Ours (MIM)
23.493
23.548
23.783
24.119
24.464
24.793
26.238
27.667
28.886
29.71
31.098
Table 5: Performance metrics (PSNR) for "Ours (mod)" and "Ours (MIM) across different opti- mization steps, for 5 and 10 tasks. The metries are evaluated at steps doubling from to 1024.
E COMPUTATIONAL COST ANALYSIS
We note that the actual computational cost can vary significantly based on the specific hardware and the software implementation used. In this work, evaluate the computational cost using our PyTorch implementation, conducted on NVIDIA TITAN X Pascal GPUs which have 12 GB of VRAM.
Table O presents a comparison of the number of parameters, GPU memory consumption (in MiB), test-time optimization speed (measured in episodes per second), and reconstruction quality (PSNR) on the CelebA dataset. An episode corresponds to single image with a resolution of 180 x 180, and each episode consists of 4 tasks, with each task corresponding to an image patch of size 180 45, The reported test-time optimization costs are based batch size of 1, with 128 optimization steps per task, performed across a total tasks.
Our proposed method demonstrates compelling balance of performance and efficiency in the meta- continual learning landscape. Despite a marginal increase in parameters (201.23K compared to 198.91K for other methods), approach achieves the highest PSNR of 30.10, surpassing all other techniques including offline and meta-offline learning. Notably, it accomplishes this while utilizing the least GPU memory (787 MiB), significantly lower than the next best performer. The test-time optimization (TTO) speed of 68.27 ep/s is competitive, outpacing several other methods including traditional continual learning approaches. This combination of top-tier performance (PSNR) with minimal memory footprint showcases our method's efficiency in resource utilization. While not the fastest in terms of TTO speed, the substantial gains in output quality and memory efficiency make our approach particularly valuable for applications where high-quality results and resource constraints are critical factors, especially in continual learning scenarios.
Category
Method
Parameters (1)
GPU Memory (L)
TTO Speed (f)
PSNR (1)
Offline Leaming
OL
198.91K
1068 MiB
65.51 ep/s
25.18
CL
198.91K
989 MiB
59.90 ep/s
15.55
Continual Learning
ER
198.91K
991 MiB
37.20 ep/s
30.83
EWC
198.91K
968 MiB
70.45 epís
19.70
Continual Meta-Learning
MER
198.91K
890 MiB
9.27 ep/s
21.69
Meta-Ofline Learning
MOL
198.91K
1044 MiB
79.02 ep/s
37.69
MAML+CL
198.91K
MiB
65.12 ep's
33.04
Meta-Continual Learning
OML
198.91K
1011 MiB
160.29 ep/s
30.85
Ours
201.23K
787 MiB
68.27 ep/s
38.10
Table 6: Computational cost comparison.
Method Un-modularized
Number of Networks
Hidden Dimension
Number Layers
Parameter Count (with Bias)
512
10
2.628.099 x659, 459 = 2, 2,637,830
Modularized
4
256
10
Table 7: Comparison of parameter counts for different network configurations, including un- modularized and modularized versions with bias terms included.
21

---

Published as a conference paper at ICLR 2025
F VISUALIZATIONS
Our proposed method demonstrates superior performance across diverse image datasets, consis- tendy achieving the highest PSNR scores. In the CelebA dataset, our approach attains a PSNR of 45.4, significantly outperforming other methods, with the nearest competitor (ER) at 43,90. The vi- sual quality of our result closely matches the ground truth, preserving fine facial details and overall image clarity. For ImageNette, our method again leads with a PSNR of 44.83, showcasing its ability handle complex, non-facial images with high fidelity. In the challenging FFHQ dataset, which features high-resolution facial images, our method achieves PSNR of 35.89, surpassing all other approaches. Notably, our results consistently show better preservation of intricate details, accurate color reproduction, and reduced artifacts compared to other methods. This performance is particu- larly impressive given that maintains high quality across different image types and complexities, from facial features intricate object details, demonstrating the robustness and versatility of our approach in continual learning scenarios for image reconstruction tasks.
GT
CL
ER
EWC
MER
MAML+CL
OML
Ours
PSNR
15.60
43.90
20.08
21.97
33.22
31.64
45,4
(a) CelebA
PSNR
16.76
45.74
19.77
21.31
29.21
26.66
44.83
(b) ImageNerte
PSNR
16.35
35.30
21.29
23.09
29.91
28.65
35.89
(c) FFHQ
Figure 3:
Qualitative results of image reconstruction. The first column represents the ground truth, while the remaining columns show the reconstruction results from different methods. The numbers below each image indicate the PSNR with respect to the ground truth. These images correspond to the Best results (those with the highest PSNR among to 4096 steps), presented in Table [ For detailed results, please refer Table
22

---

Published as a conference paper at ICLR 2025
Step
Step
Step 4
Step 8
Step 16
Step 32
Step 64
Step 128
Step 256
Step 512
Step 1024
Step 2048
Step 4096
GT
(a) mod
Step
Step 2
Step 4
Step 8
Step 16
Step 32
Step 64
Step 128
Step 256
Step 512
Step 1024
Step 2048
Step 4096
GT
(b) mim
Step 1
Step 2
Step 4
Step
Step 16
Step 32
Step 64
Step 128
Step 256
Step 512
Step 1024
Step 2048
Step 4096
GT
(8) mod
Step
Step
Step 4
Step
Step 16
Step 32
Step 64
Step 128
Step 256
Step 512
Step 1024
Step 2048
Step 4096
(b) mim
Figure 4: Visualization of Reconstruction Progression Over Steps for "mod" and "mim" Methods. (a) shows the progression of reconstruction for the "mod" (modularized) version, and (b) shows the "mim" (modularized with Fisher Information Maximization) version. Each row presents the reconstructed scene at different optimization steps (Step I to Step 4096) along with the ground truth (GT). The visualization demonstrates how the quality of reconstructed details gradually improves as the number of steps increases, highlighting the efficiency of both methods in learning the underlying structure of the scene, with "mim" exhibiting more rapid refinement.
23


Reviews:
[{"review_id": "8r9k0ACfr3", "reviewer": "Reviewer_pV54", "summary": "Briefly, this paper presents a strategy to continually and rapidly learn neural fields in a meta-learning manner. The paper further introduces a fisher information maximization loss for neural radiance fields. The proposed method advances in leading to no performance degradation incurred by forgetting during test-time by synergizing modular architecture with meta-learning.", "questions": "Please refer to the Weaknesses section.\n\nThere is a wrong citation in either Line 322 (EWC Chaudhry et al., 2018) or Line 402 (EWC (Kirkpatrick et al., 2016) ). Actually, Chaudhry et al., 2018 denote their method as EWC++.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well written and easy to follow. \n+ Extensive and comprehensive experiments demonstrate the effectiveness of the proposed method.", "weaknesses": "+ The technical novelty of the proposed method seems to be marginal since the authors directly employ the existing techniques. For instance, Fisher Information against catastrophic forgetting has already been proposed by Kirkpatrick et al., 2016. The authors do discuss the relation to this method and claim that their proposed FIM loss operates at the sample level rather than the parameter level. However, it is difficult to identify the advantage of the proposed FIM loss over the parameter-level approaches in CL (Chaudhry et al., 2018; Konishi et al., 2023) without fair experimental comparison and detailed discussion.\n\n+ More detailed discussions and analyses in Table 1 are required to demonstrate the contribution of the proposed method. \n\n+ More recent state-of-the-art methods should be included for comparison to demonstrate the superiority of the proposed method. For instance, missing some SOTA methods, e.g., (Chung et al., 2022 and Po et al., 2023), for comparison on the MatrixCity dataset in Table 1.\n\n+ More ablation studies in the main paper are required to demonstrate the contribution of the main component (i.e., the FIM loss) of the proposed method in the main paper. The authors only provide some results about the mod and MIM  with two different hidden dimensions in Appendix Table 2. \n\nReference:\nPo et al., Instant Continual Learning of Neural Radiance Fields, In ICCVW 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b5CcDQrlkW", "reviewer": "Reviewer_6fGr", "summary": "The problem of continual learning in neural fields is an important topic. This work focuses on addressing the issues of catastrophic forgetting and slow convergence in existing approaches by proposing a new paradigm called MCL-NF. It introduces the Fisher Information Maximization loss. The experiments demonstrate that the proposed method can achieve satisfactory results.", "questions": "Please kindly refer to the above weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThis paper is well-written.\n2.\tThis work combines meta-learning and continual learning paradigms into neural field training to address the challenges of catastrophic forgetting and slow convergence, which is interesting.\n3.\tThis paper demonstrates the advantages of their method in experiments, which alleviates the issue of slow convergence to some extent.", "weaknesses": "1.\tIn the experimental section, the limited scenarios are my main concern, which makes me worry about the limitations of the method.\n2.\tThe paper uses a large number of quantitative metrics but lacks qualitative comparisons, especially for neural radiance fields.\n3.\tIn neural fields, some incremental learning methods [1] might also need to be discussed, but they are not mentioned in the article.\n\n[1] Zhang, L., Li, M., Chen, C., & Xu, J. (2023). IL-NeRF: Incremental Learning for Neural Radiance Fields with Camera Pose Alignment. arXiv preprint arXiv:2312.05748.\n\n4.  The paper is recommended to discuss the differences between continuous radiance fields and traditional continual learning. You know, issues like catastrophic forgetting and slow convergence have long been a focus of continual learning researchers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sFLa2KpZtd", "reviewer": "Reviewer_R5BD", "summary": "The paper introduces Meta-Continual Learning of Neural Fields (MCL-NF), a framework merging modular neural architectures with meta-learning strategies to support continuous and adaptive learning of neural fields. The proposed approach aims to address the challenges in traditional continual learning for neural fields, such as catastrophic forgetting and slow convergence. The authors implemented Fisher Information Maximization loss (FIM loss) to optimize learning by emphasizing more informative samples, achieving improved generalization and learning speed. The paper’s extensive experiments across various tasks including image, audio, video and NeRF-based view synthesis, highlight the performance advantages of MCL-NF over existing methods. They offer an efficient and scalable solution suitable for large-scale and resource-constrained environments, advancing the capabilities of neural fields in sequential learning settings.", "questions": "1. Could additional strategies beyond Fisher Information, such as curriculum learning, improve MCL-NF's adaptability and reduce reliance on specific meta-learning techniques?\n2. Should the paper also use different techniques of splitting the image than cropping into patches? The idea behind the method is based on that, and perhaps applying Fourier Transformations would be beneficial.\n3. Could alternative loss functions achieve similar sample prioritization for continual learning? If so, what criteria would make FIM the preferred choice?\n4. As far as I understand, we treat each image separately (batch size 1). How do we find the appropriate number of coordinates?\n5. What insights can this setting bring to the continual learning community?\n6. In the methods compared without modularization, do these approaches utilize the full MLP network capacity to match the parameter count of the modularized versions?\n7. The Fisher Information Maximization loss introduces a new hyperparameter. Could you provide more details on how this hyperparameter was selected and discuss its sensitivity and impact on the model's performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. MCL-NF’s combination of modular architecture and optimization based meta-learning is innovative and meets the adaptability needs of neural fields. \n2. The method mitigates catastrophic forgetting through modularization and shared initialization, without the need for experience replay that can be resource-intensive. The modular approach within meta-continual learning sounds pretty compelling.\n3. The FIM loss implementation is theoretically sound, supported by convergence guarantees, and introduces a detailed weighting mechanism for learning stability and efficiency. \n4. The paper provides thorough experimental results, showcasing the model’s performance on various datasets, supporting the robustness of MCL-NF. The concept of evaluating the method in different modalities definitely deserves recognition. \n5. This work helps understand the broader scope of how meta-learning techniques could be integrated successfully with frameworks for continual learning, particularly in the context of neural fields. This can help further research to consider similar synergies between learning paradigms, which may result in even more powerful and generalizable machine learning models.\n6. The paper is well-organized and written in a clear, academic tone. It is suitably embedded in the meta-continual learning world  with relevant citations. The authors show their deep understanding of the topic by presenting the setting, highlighting its motivation and organizing the experiments in a structured way.", "weaknesses": "1. The setting, at least in the image domain, seems artificial - dividing the images into four patches (regardless of the size) seems to raise some concerns. It would be desirable to present an ablation study on it.\n2. From a continual learning perspective, this setting is highly confusing. It should be explicitly described how the data is selected, processed and how the model is evaluated.\n3. The paper lacks more recent methods in the main table. And even if MAML is a well-known method, it is quite outdated, with lots of recent enhancements such as La-MAML [1] for continual learning.  Moreover, the comparison is mostly made against methods in a different setting with only two methods in MCL (and MAML was published in 2017).\n4. And even in different settings (not MCL), the papers are well-established, but some of them nowhere near the current state-of-the-art. There is no elaboration how this approach differs from recently presented settings/methods such as [2]. And even if they are mentioned, such as ANML they are not compared against.\n5. Experimental results show the performance with a number of continual tasks set to only four, which raises concerns regarding the method’s scalability.\n6. These experimental results are largely dependent on PSNR as the main metric of reconstruction quality. Additional metrics, such as Structural Similarity Index (SSIM) for image and video tasks, or perceptual evaluation metrics for audio, would provide a more comprehensive validation of MCL-NF’s performance.\n\nMinor issues:\nLine 155 - Redundant closing bracket", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9GrPcRlXTs", "reviewer": "Reviewer_381t", "summary": "In this paper, the authors propose a new problem setting and strategy to introduce meta-continual learning in neural fields. They conduct extensive experiments to demonstrate that their approach outperforms baseline methods.", "questions": "See Weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tExperiments on various datasets are conducted to validate the effectiveness of the method.\n2.\tThe method is general and could be applied to many different applications.", "weaknesses": "1.\tThe implementation and comparison are conducted based on methods published in 2021 and 2022. Only SwitchNeRF was published in ICLR 2023, which was also long ago. It is kindly suggested to conduct experiments on more recent methods to evaluate the proposed approach. I would like to see some comparative experiments with [a].\n2.\tI would expect more visualization in the experiments, for example, comparison of view synthesis on MatrixCity dataset.\n3.\tThe experiments should include some commonly used evaluation metrics, such as SSIM.\n\n[a] GF-NeRF: Global-guided Focal Neural Radiance Field for Large-scale Scene Rendering, WACV 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)