Learning Large-scale Neural Fields via Context Pruned Meta-Learning
Authors: Jihoon Tack
Subin Kim
Sihyun Yu
Jaeho Lee
Jinwoo Shin
Jonathan Richard Schwarz
Venue: NeurIPS 2023 poster (2023)
Keywords: Meta-Learning
Efficient Meta-Learning
Neural Fields
Implicit Neural Representations
Data Pruning
TL;DR: We propose an efficient meta-learning framework for scalable neural fields learning that involves online data pruning of the context set.

Abstract: We introduce an efficient optimization-based meta-learning technique for large-scale neural field training by realizing significant memory savings through automated online context point selection. This is achieved by focusing each learning step on the subset of data with the highest expected immediate improvement in model quality, resulting in the almost instantaneous modeling of global structure and subsequent refinement of high-frequency details. We further improve the quality of our meta-learned initialization by introducing a bootstrap correction resulting in the minimization of any error introduced by reduced context sets while simultaneously mitigating the well-known myopia of optimization-based meta-learning. Finally, we show how gradient re-scaling at meta-test time allows the learning of extremely high-quality neural fields in significantly shortened optimization procedures. Our framework is model-agnostic, intuitive, straightforward to implement, and shows significant reconstruction improvements for a wide range of signals. We provide an extensive empirical evaluation on nine datasets across multiple multiple modalities, demonstrating state-of-the-art results while providing additional insight through careful analysis of the algorithmic components constituting our method. Code is available at https://github.com/jihoontack/GradNCP

Learning Large-scale Neural Fields via Context Pruned Meta-Learning
Jihoon Tack', Subin Kim', Sihyun Yu', Jaeho Lee", Jinwoo Shin' Jonathan Schwarz²
Korea Advanced Institute of Science and Technology
Pohang University of Science and Technology
University College London
{ siihotaack subin-kim, sihyun. yu, j.w.joossss(aaiistt ac. ac.ke kr
j jaeho. lee@postech.... ac. kr, schwarzjndgmail.. com
Abstract
We introduce an efficient optimization- based meta-learning technique for large- scale neural field training by realizing significant memory savings through auto- mated online context point selection. This is achieved by focusing each learning step the subset of data with the highest expected immediate improvement in model quality, resulting in the almost instantaneous modeling of global structure and subsequent refinement of nigh-frequency details. We further improve the qual- ity our meta-learned initialization by introducing a bootstrap correction resulting in the minimization of any error introduced by reduced context sets while simulta- meta-learning.
neously mitigating the well-known myopia of optimization-baal Finally, show how gradient re-scaling meta-test time allows the learning of extremely high-quality neural fields in significantly shortened optimization proce- dures. Our framework model- aggnosii,, intuitive, straightforward and shows significant reconstruction improvements for wide range of signals. We provide an extensive empirical evaluation on nine datasets across multiple multiple modalities, demonstrating state-of-the-an art results while providing additional insight through careful analysis of the algorithmic components constituting our method. GradlICP
implement,
Code is available at https: / /ttp//giiit github. com/
Introduction
Neural fields (NFs), also known as implicit neural representations (INRs), have emerged as new paradigm for representing complex signals continuous functions (e.g., the implicit pixel coordinate to feature mapping (x, y) + (r,, g, b) of a 2D-image) parameterized by neural networks [43, 60]. In contrast to multi-dimensiioal array representations, this approach has the benefits of allowing querying at arbitrary resolutions or from arbitrary viewpoints [39, 52], removing modality-agnostic
architectural choices for downstream tasks (e.g., compression [10, 48, 49), classification [11, 2] or generative modeling [55, 68)), representing a more compact parameterization and providing an elegant framework for dealing with data unsuitable for discretization (e.g., vector fields). Thus, the approach has shown great promise in modeling diverse forms of data, including images [38], videos [5], 3D scenes [43], voxels [11], climate data [25], and audio [12].
Unfortunately, fitting even single NF to a signal is often prohibitively expensive (c.g., more than half GPU day for single HD video [30]), limiting the field's 's ability learn NFs at scale for large set signals. To alleviate this issuc, optimization based meta-learning has recently emerged [53, 61], demonstrating rapid improvements in learning efficiency and thus quickly becoming standard technique in NF research. Meta-learning allows signals to be represented at good quality within only handful of optimization steps by replacing tabula rasa optimization from hand picked
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Selected Context
Zoomed in
Reconst. Difference
Ground truth
1=1 k=
k 16
==1 k 16
Figure 1: Online context point selection learns interpretable curricula: Visualization of selected con- text points (top), residuals relative to the original signal (middle), and the reconstruction (bottom) of GradNCP trained on ImageNet. Selected coordinates are highlighted in red (top) where our selection scheme focuses on only 25% of available points at each of the adaptation steps. GradNCP first focuses modeling global structure while subsequently learning nigh-rreueecy details.
with optimized initialization closer to minima reachable within few steps of gradient descent. This is achieved by defining a second-order optimization problem over the initialization and other meta- pettaparammttes, computed through the iterative learning process of individual tasks. all started from a shared initialization [1, 15]. In the context of this paper, the inner learning task is the fitting of an NF to specific signal (c.g.. a single audio clip), while the outer problem corresponds to learning an initialization suitable for a whole set of signals of the same type (e.g., an entire speech dataset).
While promising in principle, existing optimization-bann lems of their own, primarily poor scaling with the dataset size (often called context ser) used in an inner learning task. This issue is particularly severe for NFs, with single 1024 x 1024 image interpreted as a context set of well over a million function evaluations (i.e.. ((x, y), (r, g, b)) pairs), forming dataset comparable in scale to ImageNet [8]. This is problematic since firstly, existing techniques are designed for the entire dataset be used in each inner optimization step, and secondly, the number of necessary inner steps required for adequate performance is known to increase with the context size. As both the computational graph and intermediate computations (c.g., gradients and activations) are stored in memory for back poopppopaagttinn w.r.t. to the parameters of the outer problem, meta-learning at scale quickly becomes prohibitively expensive, particularly in terms of memory requirements. Moreover, default optimization-bu based meta- weeannee is known to suffer from myopia [16, 66]. i.e., the tendency for plasticity to drastically decrease when the inner learning problem is run for more than the number of steps the initialization was optimized for. This strongly decreases the appeal of meta-learning when extremely high-quality NF representations are desired.
meta- schemes suffer from prob-
In this paper, investigate whether this high computational burden can be alleviated by changing the form of the inner learning problem. In particular, we draw inspiration from curriculum learning [21] and recent interest in data pruning [44, 58], selecting a relevant subset of data points non-uniformly at each inner learning iteration to reduce the memory, and more crucially, reducing myopia through long horizon optimization made by the saved memory.
Contributions. We present a novel optimization- baased meta-learnigg framework, coined Gradient Norm-based Contexi Pruning (GradNCP), designed for large-scale problems and thus particularly suitable for learning complex NFs. Specifically, GradNCP involves online data pruning of the context set based on new score, making memory efficiency the primary objective of our algorithmic design. By focusing learning on the most important samples, GradNCP effectively implements an automated curriculum, rapidly extracting the global structure of the signal before dedicating additional iterations to the representation of high-frequency details, resembling a hand-crafted technique for efficient NF training in previous literature [33] (see Figure I for illustration of this phenomenon). As this allows the computational graph to be released for all unused context points, the resulting memory savings enable longer adaptation horizons leading to improved meta-in inttiaiiiaais and crucially, make it possible to train on nigher-dimensional signals.
To realize efficient context pruning, we argue for two key considerations: (a) the minimization of information loss caused by the context pruning by careful selection based on per-example scoring statistics (b) the correction of any remaining performance degradation and avoidance of the afore-

---

Re-zann Chigh
Re-rank Crigh
Re-rank Chigh
Cral1
Secord-order
First brrdr
0o
01
OK-1
Ox
K+L
setate
stopgrad
Case (0x; Cruza) + Ap(Ox: x+L)
Figure 2: Computational diagram of GradNCP. A meta-learned initialization 6n adapted for K steps to obtain 0x, re-ranking and pruning the context set Chigh at each step for memory efficiency. Subsequently, we obtain bootstrapped target 0 through L additional steps of (reduced-memory) optimization using the fial context set Crai. Meta-optimization then minimizes both reconstruction error and the distance between the two parameter states. Op is thus updated to allow minimization of this distance in the original K steps, correcting for context pruning and reducing myopia.
mentioned myopia phenomenon, allowing for longer optimization horizons using the full context at test time (when memory requirements are lower). To this end, we propose the following techniques:
1. Online automated context pruning: As large datasets involve a range of information content in its examples (e.g., in long-tail distributions), by focusing each training step on the examples with highest expected model improvement. To achieve this, we approximate the overall effect of updating the model on each example by the norm of the last layer gradient. an extremely cheap operation that results in virtually indistinguishable performance when compared to significantly more expensive methods while outperforming random sub-sampling by large margin. Furthermore, we show how this technique is particularly suitable for meta-learning, demonstrating how the learning dynamics increase the correlation between the full gradient and our last layer approximation when compared to the random initialization.
hypothesize that learning can be greatly accelerated
2. Bootstrap correction: To counteract possible information loss introduced by the context pruning, propose to nudge parameters obtained through optimization with a pruned context set to be as close as possible to an informative target obtained using the full set. We achieve this by constructing the target through continued adaptation using the full context and then defining meta-loss involving the minimization of a distance between the two parameters. This acts both as a correction term while also reducing myopia, enabling advanced test-time adaptation. Importantly, this correction introduces significantly less computational overhead than simply extending meta- learning, as the target is deliberately generated without the need save any intermediate gradients (i.e., meta-optimiization is oblivious to the bootstrap generating procedure).
3. Test-time gradient scaling: Finally, we introduce gradient scaling technique that allows the use of full context sets at meta-test time (when memory consumption is a significantly smaller concern) enabling the learning of the highest quality NFs while continuing to benefit from significant speed-ups afforded by meta-learning.
We verify the efficacy of GradNCP through extensive evaluations on multiple data modalities, including image, video, audio, and manifold datasets. We exhibit that our GradNCPconsistenlly and significantly outperforms previous meta-learning methods. For instance, measured by peak signal-to-noise ratio (PSNR), GradNCP improves prior state-of-the thetthe-art results by 38.28 - 40.60 on CelebA [36], and 28.86 35.28 on UCF- [57]. GradNCP can also lead accelerated training, reaching the same performance as existing meta- meta-learning methods whiled reducing wall-clock time by 45% Moreover, we show that GradNCP can meta-learn on extremely high-dimensional signals, e.g., 256x 256x videos, where prior works suffer from memory shortage on the same machine.
2 GradNCP: Meta-Learning via Gradient Norm-based Context Pruning
2.1 Meta-Learning Neural Fields
We consider the problem of learning neural fields (NFs) for N signals S SN by taking a functional view of each signal s, represented by a context set c(1) = ((x,, consisting of M coordinate- value pairs (x) with X, _ and y, E RD For each signal s,, aim to
3

---

find parameters B($) for the NF faco R. + RD which reconstructs when evaluated on the full coordinate set (we also consider the generalization setup in Section 3.3). To learn NF, we minimize
the mean-squared error (MSE) loss, i.c., ((0 = CYS (0; C+)) == 1 Σ5= (lf)(x)) y; y
M
However, this requires a large number of optimization steps (on the order of 10ks for even moderately complex signals) thus limiting the development of NFs for large databases.
popular approach for this acceleration [61] relies on model-agnostic meta-learning (MAML, [15]) by defining an optimization process over the initialization 8g with given step size a > 0:
N
0o- = arg min do
e(0o a E(Oo; C(1),
N
where the inner optimization is iterated for K steps (typically K << 100) before a vetaa-optiizztion step w.r.t 8g is taken. While mini-batch SGD is commonly used for the optimization w.r.t 6g the entire context set [c(() = M (for NFs in the order of 100ks) is required for the update in each of the inner K steps legacy of MAML's predominant use in few-shot learning) with both intermediate computations and the computational graph maintained in memory, causing significant memory concerns. Unfortunately, both simple sub-sampling of C69 at each step or common first-order simplifications of the MAML objective [15, 41] have been shown to significantly under-perform in the NF context [11, 12], thus leaving practitioners with no easy solution, forcing a sub-optimal reduction of K to single digits or abandonment of the meta-learning paradigm altogether.
2.2 Gradient Norm-based Online Context Pruning
To alleviate this memory burden, we propose an online context pruning strategy, resulting in the choice of a context subset Chggh C Crun at each step l according to some scoring function Rg (x, y) evaluated on each example yx.y) E Cruz- We denote this pruned context set high to distinguish from the full context Crua1 and define Chigo the >/Crun elements with highest R2 score. Here 7 E (0, 1) is a hyper-parameter controlling the trade-off between (expected) performance and memory demands. We postulate that [x y) rank examples according to their effect on the overall loss when being used to update the model in the inner loop of Eq. (I). hypothesizing that a suitable R2 ought to improve on random sub-sampling.
(lopK(Czax)) Rg, 7) where TopK is a function returning
Estimating sample importance via one-step adaptation. The most direct but highly impractical method full-filing the requirement posed on R2 is the curriculum-learning metric target-prediction R (x, y) E(k,, C yx, E(6'., C (x,y)), i.e. the direct measurement of the reduction in loss on all other training examples after updating the full network &g to 8% using y), thus requiring O(2M + MMM - 1)) O(M2) operations (A forward and backward pass per example and evaluation on all other M - 1 examples) rendering this metric inconsistent with the primary objectives of this work. A simpler alternative is the related self- prediction gain (SPG) (x. y) = E(0x, {x, learning progress only on the example used update the model, reducing the operations to linear complexity (3MM) (forward pass, model update. another forward pass). While undoubtedly a simplification, this metric has increased appeal for reconstructing NFs where we do not require generalization to previously unseen data in a fashion. Nevertheless, despite significant complexity reduction relative to RTPG the 3M operations of R2 remain impractical.
gain (TPG) [21], defined R.
()) approximating 12/1PG by measuring
Motivated by recent studies that emphasize the critical role of the last output layer in meta-learning NFs [70, 48, 49] and recognizing the relative inexpensiveness of the operation, we instead focus on an update 0% from Og involving only the last layer, leading to ((M) complexity. We then further enhance efficiency by using a first-order Taylor approximation, showing SPG under this update be well upproximated by the squared gradient norm of the last layer. Specifically, letting Øluse be the set of parameters excluding the final layer and gk = [0, Vw,4, l, Vo, E a padded gradient involving zeros for all but the last layer's 's weights W₁ and biases by we start from the SPG metric:
(fxx: {(x, y)) - E06: y))
2 E(Oe: y(), - E00 - aggi: ((x,
(Last layer update only)
so E(Ox: ((x,))) - (e(Bus ((xx x))) - agt Vo. ((bx: (xx,))())))) (Taylor approximation)
= agg T Vo, ((ks ((x,))) = =a11gh11.

---

Algorithm 1 Meta-training of GradNCP
Algorithm 2 Meta-testing of GradNCP
Input:
Initial Ob, (s. (5.) a, K, L
Input: Test signal s, learned initialization 80 Kust
while not converge do
1: Extract context Craz from S.
2: Sample batch (51)...,88).
Where typically Kust
3: for all b = 1 to B do
2: for all t to Kist
4:
Extract context Cran from 8b- for all to K do
3:
Contaxt pruning
4: Chigh TopK(Crant Re 7))
6:
Online Contoxt pruning Clige TopK(Cral1: Re,
Compute gradient scaling
7:
gium 1V Cage (0n,,Ggg
[V Crall Vo, Crsx Crai)
8:
8c+1 f 0b - VV Cusx (fu: Chigh)
# Adaptation with rull context
9:
end for
5:
tant. P + 0c or 9r
10:
Ganerate target in steps
6: end for
11:
ORET 0K- aV Bzc Luse (0x; Crai)
Output: Lxm Cru11),0
12:
cs. CRota: = Crse (0sz Cruza) + Au(Ox (x, Bloot
13:
14:
end for
15:
do 0o B Don Van Chotal
16:
end while
Output: Og
where the equivalence in the last layer results from all gradients w.r.t 6f in Vo, {(0; {xx,)) being multiplied with zero by their corresponding entry in gk. As the gradient for the final layer takes the standard form for linear model (see non-linear version and the full derivation in Appendix B), we can define our sample importance score R.GradNCP under LMSE at adaptation step k as:
R GGGINCP (x,y) = |((5 Jo,
Jo, (x)) A (x), (x), Il'll
(2)
where ||-I| the Frobenius norm and O0 (-) the penultimate feature vector, such that for (x) We (x) + by. This score only requires a single forward pass (and an additional small matrix multiplication) without the need for full backpropagation, making it straightforward and scalable to use for high-resolution signals. This particularly suitable for an extensions of MAML which we use in practice [35]. that considers the step size at each iteration K of the inner learning problem part of the metaaoptimization parameters, adding increased flexibility in terms of both choosing the most suitable data and update size for each iteration. We further emphasize that while motivated in the NF context, this approximation can be generally applicable in curriculum learning settings [44, 58).
=
2.3 Reducing Bias and Myopia through Bootstrap Correction
Despite the careful selection of Ch Caghh large reduction the size of the context set may inevitably introduce information loss. To mitigate this issue, we suggest counteracting any bias by reducing distance to parameter state obtained using optimization carried with the full context set. Specifically, after adapting the meta-learner for K steps with the pruned context set, we generate a bootstrapped target k through optimization for L additional steps starting from 0k- Crucially, we consider meta- optamiztiin to be oblivious to the target generating process, i.e., do not backpropagate through its updates, saving memory otherwise required for second-order optimization. In automated differen- tiation frameworks, this can be implemented with stop-gran -rogdint operation on Obeet, leading to each of the L additional steps being significantly cheaper as well as improved target performance.
Given both sets of parameters, the desired correction effect can be achieved by regularizing 0g to be close to the bootstrap target 0beet, according to a distance function p(0k, 0beet with Eg distance being a suitable choice. This correction has two effects: Firstly, it forces increased plasticity in the initial steps as the only means reduce the regularization loss target obtained using both additional information (i.e., access Cfull) and further optimization. This is possible as information about future learning dynamics infused in the initial steps of learning through bootstrapping. In effect, the meta-learner teaches to improve itself. Secondly, as noted in the related work [17], bootstrapping reduces the phenomenon of myopia in meta-learning- reducing the effectiveness of longer optimization loops at meta-test time (when no second-order optimization is needed and thus longer adaption procedures are possible). Flennerhag et al. [17] show that bootstrapping for this purpose leads to guaranteed performance improvements.
well-known issue significantly
5

---

Overall meta-learning objective, In practice, we find that it is useful to calculate the final loss as combination of the target correction term and the performance of 0x on the full context set. For given nyper-parameter À > 0, the meta-objective of GradNCP becomes:
Leatal (fo: Cra1n) LusE (Oxx Cran) + λμ((Ox k. +
choice inspired by the common meta- meta-learning practice of optimizing the meta-loss w.r.t the inner validation set performance, infusing further information into the initial K adaptation steps. The full meta-training and -testing procedures are shown Algorithms
2.
Connection with target model regularization frameworks. Addressing the issue of information loss through target model can be viewed as application of prior methods that engage the target model in instructing the meta-learner. While prior works aim to adapt better target by further optimization [17. 31], or adapting from a better initialization [59], we believe such methods can be further improved by utilizing the full context set. Importantly, to the best of our knowledge, this work the first attempt of introducing bootstrapping as means to correct the effects of data pruning.
2.4 Gradient Re-scaling enables Effective Meta-test Optimization
Relieved from the memory requirements of second-order optimization and the burden of meta- learning myopia, the use of the full context set at meta-test time appears a natural choice. The naive combination of these ideas may not lead to desired results however, as the norm of the inner-loop gradient significantly deviates between meta-training and testing by construction, i.e., | (0; Caiga)l || > y Cruzn)ll: in accordance with the importance score R RGradNCP Ac- cordingly, use simple correction that ensures test-time gradient statistics on the same order as statistics optimized for during meta- vreaa-training. At step meta-test step we correct as:
|| V Lusz (O: Crags) Vo, LusE(O.; Cruzt), Vo, Lyse (g:: Ctara)]
gest
(4)
and then update Og with gf instead of Vo, LuSE (0;; Ctuli). We observe this to be a simple and sufficient means to drastically improving test-time performance when using Ctall.
Experiments
We now provide an empirical evaluation of GradNCP, systematically verifying claims made through- out the manuscript and thus supporting the suitability of its constituent components. At this point, we remind the reader of the key questions raised thus far:
1. Efficiency and context scoring. Can GradNCP's online context selection mechanism reduce memory and compute requirements vis-a-vis standard meta-learning? If so, what is the quality of the introduced scoring function ReGradNCPP and how can its approximations be justified? Finally, do selected context points correspond to humanly interpretable features?
2. Bootstrap correction and myopia. What is the effect of bootstrap correction on any irreducible error introduced through context pruning? Do we observe improvements on the issue of myopia? How do we know that bootstrapping provides a well-conditioned target model?
3. Overall evaluation. How does GradNCP compare to satte-of-the-ar meta-learning techniques for NFs? What are the necessary conditions for the highest-quality results at test time? Do introduced efficiency techniques enable learning on more complex signals?
Before answering each question, we first lay out the experimental protocol used throughout:
Baselines. We compare GradNCP with existing meta-learning schemes for NFs, including the MAML [15] (known as Learnit [61] in the NF literature), transformer based meta-learning (TransINR; [6]) and the recent Instant Pattern Composer (IPC; (29]). Where appropriate, we also show the effect of tablua rasa optimization from random initialization (Random Init.) to highlight the need for
accelerated learning. Presented TransINR and MAML/Leard results have been obtained with
reference to the official implementation (and recommended hyperparameters for TransINR) while we
adjust MAMI/I Larnitt's 's number of inner gradient steps for a fair comparison with GradNCP, IPC
values correspond to results directly published by the authors.

---

44
38
38
Test PENR (08)
42
Test PaPPP (dB)
40
Text PEWA
34
96
32-
Random
Learnil MAML BradNCP (Curs]
Learnt / MAML GradNCF (Duns)
TPG
36
16
20
30
60
100
150
200
250
80
25
60
76
Grad Norm [Durs]
100
126
150
Meeoory usege (OB)
Training wall-clokk (min)
Training heration (x10*)
Memory consumption
(b) Runtime efficiency
(c) Context scoring quality
Figure 3: Comparing the learning efficiency of GradNCP to standard meta-learning. (a) GradNCP drastically decreases memory usage when varying K at meta-training time. (b) GradNCP achieves the same performance in 55% of wall-clock time (1.8x speedup). (c) GradNCP scoring maintains equivalent performance while reducing runtime complexity: ((M²) to (M) relative to self-
prediction gain (SPG) and target-prediction gain (TPG) [21] approximated with last layer update.
NF architectures. We use SIREN [54] as the base architecture for all experiments, and additionally consider NeRV [5] in the video domain. We provide more comparisons other architectures, such as Fourier feature networks (FFN) [60] in Appendix D.3.
Evaluation setup. To ensure a fair comparison, we control for the number of adaptation steps and/or memory requirements when comparing Learnit/MAML with GradNCP. Quantitative results are reported using Peak Signal-to-Noise Ratio (PSNR; higher is better) as well as perceptual similarity for image and video datasets by using LPIPS (lower better) [69] and SSIM (higher is better) [65]. For video datasets, we evaluate these metries in Trame-wisc manner and average over the whole video following NeRV[S]. All ablation studies presented show results on CelebA (178 x 178) using K = 16 inner gradient steps, L = 5 bootstrap target construction steps and 0.25 for GradNCP. The MAML Learnit configuration with equivalent memory use corresponds to K = 4 while both methods use KEm 16 test time adaptation steps unless otherwise stated.
Datasets. Highlighting the versatility of NFs, we consider four different modalities, namely images, videos, audio, and manifolds. For images, we follow Learnit and use CelebA [36]. Imagenette [23], and pictures including Text [61] while additionally considering the high-resolution fine-grained datasets CelebA-HQ [26] and AFHQ [7]. To test under high visual diversity we also use all 1,000 classes of ImageNet [8]. Pushing memory requirements past current limits, we consider the video dataset UCF-101 [57] using resolutions (128x 128 x 16, 256x256 x32) demonstrate the scalability of GradNCP. Finally, we also test on audio using the speech dataset Librispeech [42] and on manifolds
by considering climate data the globe using ERA5 [22].
3.1 Online Context Pruning leads to Highly Efficient Meta-Learning
First turning to the primary motivation for our work, we show comparison in terms of memory efficiency and runtime in Figures 3a and 3b, noting that GradNCP leads to both a significant improvement in the memory/performance trade-off. This enables either significant runtime speedups on the same machine or meta- meta-learning on significantly higher dimensional signals (as we will show later). Given such encouraging results, how do we know whether approximations made for R2 [GGaNNC in Section 2.2 are justifiable? We attempt to answer this in Figure 3c by comparing to existing aeveuluumleaarinn metries, noting (i) that approximating the effect of full network update using the last layer only continues to consistently outperform random sub-sampling (ii) measuring the learning effect of context point by the reduction in error on itself does not decrease performance (showing equivalent performance of SPG and TPG) and (iii) that our proposed first-order Taylor series approximation does successfully alleviates the need for update without performance degradation.
What explains the robustness of both approximations? In Figure 4 we remarkably notice that when used in meta-learning setting, the correlations between the last layer and full network gradient consistently improve with the number of inner adaptation steps (unlike for a random initialization). Remarkably, this suggests that meta-learning dynamics push parameters to state where the approxi- mation assumptions above are satisfied while simultaneously improving performance. Moreover, as mentioned in Figure 1, visualization of the context set Cligh generally corresponds to interpretable features, with many more examples of this phenomenon shown in Appendix D.
7

---

39
42
40
Test PSPP (dB)
Tran PaNR (cB))
Test PEWAR
30
36
Learnt MAML
Nomm-basa based
Nomn-based Norm based Book (Gurs)
Metaeleamen (K
Norm based Boot (Ours)
Boonstrap targat IK stops)
30
25
50
75
100
160
30
12
16
20
24
26
32
25
50
100
150
Training
(x10*)
adaptaton
Training heration (x11²)
(a) GradNCP component analysis
Myopia
Bootstrap target performance
Figure 5: Analysing the effects of bootstrapping. (a) Bootstrapping reduces error introduced by context pruning and stabilizes meta-training (b) Bootstrap correction allows learning past the meta-
training horizon (K 16), reducing myopia. (c) The bootstrap model achieves consistently higher performance, providing well-conditioned target.
PSNR: 28.58
PSNR: 23.43
PSNR: 29.61
Learnit (MAML)
TransINR
GradNCP (Ours)
Ground truth
Figure 6: Qualitative comparison between GradNCP and baselines on AFHQ (512x512).
Understanding the Effects of Bootstrap Correction
Given the high quality of online-context point selection scheme, what additional contributions are made by the boot- strap correction (Section 2.3)?
1.00 monss
Comeeation coefiiieent
0.95
answer this question in Figure 5a, noting that bootstrapping indeed leads to consistent performance improvements while also stabilizing training. Note that this effect distinct from any reduction in myopia (as keep K Kust fixed). Myopia nevertheless exists, as we show in Figure 5b, noting that both Learnit/ MAML and GradNCP w/o bootstrap correction result noteworthy that the rate of improvement for each additional step drastically decreases for Learnit/ / MAML, showing the reduced plasticity effect discussed throughout the manuscript.
0.90
0.204
0.574
11
13
Adaptation step (K]
(a) Random init.
1.00
in reduced test-time adaptation results.
Conelation eceffdoont
odee 0deo 0u50a 0.77
0,507
0.009
0.80
One possible concern for bootstrapping in our setting is that the GradNCP loss (3) could be trivially minimized by finding a local minima that cannot be escaped in the L additional steps afforded the target model, thus leading to low regulariza- tion loss but also removing any positive effect of bootstrap- ping. Fortunately, this is not the case, clearly highlighting that the step meta-learner consistently trails improved target model, providing well-conditioned regularization target.
0.85
Adeptation
(b) Meta-learned init.
Figure 4: Meta-learning dynamics improve approximation quality over time. Shown is the correlation be- tween the last layer and the entire network gradient.
show in Figure 5e,
3.3 GradNCP achieves State-of-the-Art Results in Meta-Learning Neural Fields
Having motivated and verified GradNCP's components, we now turn to our main results, presenting results for images Table 1, videos Table as well as audio and manifolds in Table 3. Overall, GradNCP significantly and consistently outperforms
tiite-of-the-aae meta-learning methods by large margin, leading to high-quality NF's as shown in Figure exhibiting clear superiority over the baselines in capturing high-frequency components a direct benefit of online context pruning, due to additional learning capacity dedicated to such high-frequency details later adaptation steps. We provide more qualitative results in Appendix D.

---

Table 1: Reconstruction performance of SIREN on image datasets of various resolutions. We report
results as PSNR [dB] (t) SSIM (1) / LPIPS (1) using the same number of adaptation steps for all schemes. OOM denotes the out-of-memory on a single NVIDIA A100 40GB GPU. Here IPC indicates Instance Pattern Composers with results directly taken from [29].
CelebA (178 178)
Imagenette (178 x 178)
Text (178 x 178)
Random Init.
19.94 0.532 0.700
18.57 / 0.443 0.810
15.37 / 0.57// 0.755
TransINR [6]
32.37/ 0.913 0.006
28.58 0.850/ 0.165
22.70 0.898 0.085
IPC [29]
35.93/ /
38.46/ /
Learnit/ / MAML [61]
38.28 0,964 / 0.0010
35.66 0.950 0.014
30.31 0.956/ 0.018
GrP (Ours)
40.60 0.976 / 0.005
38.72/ 0.972 / 0.005
32.33 / 0.976 0.007
ImageNet (256 x 256)
AFHQ (512 x 512)
CelebA-HQ (1024 x 1024)
Random Init.
18.72 0.434 /0.839
18.57 0.488 0.856
12.21 /0.574/0.820
TransINR [6]
28,01 0.818 /0.19
23.43 0.592 0.573
O0M
Learii/ / MAML [61]
31.44 / 0.887 /0.100
28.58 0.751 / 0.354
27.66 0.781 0.513
GradNCP (Ours)
32.52/ / 0.898 0.068
29.61 0.786 / 0.286
28.90 / 0.789/ / 0.438
Table 2: Reconstruction performance of meta-learned SIREN Table 3: PSNR of meta-learned and NeRV. We consider two different resolutions of the UCF- SIREN on (a) Librispeech and 101 dataset. OOM denotes unavailable results due to running out-of-memory on a single NVIDIA A100 40GB GPU.
(b) ERA5 (181 x60)..
(a) Librispeech
Resolution
Network
Method
PSNR (1)
SSIM (f)
LPIPS
PSNR (*)
TransINR [6]
15.14
0,360
0.636
Method
1 sec
sec
SIREN
Learnt/ / MAML [61]
25.46
0.720
0.363
TransINR [6]
39.22
33.17
128 16
GradNCP (Ours)
26.92
0.781
0.223
IPC [29]
40.11
35.38
Learnt (MAML) [61]
28.86
0.871
0.140
Leamit MAML [61]
39.55
31.39
NeRV
GradNCP (Ours)
35.28
0.959
0.015
GradNCP (Ours)
43.25
36.24
TransINR [6]
0OM
(b) ERA5
SIREN
Leamit/ / MAMI [61]
00M
256x256x32
GradNCP (Ours)
22.92
0.640
0,521
Method
PSNR (t)
Leamit/ / MAML [611
23.75
0.659
0.422
Learnit/ MAML [61]
64.91
NeRV
GradNCP (Ours)
28.65
0.842
0.201
GradNCP (Ours)
75.11
We remark that one of the nice properties of GradNCP its modality- and model-agnosticcsm. On the other hand, we find TransINR struggles to generalize for some modalities and architectures: namely requires (a) tokenization, which is not straightforward to apply for spherical coordinate datasets (e.g., ERA5), and may require the framework to deal with sequences of tokens that are too long (e.g.. videos), and (b) notable modifications to the architecture when it consists of non-MLP layers (e.g.. NeRV) as the framework is specific to MLPs.
High-resolution signals. One of the significant advantages of GradNCP is its exceptional memory efficiency, which allows us to meta-learn on large-scale high-resolution signals. As demonstrated in Table 1 and Table 2, GradNCP can even be trained on x56x256x33 resolution videos or 1024 x 1024 resolution images, a scale previously not possible for prior work (even under the constraint of NVIDIA A100 40GB GPU) due to their intensive memory usage (see Figure 3a).
GradNCP for generalization tasks. To also verify that GradNCP can be used for generalization task where the meta-learning context and target sets are disjoint. To this end, we apply GradNCP for the NeRF [40] experiment on ShapeNet Car [4] to verify the generalization performance of GradNCP. Here, following the setup in Tancik et al. [61], we train NeRFs with given single view of the 3D scene and evaluate the performance on other unseen views. As shown in Ta- ble 4, GradNCP significantly outperforms other baselines including Learitt / MAML (e.g., 22.80 - 24.06 dB), and even achieves better performance than TransINR which utilizes an additional (large) Trans- former. We believe using GradNCP for large-scale generalization of NFs (e.g., NeRF for Google street view) will be an interesting future direction to explore, where we believe the efficiency of GradNCP can contribute to this field.
Table 4: PSNR (dB) of un- seen view synthesis of NeRF on ShapeNet Cars. We adapt NeRF with a single view of a 3D scene then evaluate the quality of unseen views.
Method
PSNR
Learnit [61]
22.80
TransINR [6]
23.78
GradNCP (Ours)
24.06

---

38
Test PSNR (dB)
36
Pruned
0.8---
Pruned
34
Costient Nom
0.00
32
Gaddent
0.04
30
28
No Gradient Re-scaling Gradient Re-scaling
0.02
26
25
50
75
100
125
150
Training iteration (x10²)
(a) k
(h) k 15
Figure 7: Effect of meta-test time gradient re- scaling. We apply re-scaling when adapting with full context set on SIREN trained with GradNCP.
Figure 8: Gradient norm of the full context set Crull and the gradient norm-based pruned con- text set Chige at iteration
Table 5: Effect of context set selection ratio on SIREN meta-learned with CelebA (178 178). controls the performance and the memory trade-off off when using GradNCP.
Selection ratio (7)
0.05
0.10
0.15
0.20
0.25
0.5
0.75
1.0
PSNR (dB)
31.20
35.00
36.67
37.59
38.90
39.25
39.70
39.98
Memory (GB)
1.93
2.41
3.05
3.45
3.91
6.31
9.22
11.04
3.4 Additional Analysis
Gradient re-scaling analysis. Here, we investigate the importance gradient re-scaling. To show the effect of gradient re-scaling the meta-test time, track the test PSNR during training SIREN with GradNCP (that does not use the Bootstrapped correction) on CelebA (178x 178). Here, we report the PSNR when the meta-initialization is adapted with the full context on the meta-test stage. As shown in Figure 7, the gradient re-scaling significantly improves the performance.
To further investigate the necessity of gradient re-scaling, we additionally analyze the gradient norm during the (inner loop) adaptation phase of meta-training Here, we plot the gradient norms at the adaptation step l measured by the full context set Crui1, and the gradient norm-base based pruned context set Chigh respectively. As shown in Figure 8, our results indicate that the norm of the gradients exhibits significant variations, with some steps exhibiting 3x larger gradients when using the pruned context set. This suggests that the meta-learner employed a larger step size during training, which highlights the importance of test-time gradient re-scaling. We also find that re-scaling the gradient with the loss ratio of the full context set and the sampled context leads to similar performance improvements and may serve as faster alternative, twice, i.e., showing 40.53 in PSNR where the gradient re-scaling with gradient norms shows 40.80.
eliminates the need for gradient calculation
Selection ratio analysis. We also have investigated the tradeoff between performance and memory requirements of GradNCP by varying the value of the context set selection ratio 7 in our meta- meta-trainin training process CelebA (178 178). As shown in Table 5, the memory usage decreases proportionally as one decreases the selection ratio, while the performance is maintained fairly well even for relatively small values of y 0.15, i.e., 36.67 (dB). For the main experiment, mainly use 7 0.25, which shows high performance while significantly reducing the memory usage, e.g., 75% of memory.
4 Discussion and Conclusion
We propose GradNCP, an efficient and effective method for fast and scalable neural field learning through online-pruning of the context set at each meta-training iteration, significantly reducing memory usage. We demonstrate that GradNCP improves performance over various modalities and, more importantly, exhibits superior memory efficiency allowing it to be the first method capable of meta-learning on exceptionally high-resolution signals.
Future work and limitation. We believe extending GradNCP to extreme high-resolution signals (e.g., long 8K video), where even single forward pass is not possible, would be an interesting future direction to explore. We believe that a great variety of techniques can be developed in this direction, e.g., iterative tree-search of high gradient norm samples by starting from low-dimension grid and incrementally increasing the resolution of the sampled area.
10

---

Acknowledgements
We thank Kyuyoung Kim and Jongjin Park for providing helpful feedbacks and suggestions in preparing an earlier version of the manuscript. This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITP) grant funded by the Korea government (MSIT) (No.2019-0-0007))
Artificial Intelligence Graduate School Program (KAIST), and No.2022- 0-00713, 13, Meta-learning applicable to real-world problems) and National Research Foundation of Korea(NRF) grant funded by the Korea governmen(MMSTT) (No.RS-2022) 2023-000211100, Neural Network Optimization with Minimal Optimization Costs).
References
[1] M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. De Freitas Learning learn by gradient descent by gradient descent. Advances in Neural Information Processing Systems, 2016.
[2] M. Bauer, E. Dupont, A. Brock, D. Rosenbaum, J. Schwarz, and H. Kim. Spatial functa: Scaling functa imagenet classification and generation. arXiv preprint arXiv:2302.031300 2023.
[3] Bronskill, D. Massiceti, M. Patacchiola, K. Hofmann, S. Nowozin, and R. Turner. Memory efficient meta-learning with large images. In Advances in Neural Information Processing Systems, 2021.
[4] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang. Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1oXiv:512.000111 2015.
[5] H. Chen, B. He, H. Wang, Y. Ren, S. N. Lim, and A. Shrivastava. NeRV: Neural representations for videos. In Advances in Neural Information Processing Systems, 2021.
[6] Y. Chen and X. Wang. Transformers as meta-learners for implicit neural representations. In European Conference Computer Vision, 2022.
[7] Y. Choi, Y. Uh. J. Yoo, and J.-W. Ha. StarGAN v2: Diverse image synthesis for multiple domains. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[8] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database. In IEEE Conference Computer Vision and Pattern Recognition, 2009.
[9] Y. Du, M. K. Collins, J. Tenenbaum, and V. Sitzmann. Learning signal-agnosti manifolds of neural fields. In Advances in Neural Information Processing Systems, 2021.
[10] E. Dupont, A. Goliński, M. Alizadeh, Y. W. Teh, and A. Doucet. Coin: Compression with implicit neural representations. In ICLR Neural Compression: From Information Theory to Applications Workshop, 2021.
[11] E. Dupont, Kim, S. Eslami, D. Rezende, and D. Rosenbaum. From data to functa: Your data point is function and you should treat it like one. In International Conference Machine Learning, 2022.
[12] E. Dupont, H. Loya, M. Alizadeh, A. Goliński, Y. W. Teh, and A. Doucet. Coin++: Data agnostic neural compression. Transactions on Machine Learning Research, 2022.
[13] Z. A. S. Emam, H.-M. Chu, P.-Y. Chiang, W. Czaja, R. Leapman, M. Goldblum, and T. Goldstein. Active learning at the imagenet scale. arXiv preprint arXiv:2111.1288) 2021.
[14] V. Feldman and C. Zhang, What neural networks memorize and why: Discovering the long tail via influence estimation. In Advances in Neural Information Processing Systems, 2020.
[15] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, 2017.
[16] S. Flennerhag, P. Moreno, N. D. Lawrence, and A. Damianou. Transferring knowledge across learning processes. arXiv preprint arXiv:1812.0000 2018.

---

[17] S. Fleeneern,,, Y. Schroecker, T. Zahavy, H. van Hasselt, D. Silver, and S. Singh. Bootstrapped meta-learning. In International Conference on Learning Representations, 2022.
[18] A. Galashov, J. Schwarz, H. Kim, M. Garnelo, D. Saxton, P. Kohli, S. Eslami, and Y. W. Teh. Meta-learning surrogate models for sequential decision making. arXiv preprint 1903.19900,1907,, 2019.
[19] M. Garnelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. Rezende, and S. A. Eslami. Conditional neural processes. In International Conference on Machine Learning. 2018.
[20] M. Garnelo, J. Schwarz, D. Rosenbaum, F. Viola, D. J. Rezende, S. Eslami, and Y. W. Teh. Neural processes. arXiv preprint 1rXivv1807.01623 2018.
[21] A. Graves, M. G. Bellemare, J. Menick, R. Munos, and K. Kavukcuoglu. Automated curriculum learning for neural networks. In International Conference Machine Learning, 2017.
[22] H. Hersbach, B. Bell, P. Berrisford, G. Biavati, A. Horányi, J. Muñoz Sabater, Nicolas, C. Peubey, R. Radu, I. Rozum, et al. ERA5 monthly averaged data single levels from 1979 to present. Copernicus Climate Change Service (C3S) Climate Data Store (CDS), 2019.
[23] J. Howard. Imagenette, 2019. URL wwtphtttp//// http://gttttbbcccc httpt/// htwp:h/ww..mm/ttttta//meeeeeeeee
[24] N. Hu, E. Mitchell, C. Manning, and C. Finn. Meta-learning online adaptation of language models. In Conference on Empirical Methods in Natural Language Processing, 2023.
[25] L. Huang and T. Hoefler. Compressing multidimensional weather and climate data into neural networks. arXiv preprint 1.XXi:222001153:: 2022.
[26] T. Karras, T. Aila, Laine, and J. Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
[27] A. Katharopoulos and F. Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International Conference on Machine Learning, 2018
[28] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan. F. Viola, T. Green, Back, P. Natsev, M. Suleyman, and A. Zisserman. The kinetics human action video dataset, 2017.
[29] C. Kim, D. Lee, S. Kim, M. Cho, and W.-S. Han. Generalizable implicit neural representa- tions via instance pattern composers. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[30] S. Kim, S. Yu, J. Lee, and J. Shin. Scalable neural video representations with learnable positional features. In Advances Neural Information Processing Systems, 2022.
[31] T. Kim, J. Yoon, O. Dia, S. Kim, Y. Bengio, and S. Ahn. Bayesian agdest meta-learning. In Advances in Neural Information Processing Systems, 2018
[32] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[33] Z. Landgraf, A. S. Hornung, and R. S. Cabral. Pins: Progressive implicit networks for multi- scale neural representations. In International Conference on Machine Learning, 2022.
[34] J. Lee, J. Tack, N. Lee, and J. Shin. Meta-learning sparse implicit neural representations. In Advances in Neural Information Processing Systems, 2021.
[35] Z. Li, F. Zhou, F. Chen, and H. Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.000 09835, 2017.
[36] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In IEEE International Conference Computer Vision, 2015.
12

---

[37] A. Luo. Y. Du. M. J. Tarr, J. B. Tenenbaum, A. Torralba, and C. Gan. Learning neural acoustic fields. In Advances in Neural Information Processing Systems, 2022.
[38] J. N. Martel, D. B. Lindell, C. Z. Lin, E. R. Chan, M. Monteiro, and G. Wetzstein. Acorn: Adaptive coordinate networks for neural scene representation. ACM Trans. Graph. (SIGGRAPHI), 2021.
[39] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3D reconstruction in function space. In JEEE Conference on Computer Vision and Puttern Recognition, 2019.
[40] B. Mildenhall, P. P. Srinivasan, M. Tancik, T. Barron, R. Ramamorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference Computer Vision, 2020.
[41] A. Nichol, J. Achiam, and J. Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: /803. 02999, 2018.
[42] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: an ast corpus based on public domain audio books. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2015.
[43] J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[44] M. Paul, S. Ganguli, and G. K. Dziugaite. Deep learning on a data diet: Finding important examples early in training. In Advances in Neural Information Processing Systems, 2021.
[45] A. Rajeswaran, C. Finn, S. M. Kakade, and S. Levine. Meta-learning with implicit gradients. Advances in Neural Information Processing Systems, 2019.
[46] M. Ren, W. Zeng, B. Yang, and R. Urtasun. Learning to reweight examples for robust deep learning. In International Conference Machine Learning, 2018.
[47] D. Rolnick, A. Ahuja, Schwarz, T. Lillicrap, and G. Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, 2019.
[48] R. Schwarz and Y. W. Teh. Meta-learning sparse compression networks. Transactions on Machine Learning Research, 2022.
[49] J. R. Schwarz, J. Tack, Y. W. Teh, J. Lee, and Shin. -odasity-agnostic variational compression of implicit neural representations. In International Conference on Machine Learning, 2023.
[50] O. Sener and S. Savarese. Active learning for convolutional neural networks: A core-set approach. International Conference on Learning Representations, 2018
[51] J. Shin, H. B. Lee, B. Gong, and S. J. Hwang. Large-scale meta-learning with continual trajectory shifting. In International Conference on Machine Learning, 2021.
[52] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous 3D-structure-aware neural scene representations. In Advances in Neural Information Processing Systems, 2019.
[53] V. Sitzmann, E. R. Chan, R. Tuckerr N. Snavely, and G. Wetzstein. MetaSDF: Meta-learning signed distance functions. In Advances in Neural Information Processing Systems, 2020.
[54] V. Sitzmann, J. N. P. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems, 2020.
[55] Skorokhodov, S. Ignatyev, and M. Elhoseiny. Adversarial generation of continuous images. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
13

---

[56] J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
[57] K. Soomro, A. R. Zamir, and M. Shah. UCFI01: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 2012.
[58] B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Advances in Neural Information Processing Systems, 2022.
[59] Tack, Park, H. Lee, J. Lee, and Shin. Meta-learning with self-improving momentum target. In Advances in Neural Information Processing Systems, 2022.
[60] M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-KKan N. Raghavan. U. Singhal, R. Ra- mamoorthi, Barron, and R. Ng. Fourier features networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems, 2020.
[61] M. Tancik, B. Mildenhall, T. Wang, D. Schmidt, P. P.P. Srinivasan, J. T. Barron, and R. Ng. Learned initializations for optimizing coordinate-be based neural representations. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[62] M. K. Titsias, J. Schwarz, A. G. d. G. Matthews, R. Pascanu, and Y. W. Teh. Functional regularisation for continual learning with gaussian processes. In International Conference on Learning Representations, 2020.
[63] M. Toneva, A. Sordoni, R. T. Combes, A. Trischler, Y. Bengio, and G. J. Gordon. empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019.
[64] T. Wang. J.-Y. Zhu, A. Torralba, and A. A. Efros. Dataset distillation. arXiv preprint arXiv:1811. 2018.
[65] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing (TIP), 2004.
[66] Y. Wu, M. Ren, R. Liao, and R. Grosse. Understanding short-horizon bias in stochastic meta-optimization. In International Conference Learning Representations, 2018.
[67] S. Xie, C. Sun, Huang. Z. Tu, and K. Murphy. Rethinking spatiotemporal feature learning: Speed-accuradl trade-offi oode-offs in video classification. In European Conference on Computer Vision, 2018.
[68] Yu, J. Tack, Mo, H. Kim, Kim, J.-W. Ha, and J. Shin. Generating videos with dynamics- aware implicit generative adversarial networks. In International Conference Learning Representations, 2022.
[69] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features perceptual metrie. In IEEE Conference on Computer Vision and Pattern Recognition, 2018
[70] L. Zintgraf, K. Shiarli, V. Kurin, K. Hofmann, and S. Whiteson. Fast context adaptation via meta-learning. International Conference on Machine Learning, 2019.
14

---

Appendix
A
Experimental Details
In this section, we describe the experimental details of Section 3, including GradNCP and the baselines. We also provide the implementation of GradNCP at https:://se //ghth//b.//..///////ggg com/ jihoontack..... CCradNC.
A.1
Dataset Details
CelebA. CelebA is fine-grained dataset that consists of the face image of celebrities [36]. The dataset comprises 202,599 images, where we use 162K for training. 20K for validation, and 20K for testing. We resize the image into 178 and then center cropped of 178, which ends with resolution of 178x x 178. We pre-process pixel coordinates into 1, and signal values ranging from to 1.
Imagenette. Imagenette is 10 10-class subset of the ImageNet [8] dataset, which is comprised of 9,000 training and 4,000 test images [23]. We resize the image into 178 and then apply center crop of 178, which ends with resolution of 178 x 178. We pre-process pixel coordinates into [-1, 1]2 and signal values ranging from 0 to 1.
Text, Text dataset consists of text image with a resolution of 178 x 178 [61]. We pre-process the pixel coordinates into [-1, and signal values ranging from 0 to 1.
ImageNet. ImageNet dataset is a large-scale natural image dataset consisting of 1,000 classes [8]. We use 1,281K images for training and evaluated the methods on 50K test images (known as the validation set of ImageNet). For image rreprroesssing we resize the image into 256, then center cropped the image to get 2 resolution image. For coordinate pre-processing, we use pixel coordinates [-1, 1) and signal values ranging from 0 to
CelebA-HQ. CelebA-HQ is high-resolution fine-grained dataset, which includes images of celebri- ties [26]. We divided the dataset into 27,000 training and 3,000 test samples and pre-processed the pixel coordinates into [-1, 1|2 and signal values ranging from 0 to 1. We consider two resolutions,
ie., 512x512 and 1024x1024.
AFHQ. AFHQ is a high-resolution fine-grained dataset, which includes animal faces consisting of 15,000 images 512x512 resolution [7]. We divided the dataset into 14,336 training and 1,467 testing points and pre-prooe pre-processed the pixel coordinates into [], and signal values from 0 to 1.
UCF-101. UCF-101 is video dataset comprising 13,320 videos (9,357 training and 3,963 test videos) with a resolution of 320x 240, where the action classification consists of 101 classes [57]. Each video clip is center-cropped to 240 240 and then resized into 128 128 and 256 256 with a video clip length of 16 and 32, respectively. We pre-process the pixel coordinates into [-1, 1]s and signal values ranging from 0 to 1.
Kinetics-400, For the cross-domain adaptation purpose in Appendix D.7, we use the mini- mini-KKnettics 200 dataset [67], which consists of 200 categories with the most training samples from the Kinetics- 400 dataset [28]. We center-crop the videos to the same height and width and then resize them into x28x 128 resolution, and use the frame of 16 clips. We pre-process the pixel coordinates into [1, and signal values ranging from 0 to
ERAS. ERAS is a dataset comprised of temperature observations global grid of equally spaced latitudes and longitudes [22]. By following [11], we use the grid resolution of 181 x 360 by resizing the grid. We interpret each time step as an independent signal, and the dataset comprises 9,676 training points and 2,420 test points. For the input pre-prous pre-processing, given latitudes p and longitudes yp transformed into 3D Cartesian coordinates = (cos p cos ip, cos p sin ip, sin p) where latitudes equally spaced between - and _ and longitudes yp are equally spaced between 0 and 2x(n-1) where n the number of distinct values of longitude (360).
n
LibriSpeech. LibriSpeech is an English speech recording collection at 16kHz sampling rate [42]. By following Dupont et al. [12], we use the train-clcan- 100 split for training and test-clean split for testing, i.c., 28,539 training and 2,620 test samples. For the main experiments, we use the first I second and 3 seconds of each example (1 second contains 16,000 coordinates). For the pre- pre-processing, scale the coordinates into [-50, 50).
15

---

A.2 Training and Evaluation Details
Network architecture details. For the main experiment, we mainly use SIREN, a multi-layer perception (MLP) with sinusoidal activation functions [54]. i.e., x + sin(wo + b)) where W, are weight and biases of the MLP layer and we the fixed hyperparameter. For image, audio, and manifold datasets, we use SIREN with 5 layers with 256 hidden dimensions, and for video, we use 7 layers with the same hidden dimension. We used we 50 for the manifold dataset and use w/ = 30 for the rest. We additionally consider NeRV [5] for the video dataset. For the UCF-101 dataset of x28x 128x x 16, we use 4 NeRV blocks, and for 256x256x33
we use NeRV blocks. Finally, we consider the Fourier feature network (FFN) in Appendix D.3, where we use the same network size as SIREN and the use Fourier feature scale o = 10.
Training details. For all dataset, we use Adam optimizer [32] for the outer loop. We use the outer step of 150,000, except for learning the ImageNet dataset where we use 500,000 steps. For SIREN, we use the outer learning rate of 3 = 3.0 x 10 6 for Librispeech and use 8 = 1.0 x 10 for the rest. For NeRV, we use the outer learning rate of 3 = 1.0 10 As for the inner loop learning rate, we use a 1.0 x 10-2 and a 1.0 x 10-1 for SIREN and NeRV, respectively. For inner step number K, GradNCP is trained on a longer horizon than Learnit by multiplying 1//7 (which uses the same memory usage). For Learnit, we mainly use K = 4 for the main table, where we use K = for CelebA-HQ (1024x 1024), K = 5 on UCF-101 (256 x 226x2266xx x x2) on NeRV, and = 20 on UCF-101 (128x 128x 16) on NeRV. We use the same batch size for Learnit and GradNCP to fairly use the memory where the size was selected differently across the dataset (e.g., under the given GPU memory budget).
Hyperparameter details for GradNCP. We find that the hyperparameter introduced by GradNCP is not sensitive across datasets and architectures. For the context set selection ratio i.e., the ratio of retaining coordinates, we use 0.25 for most of the dataset except for Librispeech and ERAS, where we used 0.5 (as we do not need to prune the context much for these low-resolution signals), and 0.5, 0.2 when training NeRV on UCF-101 on 128 and 256 resolution, respectively. We found that for most of the datasets, the performance does not significantly decrease until 7 = 0.2 while significantly reducing the memory, i.e., about 5 times. For bootstrap target correction hyperparameters, we used L = 5, and A~ = 100, where we believe tuning these hyperparameters will indeed improve the performance much more (we did not tune extensively).
Evaluation details. For the evaluation, to fairly compare with the baseline, we use the same test-time adaptation step for Learnit and GradNCP (c.g., for CelebA experiments we use Kies = 16) which is the same step number that is used by GradNCP on meta-training. We additionally compare the results of test-time adaptation of TransINR in Section D.8.
NeRF details. By following prior works [61, 6]. we use the simplified NeRF model [40] which uses single network rather two networks (coarse and fine), and do not use the view direction for the network. We use a MLP with 6 layers with 256 hidden dimension with ReLU activation. We follow the same training detail of Tancik al. [61], except for inner loop steps where we 16 steps (rather than 100 steps) and first random sample 512 rays then select the important 128 rays by using 7 0.25 (same ray size as the prior works).
Resource details. For the main development, we mainly use Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz and single RTX 3090 24GB GPU, except for high-resolution signals including -e of 1024 x 1024 and UCF-101 of 256 x 256 x 32, where we use AMD EPYC 7542 32 Core Processor and single NVIDIA A100 SXM4 40GB.
A.3 Baseline Details
In this section, we explain the meta-learning baselines we used for evaluating GradNCP. We note that we additionally compare with memory-efficient meta-learning schemes in Appendix D.5.
Learnit [61] mainly uses the second-order gradient version of MAML [15] for learning NFs.
NNR [6] utilizes Transformer as meta-learner to predict the NF parameter with given
context set, and additionally proposed a parameter-efficient NF architecture specialized for MLP.
IPC [29] denotes instant pattern composer which is modulation technique for meta-learning NFs. This modulation uses low-rank parametrization of NF weights where the authors used TransINR as a meta-lcarning approach to train these modulations.
16

---

B Derivation of Our Data Importance Score
In this section, we prove our sample importance score (shown in the following equation) well approximates the updated loss value when adapting the network using the given sample.
Rk RGradNCP y) = [ fer (xx) (do (x), 1]
(5)
We first review the derivation in the main text. Let Of be the set of parameters excluding the final layer and 9h = [0, Vw, l, Vo, f] a padded gradient involving zeros for all but the last layer's weights W& and biases by. we start from the SPG metric:
{(xx: ((x, y))) ((%: y))
E(On: (xxx))) - E(0x= -agi: ((x,)))
(Last layer update only)
= E(O:: ((x, - (e(0u: ((x)x:((x))))) y)), agh Vo. ((b: ((xx)()x)))))
(Taylor approximation)
= agg Vo, (02; ((x, all9x111
where a 0 is the step size of the optimization. We remove a, the sample importance ranking does not change as a is a positive value. Since NF training considers MSE loss for é, the gradient of the last layer weight and bias are as:
Vw. (0x: ((x, y)3) -waal (ee--WWir (Wa (x) + tbb))) 1
= =-2(y - (Wadigger (x) + bbs) @opn (x)
(7)
-2 Ser, (x)
(8)
Vo, E(2i x() = Vou (x- (Wa (x) + bx)) 1|2
(9)
= =-2 (Weoggee (x) + bs))
(10)
-2 foz (x))
(11)
Given the gradient of the last layer in Eq. (8) and (11), the final score is derived as follows:
R
RGGNNNCP (x, y) = |1 112 = [[ Vw (0x: v)}) Vn, ((xi ((x,))))]
(12)
= - (y (x)( (olggoe (x), 1] 7/
(13)
Since the order (or the ranking) of the gradient norm value does not change by multiplying the positive value, we simply removed 2 from the final score. For NF with non-lined final layer (e.g., NeRV), one
can easily derive the following GradNCP criteria: ||yy- for (x)) of (Wed ('e))o'(WWoppee (x) bb.) @ (xx) 1
where the output of is fae (x) = o (W (W (x) (x)+ bb), o()) the element-wise non-linear operator (c.g., Sigmoid) and o' is the derivative of a.
C Related Work
Neural fields (NFs), NFs, also referred to as implicit neural representations (INRs), have emerged as a new paradigm for representing complex, continuous signals [43, 60]. This approach encodes the signal with a neural network, typically multi-layer perceptron (MLP) conjunction with sinusoidal activations [54] or positional encoding [40]. They have become popular as their number of parameters do not strictly scale with the resolution of the signal [39, 52]. easing modeling of multi-modal signals [9, 37] and showing potential for new approaches to prominent applications and downstream tasks, including data compression [10, 49], classification [11, 2], and generative modeling [55, 68). However, fitting NFs is quite costly, especially for high-resolution signals [30].
To tackle this issue, we develop an efficient meta-learning framework for large-scale NF training which significantly reduce the memory usages of meta-learning.
17

---

Efficient meta-learning, There have been several works on developing (memory) efficient algorithms the field of learning...ng...... Typically, such algorithms have been explored in amortization- (or mncoerrhassed) schemes, e.g.. Prototypical Networks [56, 3] and Neural Processes [20, 19, 18]. Unlike optimization-banl -based meta-leamen meta-learning. however, these methods are somewhat limited in applicability to diverse modalities and NF architectures (requiring modality and model-specific design).
In the optimization-enn based regime, several memory efficient schemes have been introduced, including first-order MAML [15], Reptile [41], Implicit MAML [45] and continual trajectory shifting [51]. that do use the second-order gradient adaptation. However, recent works have shown that first-order optimization- based schemes can underperform for NFs [11, 12]. We also observed similar result as summarized Table of Appendix D.5. Recently, network sparsity has been combined with meta-lcarning NFs [34, 48], which may help reduce computation at inference time but still requires some memory usage when meta-learning and more importantly can reduce the network expressive power. In this paper, we focus on developing a memory-efficies meta-learning framework that is built upon second-order gradient-bane basedd schemes for effective NF learning.
Sparse data selection. GradNCP is related to areas of machine learning which focus on identifying subsets of dataset. For instance, data pruning focuses on designing data importance metrics to reduce the dataset without comprising the performance [63, 14, 27, 58], memory-based techniques of continual learning use subsets of past tasks prevent catastrophic forgetting [62, 47]. dataset distillation proposes the use of optimized few synthetic data for the same purpose [64]. and active learning study way identify and label data points facilitate efficient learning progress during subsequent online updates [50, 13]. In this paper, we develop an efficient online context pruning scheme for meta-learning NFs by drawing inspiration from recent work curriculum learning to estimate the sample importance [21].
D
More Experimental Results
D.1
Gradient Norm Correlation
1.00
0.988
1.00
0.989
coefficient
0.977
Correlation coefficlent
0.96
0.961
0.95
0.947
0.909
0.934
0,9G3
0.95
0.904
0.874 0.876
0.88
0.85
0.85
11
13
15
15
Adaptation stop (k)
Adaptation stop (k)
Random init Adapt with the full context set
(h) Random init Adapt with the pruned set
1.00
1.00
Correlation coeffiiient
0.962
0.963
0.563
0.963
0.963
0.963
0.962
Correlation coefficient
0.957
0.152
0.957
0.962 0.965 0.968
0.971
0.95
0.95
0.908
0.90
0.901
0.90
0.85
0.85
9
11
13
15
11 13
15
Adaptation step
Adaptation step (K)
(c) Meta-learned init Adapt with full context set (d) Meta-learned init Adapt with the pruned set
Figure 9: Correlation between the gradient norm of the entire network and the last layer network. We measure the Pearson correlation coefficient between the entire gradient norm and the last layer gradient norm individual signals of the CelebA test dataset (i.e., correlation computed on 178x 178 elements), then report the average correlation across signals. Here the pruned set indicates the context pruning with GradNCP.
18

---

In this section, we compare the correlation between the gradient norms of the entire network and the last layer to verify that our last layer gradient norm reasonable approximation. We first note that the entire network gradient norm is an approximation for the SPG score (i.e., by using the first-order Taylor series on SPG) without the last layer gradient approximation, hence, serves as reasonable measure for the sample importance. It is worth noting that, while the entire gradient norm a reasonable score, this still computationally too heavy to use online, as requires computing the gradient w.r.t to the entire network parameter with (MM) time complexity where our last layer gradient norm only needs a forward pass of (M) (remark that M the context set size).
To this end, we measure the Pearson correlation coefficient between the entire gradient norm and the last layer gradient norm on individual signals of the CelebA test dataset (i.e., correlation computed on x 178 elements), then report the average correlation actoss signals. As shown in Figure 9, first the entire gradient norm and the last layer gradient norm are both highly correlated in all cases indicating that our approximation using the last layer only optimization is quite reasonable. Furthermore, remarkably, when computed on the meta-learned initialization, the correlations between the last layer and the entire network gradient norm consistently improve during the inner adaptation steps (unlike for the random initialization). This indicates that the meta-learning dynamics push parameters to state where the approximation assumptions above are satisfied while simultaneously improving performance.
D.2 Ablation Study on Boostrapped Target Adaptation Steps
Table 6: PSNR (dB) of SIREN meta-leared on CelebA dataset (178x 178) when increasing the num- ber of gradient steps (L) for generating bootstrapped target. Note that 'O' indicates no bootstrapped correction (i.e., only the gradient norm-based pruning over MAML).
L
2
5
10
PSNR (dB) 37.49
37.78
38.20
38.90
38.99
We also perform an ablation study regarding the step for generating bootstrapped target (boot, As shown in Table 6, we found that the performance rapidly improves until == 5 and observed moderate improvement by further increasing I.. During the development, we found that is already quite effective across all modalities and datasets, and we believe tuning the hyperparameter will improve the performance.
D.3 Comparison with Loss-based Selection
Table 7: Comparison of data selection criteria. We report the PSNR (dB) of meta-learned neural fields across various modalities when using loss-based (Loss) and gradient norm-based (Grad) top-K data selection. Bold indicates the best result of each group.
(a) Bounded activation
SIREN
CelebA
Imagenette
Text
AFHQ
CelebA-HQ
UCF-101 (128)
UCF-101 (256)
Libri (1sec)
Libri (3seo)
ERA5
Loss
40.54
37.71
33.11
29.37
28.89
26.59
22.76
43.40
36.45
74.10
Grad
40.60
38.72
32.33
29.61
28.90
26.92
22.92
43.25
36.24
75.11
(b) Non-bounded activation
NeRV
FFN Libri (1sec)
101 (128)
UCF-101 101 (256)
CclebA
Text
Libri (3scc)
Loss
33.99
28.58
28.81
21.93
33.78
31.56
Grad
35.28
28.65
29.36
22.52
33.93
32.04
During the initial development, we also tried the loss-based (or error-based) selection [44], showing a good performance as well, where GradNCP consistently exhibited better performance (see Table 7, e.g., 33.99 + 35.28 dB in UCF-101). We believe this is because the loss-based selection an approximation of GradNCP and thus GradNCP can provide better data pruning results. Specifically,
19

---

assuming that the penultimate feature has a bounded norm, i.e. ||o(-)1 S C' where C E R*, the GradNCP score can be approximated to loss-based selection as follows: (y - (x)))((((x 1J ||] S Cily f(x)]. Since some setups consider SIREN (i.e., sine activation), the assumption holds for these setups, thereby showing similar performance. While this condition does not always hold for general NF architectures that use ReLU activations (e.g.. NeRV and FFN), thereby showing consistent improvement than loss-based selection under NF architectures with ReLU activations. Therefore, we emphasize that our gradient- selection can provide more accurate results than the loss-based selection, regardless of NF architectures.
lyy- - fx
D.4
Last Layer Gradient Norm Statistic of Coordinates
Last layer gradient norm
Last layer gradient norm
0.5
15
100%
100%
50%
50%
25%
0.4
25%
10
12.5%
12.5%
0.3
5-
0.2
0
10,000
20,000
30,000
10,000
20,000
30,000
Number of coordinates (sorted)
Number of coordinates (sorted)
(a) 1:
(b) = 15
Figure 10: Last layer gradient norm statistic of coordinates where the indexes are sorted with the loss value and the highlighted region indicates the pruned context set (with gradient norm- corm-based) by given sampling ratio aper-pprameter y x 100 (%). The loss is measured on the meta-training set. indicates the adaptation step number. We meta-learn SIREN on CelebA (178 x 178) with GradNCP.
To understand the behavior of GradNCP, we analyze the gradient norm statistics of the coordinates. Here, we visualize the last layer gradient norm of the given context set Crun at adaptation iteration k namely { RR xx (x. y) E Crul1). ). Figure 10 shows the loss statistics where the indexes are sorted by the loss value. As shown in the figure, the distribution of the sample importance is quite similar to the Pareto distribution (or the long tail distribution; as the main text), indicating that selecting the elements with high scores can represent the full context set.
D.5
Comparison with Memory-Efficient Meta-Learning Methods
Table 8: Comparison with first-order gradient-base based meta-learning schemes SIREN meta-learned under the CelebA (178x 178) dataset.
Method
Kmnin
Kis
PSNR
FOMAMI [15]
16
16
25.43
Reptile [41]
16
16
32.98
GradNCP (Ours)
16
16
39.80
FOMAML [15]
128
128
37.27
Reptile [41]
128
128
39.87
GradNCP (Ours)
16
128
46.50
We also compare GradNCP with other efficient optimization-baser meta-learning schemes, including first-order MAML (FOMAML) [15] and Reptile [41]. As shown in Table 8, GradNCP significantly outperforms the basclines by large margin, even when we consider extremely long-horizon training for first-order meta-learning schemes, i.c., we consider first-order methods with 8 times longer adaptation during meta-training than GradNCP. This observation is consistent with prior works [11], which have also shown that first-order meta-learning schemes tend to struggle with NF learning tasks. Given this, we believe that efficient second-order meta-learning techniques such as GradNCP will be promising direction in this field.
20

---

D.6 Context Set Selection for Bootstrapping
Table 9: Comparison of context set choice for generating the target model on SIREN meta-lcarned with CelebA (178x 188). We consider no target correction (None), random sampling (Random), gradient norm based pruning (Norm-based), and the full context (Full) when adapting the target.
Context set
PSNR (1)
SSIM (t)
LPIPS (4)
None
37.49
0.952
0.017
Random
37.56
0,951
0.016
Norm-based
38.19
0.958
0.013
Full (Ours)
38,90
0.967
0.009
We performed an experiment to investigate whether the gain of the bootstrapped correction mainly comes from the information recovery (by using the full context set) or the longer horizon effect. To this end, we examined various selection schemes for adapting the full context target, including random context (online), gradient norm-based pruned context, and full context set. The results, presented in Table indicate that adaptation using the full context set is indeed effective, and the improvement is attributed to the recovery of information by the target model. Note that even random sampling and gradient norm-based pruning also improve the performance as they additionally provide more samples but less so than the full context set.
D.7 Cross-domain Adaptation Experiments
Table 10: Cross-domain reconstruction performance (PSNR; dB) of meta-learned SIREN under
UCF-101 (128 x 128x 16) dataset. We adapt the network to a different dataset and modalities.
Modality
Dataset
Method
PSNR (t)
CelebA (128x 128)
Learnit / MAML [61]
27.74
GradNCP (Ours)
28.57
Image
Imagenette (128x 128)
Learnit / MAML [61]
25.18
GradNCP (Ours)
26.32
Kinetics-400
Learnit / MAML [61]
26.42
Video
(128x 128x 16)
GradNCP (Ours)
27.41
We also consider the cross-domain adaptation scenario: we adapt the meta-learned model on different datasets or even different modalities from the meta-training. In particular, we train our method on UCF-101 and adapt to two different image datasets (CelebA and Imagenette) and one video dataset (Kinetics-400) [28]. Table 10 summarizes the results: GradNCP significantly improves performance over the baseline, indicating GradNCP has learned transferable initialization from the diverse motion of UCF-101.
D.8 Additional Comparison with TransINR under Test-time Adaptation
Table 11: Comparison with TransINR under same test-time adaptation (TTA) steps on SIREN meta- learned with CelebA (178 x 178) dataset, We further adapt the same adaptation steps (as GradNCP) from the predicted network from TransINR.
Method
PSNR (t)
SSIM (t)
LPIPS (L)
TransINR [6]
32.37
0.913
0.068
TransINR [6] + TTA
34.12
0.932
0.046
GradNCP (Ours)
40.60
0.976
0.005
To further verify the superiority of GradNCP, we additionally compare with the test-time adaptation performance TransINR. Here, we use SGD with the same adaptation steps (as GradNCP) to further optimize the predicted NF by TransINR. As shown in Table 11, GradNCP significantly shows better results even under this scenario. Still, note that such comparison is not fair for GradNCP (in terms of # of parameters), as TransINR additionally uses a large Transformer encoder over GradNCP.
21

---

D.9
Effectiveness of Using Full Context Set for Meta-testing
44
Test Test PSNR (dB)
36
Pruned Context
32
Full Context
16
20
24
28
Test-time adaptation step (k)
Figure 11: Comparison test reconstruction performance between the utilization of full context set (Full Context) and gradient Norm-based pruned context set (Pruned Context) during meta-test. The experiment is conducted over a meta-learn SIREN on CelebA (178x x 178) dataset with GradNCP.
To verify that the utilization of full context set for meta-testing is truly effective, compare the adaptation performance when using full and the gradient norm-based pruned context set. Here, we only apply the gradient scaling when using the full context set, pruned context set gradient norm mismatch does not occur between the meta-training and testing. As shown in Figure 11, using the full context for the meta-test time indeed effective and shows consistent improvement over the pruned context set. Note that using pruned context set is also quite effective as it shows better adaptation performance than Learnit on a long adaptation horizon.
D.10 GradNCP for General Meta-Learning, Setup
Table 12: Many-shot in-domain adaptation accuracy (%) on mini-ImageNe datasets. We train Conv4 on 5-way and way classification by using the same memory constraint and the same number of inner-adaptation steps during meta-testing for fair comparison. Reported results are averaged over three trials, subscripts denote the standard deviation, and bold denotes the best result.
Method
5-way 100-shot
10-way 50-shot
MAML [15]
66.03=0.82
48.95
GradNCP (Ours)
73.45 -3.45-0.83
55.71
While we primarily focused on resolving the memory issue of meta-learning NFs due to their unique aspects different from common meta-learning setups (i.e., training with extremely large context sets which suffers from severe memory burden), we show that GradNCP can be used for general meta-learning setups that uses second-order gradient-based meta-learning. To this end, we consider many-shot classification (c.g..100-shoo scenario mini- Imaneeet verify the efficacy of GradNCP. As shown Table 12, GradNCP significantly outperforms MAML under the given memory constraint, e.g., 66.03 + 73.45 %. Based on this, we believe exploring GradNCP for general meta-learning setups also has great potential for application that requires second-order gradient-bar based meta-learning, e.g., learning automatic reweighting [46, 24].
D.11 More Qualitative Comparison with Baselines on High-resolution Signals
Ground Thuth
Learrit
TronstNR
GradKCPP (Ours)
Ground Truth
Learrit
Transtivl
GadNCP (Ours)
Figure 12: Qualitative comparison between GradNCP and baselines on AFHQ 51)2
22

---

Billu
Bollv
Bofv
Bollv
Bolty
Bolly
Ground Truth
Learnit
GradNCP (Ours)
Ground Truth
Learnit
GradNCP (Ours)
Figure 13: Qualitative comparison between GradNCP and baselines CelebA-HQ (1024x 1024).
We compare GradNCP with Learnit as TransINR suffers from the out-of-memory issue.
Ground Truth
Learnit
GradNCP (Ours)
t=
t=3 3
t= 6
t
12
t 15
(a) UCF-101 (128x 128x 16) on SIREN
Ground Truth
Learnit
GradNCP (Ours)
t= 0
t=4
t=8
t=
12
t= 16
t=
20
== 24
t 28
(b) UCF-101 (256x256x32) on NeRV
Figure 14: Qualitative comparison between GradNCP and Learnit on UCF-101 dataset.
23

---

E
More Visualizations of Selected Context Points via GradNCP
Selected Context
Zoomed
Difference
Reconst
Ground truth
16
8
== 16
(a) CelebA-I1Q (1024x 1024)
Selected Context
Zoomed
Difference
Reconst
Ground truth
==1
==
k= 16
==1
k= 16
(b) AFHQ (512x512)
Selected Context
HVI
BMG
BMG
Zoomed in
pANG
ALL
VQLS 4 In
HV1
Difference
OMG
-PbM
Reconst.
LIVZ
HV
HV
ANO
bMG
SekOz
Ground truth
ce
ceLTXAL
SehoZ
Sehoz
16
k= 16
(c) Text (178x 178)
Selected Context
Zoomed in
Reconst. Difference
Ground truth
ImageNet (256x256)
Figure 15: Visualization of selected context points (top), residuals relative to the original signal (middle), and the reconstruction (bottom) of GradNCP trained on (a) CelebA-HQ, (b) AFHQ, (c) Text, and (4) ImageNet. Selected coordinates are highlighted in red (top) where our selection scheme focuses on only 25% of available points each of the adaptation steps. GradNCP first focuses on modeling global structure while subsequently learning high-frequency details.


Reviews:
[{"review_id": "SwLBetMSGb", "reviewer": "Reviewer_hvG5", "summary": "The paper proposes a set of new techniques for optimizing neural fields with meta-learning. The new techniques includes using gradient norms to prune less important examples, bootstrapping to correct for myopia of the inner loop optimization, and gradient rescaling to correct for different context sizes at train and test time. The method shows improved reconstruction on a diverse set of data across multiple modalities. The paper also does ablation experiments and analysis to understand how each component affects performance. ", "questions": "- In meta-learning for classification literature, works have found that pretrained initializations with fine-tuning are competitive with meta-learning approaches. It would be useful to see the baseline of standard batch training without meta-learning in the context of NFs (i.e. use first-order gradients and no outer loop). Considering one motivation of GradNCP is to allow for larger context size, I would think standard training could perform well given that you can use as large of a context size as you want with gradient accumulation.  \n\n", "limitations": "Authors adequately address the limitations of their method. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is overall well written, and the method is straight-forward. \n\n- Context-pruning for meta-learning seems like a generally applicable technique that could benefit applications beyond neural fields. \n\n- GradNCP improves over current state-of-the-art neural field meta-learning techniques on most of the standard benchmarks.\n\n- The ablation experiments and analysis give good insight on how each component of GradNCP affects reconstruction quality. \n", "weaknesses": "- I am unclear on the motivation given by the paper. The stated motivation for GradNCP is improved reconstruction quality using Neural Fields. The main application of neural fields and the one where reconstruction quality is the focus is neural radiance fields. However, the paper does not mention NeRF or do any evaluations on NeRF benchmarks. For reference, [1] runs experiments on the SRN Cars dataset. I’m wondering why this is the case, and if GradNCP cannot be used for NeRFs I think it should be made clear. If GradNCP improves reconstruction quality or optimization efficiency for NeRF this could be highly impactful.  If the motivation is learning higher quality reconstructions on high resolution images and videos, then my question is what is the motivation or application for that? \n\n- I think the contributions for bootstrapped corrections are not made clear in the paper. In the introduction, the paper proposes bootstrap correction as one of the contributions, but does not mention [2]. Later the paper mentions [2] and say that it’s discussed in the related works, but I didn’t see any discussion of [2] in the related works in the appendix. It would help to clarify the papers contributions and discuss relevant work more thoroughly. For example, this paper [3] uses gradient-norms to select training examples. \n\n1. *From data to functa: Your data point is a function and you can treat it like one*\n2. *Bootstrapped Meta-Learning*\n3. *Deep Learning on a Data Diet: Finding Important Examples Early in Training*\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer hvG5,\n\nWe sincerely appreciate your efforts and comments to improve the manuscript. We respond to your comment in what follows.\n\n---\n\n**[S2] GradNCP could be beneficial for meta-learning applications beyond NFs.**\n\nThank you for your insightful comment! We primarily focused on resolving the memory issue of meta-learning NFs due to their unique aspects different from common meta-learning setups, i.e., training with extremely large context sets (can be >1 million) which suffers from extreme memory burden. Note that this issue easily becomes more fatal since second-order gradients are necessary for meta-learning NFs [1] in contrast to other meta-learning problems that sometimes show comparable performance with memory-efficient first-order methods.\n\nNevertheless, we also agree that GradNCP can be used for general meta-learning setups. Accordingly, we consider a many-shot classification (e.g., 100-shot) scenario on mini-ImageNet to verify the efficacy of GradNCP. As shown in Table 1 in the attached pdf, GradNCP significantly outperforms MAML under the given memory constraint, e.g., 66.03$\\to$73.45\\%. Based on this, we believe exploring GradNCP for general meta-learning setups also has great potential.\n\n---\n\n**[W1] GradNCP for NeRFs.**\n\nWhile NeRF is an interesting application of NFs, we primarily focused on achieving higher reconstruction quality of *higher resolution* signals (including image, and video) due to various applications. For instance, a recently emerging paradigm in exploiting NFs, first converts signals into NFs by reconstructing them and casting them for various downstream tasks (e.g., generation or compression [1,2]). In particular, these approaches have shown great promise, e.g., [2] outperforms image, video, and audio codecs in compression, [3] achieves state-of-the-art results in 3D shape generation each of 3D shape data and do generative modeling in this NF weight space (i.e., not NeRF), to name a few. Hence, prior works highlight the importance of the *fast* reconstruction of NFs [1,2,4], where scaling to high-resolution signals was also a challenge.\n\nNonetheless, we also agree that NeRF is also an important application of NFs. To this end, we apply GradNCP for the NeRF experiment on ShapeNet Car by following the experimental setup in [4]. Here, we train NeRFs with a given single view of the 3D scene and evaluate the performance on other unseen views. As shown in Table 2 in the attached pdf, GradNCP significantly outperforms other baselines including, Learnit (e.g., 22.80$\\to$24.06 dB), and even achieves better performance than TransINR which utilizes an additional (large) Transformer. We believe extending GradNCP for scaling large-scale NeRF training (e.g., Google street view) will be an interesting future direction to explore, where we believe the efficiency of GradNCP can contribute in this field.\n\n---\n\n**[W2-1] Contributions of bootstrapped correction.**\n\nThe contributions of bootstrapped correction are mainly two-fold. First, information loss correction by minimizing the distance between the meta-learner and the bootstrapped target adapted with the full context set. This is a unique contribution of GradNCP where we believe it can also be applied to prior methods that use the target to regularize the meta-learner [5] (e,g., use full context for training the target). The second is to further reduce the myopia of the inner optimization, i.e., short-horizon bias, which is known to be the effect of [5]. In particular, achieving better performance in long horizon optimization is crucial, where the joint usage of (i) many inner steps by using reduced memory with context pruning and (ii) bootstrap correction, highly improve the long horizon performance (see Figure 5 (b) in our manuscript).\n\n---\n\n**[W2-2] Discussion of the gradient norm-based selection.**\n\nThank you for pointing this out. The gradient norm w.r.t the full parameter is a commonly used selection scheme for data pruning [6], but is very inefficient in our situation that needs to select samples at *every training iteration* (i.e., online). Specifically, it requires the gradient of every individual sample, which is too computationally expensive to compute repeatedly. On the other hand, GradNCP uses the gradient norm w.r.t the last layer which can be efficiently calculated with a single forward pass, thereby enabling to use for online selection. Furthermore, in Figure 4 and Appendix D.1 in our manuscript, we show that our last layer gradient norm and the full gradient norm have a very high correlation, indicating GradNCP is a quite reasonable approximation for sample importance.\n\n---\n\n**[Q1] Pre-training and finetuning for NFs.**\n\nUnlike conventional meta-learning for classification, standard batch training for NFs is hard to capture the knowledge of the given task distribution. Note that for meta-learning NFs, each data itself (e.g., image) becomes a meta-learning task where each task consists of the same input coordinates with different output signal values. Therefore, standard batch training in NFs performs *regression* on the same input with multiple output targets, which does not lead to a good initialization (e.g., for image, it is a similar effect of training NFs on an averaged pixel value of all CIFAR-10 data). Due to the absence of effective pre-training schemes for NFs, prior research highlighted the importance of meta-learning to learn the initialization of NFs [4].\n\n---\n\n**Reference**\\\n[1] From Data to Functa: Your Data Point is a Function and You Can Treat it Like One, ICML 2022\\\n[2] Modality-Agnostic Variational Compression of Implicit Neural Representations, ICML 2023\\\n[3] HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion, ICCV 2023\\\n[4] Learned Initializations for Optimizing Coordinate-Based Neural Representations, CVPR 2021\\\n[5] Bootstrapped Meta-Learning, ICLR 2022\\\n[6] Deep Learning on a Data Diet: Finding Important Examples Early in Training, NeurIPS 2021"}, {"review_id": "KUOczGDG92", "reviewer": "Reviewer_EMYi", "summary": "This paper addresses the issues of high memory usage and myopia problem in optimization-based meta-learning for large-scale neural field training. Three techniques are proposed: 1) online context pruning improves memory efficiency by using a selected context set in the inner learning problem, 2) bootstrap correction counteracts the information loss introduced by the context pruning, and 3) gradient scaling is introduced at meta-test time to allow the use of full context set. Experiments have demonstrated improved memory efficiency and model performance of the proposed method in comparison with existing models.", "questions": "Please check the weaknesses mentioned above.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The presentation of the work is clear and well-structured: The problem setting of memory efficiency in optimization-based meta-learning on large-scale tasks is important; The techniques of context pruning and bootstrap correction are straightforward and sound; Empirical results are also extensive.", "weaknesses": "1. It looks like the derivation of Equation 2 relies on the linearity of the last output layer. How about the setting when the final layer is a more general function? Is Equation 2 applicable in such a general setting?\n2. Could the author explain the jittering in Figure 3c and 5c? From the two figures, it seems that the training of the meta-model is not stable. \n3. I think the ablation study could be more sufficient. In Figure 5b, does the “test-time adaptation steps” refer to $K_{test}$ or $L$? There should be ablation studies on $L$ in bootstrap correction and the effect of gradient rescaling at test time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer EMYi, \n\nWe sincerely appreciate your efforts and insightful comments to improve the manuscript. We respond to each of your comments one-by-one in what follows. \n\n---\n\n**[W1] GradNCP for non-linear last layer architectures.**\n\nWe follow the setup of common design choice for NFs [1,2,3,4] that uses the linear output layer and derive Eq. 2. For the non-linear last layer, one can simply calculate the gradient norm w.r.t last as well as follows:\n\n$\\Big\\lVert \\big(y- f (x) \\big) \\sigma'(W \\phi(x) + b)\\big[\\phi(x), \\mathbf{1} \\big]^{\\mathsf{T}}  \\Big\\rVert$\\\nwhere the output of NF is $f(x) = \\sigma(W \\phi (x) + b)$, $\\sigma$ is the non-linear function, e.g., Sigmoid, and $\\sigma'(x)=\\frac{d\\sigma(x)}{dx}$. Note that this score also requires only a *single* forward pass to compute, thereby highly efficient. We thank the comment and in the revision, we will mention that Equ. 2 is for NFs with the linear output layer and add the score function for the non-linear output layer as well.\n\n---\n\n**[W2] Instability in the meta-training process.**\n\nWe first note that such an unstable training process is quite prevalent in meta-learning NFs, e.g., in the conclusion section of [5] and Figure 12 of [6]. Specifically, it is a common observation in meta-learning that models without non-skip connections may suffer from training instability [7] (note that many NFs do not use skip connections [1,2]). We believe a similar phenomenon is taking place in our experiments. Still, we emphasize that the final performance is usually not affected by the instability during meta-training, and the performance gain of GradNCP is significant compared to the baselines while sharing the same instability issue.\n\nNevertheless, we think stabilizing meta-learning NFs is an important and interesting future direction where GradNCP can also be beneficial in this direction. Given that the prior work highlighted the importance of task batch size for stabilizing meta-learning NFs (Appendix B.2 of [2]), we believe increasing the task batch size with the reduced memory by context pruning with GradNCP can be an effective approach for stable training.\n\n---\n\n**[W3-1] Clarification of Figure 5b.**\n\nWe clarify that the “test-time adaptation steps” refer to $K_{\\text{test}}$ (not $L$): note that we do not use bootstrapped correction during meta-testing.\n\n---\n\n**[W3-2] Ablation study of test-time gradient re-scaling.**\n\nThank you very much for pointing it out. We have the ablation study of test-time gradient re-scaling in Appendix D.2; we will explicitly mention it in our final manuscript. In this experiment, we showed how the using test-time gradient re-scaling affects the reconstruction performance when using a full context set at the meta-test time. We also provide the analysis that the magnitude of the gradient norm significantly differs when using full context and pruned context sets, which indicates the necessity of test-time gradient re-scaling.\n\n---\n\n**[W3-3] Ablation study of $L$ in bootstrapped correction.**\n\nThank you very much for your constructive comment. Following the reviewer’s suggestion, we perform an ablation study regarding the step for generating bootstrapped target $L$. As shown in Table 3 in the attached pdf, we find that the performance rapidly improves until $L=5$ and observed moderate improvement by further increasing $L$. During the development, we found that $L=5$ is already quite effective across all modalities and datasets, and we believe tuning the hyperparameter will improve the performance.\n\n---\n\n**Reference**\\\n[1] Implicit Neural Representations with Periodic Activation Functions, NeurIPS 2020\\\n[2] Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, NeurIPS 2020\\\n[3] NeRV: Neural Representations for Videos, NeurIPS 2021\\\n[4] Scalable Neural Video Representations with Learnable Positional Features, NeurIPS 2022\\\n[5] From data to functa: Your data point is a function and you can treat it like one, ICML 2022\\\n[6] COIN++: Neural Compression Across Modalities, TMLR 2022\\\n[7] How to train your MAML, ICLR 2019"}, {"review_id": "wAdOVQNk8d", "reviewer": "Reviewer_3CuA", "summary": "This paper presents an efficient optimization-based meta-learning technique for large-scale neural field training. It introduces automated online context point selection, resulting in significant memory savings and improved model quality. The authors also propose a bootstrap correction to enhance the meta-learned initialization, reducing errors introduced by reduced context sets and addressing the myopia of optimization-based meta-learning. The method achieves state-of-the-art results in signal reconstruction across multiple modalities, providing insights into the algorithmic components of the approach.", "questions": "(1) It would be helpful to understand the criteria or scoring mechanism used for online context point selection in GradNCP. How does the algorithm determine the importance of each data point and decide which subset to focus on during each learning iteration?\n\n(2) The paper mentions the bootstrap correction to minimize errors introduced by reduced context sets. Could the authors provide more details on how the bootstrap correction works and how it improves the quality of the meta-learned initialization?\n\n(3) Are there any specific limitations or trade-offs of GradNCP that should be considered? Are there scenarios or types of signals where GradNCP might not perform as effectively?\n", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper demonstrates several strengths across the dimensions of originality, quality, clarity, and significance.\n\n**Originality**: This paper introduces Gradient Norm-based Context Pruning (GradNCP). This original approach combines curriculum learning and data pruning techniques to address the computational burden and myopia issues in optimization-based meta-learning for neural field training.\n\n**Quality**: The paper demonstrates high technical quality through a well-formulated GradNCP framework and extensive empirical evaluation, showcasing state-of-the-art results on multiple datasets and modalities. The algorithmic component analysis further enhances the proposed method's quality and understanding.\n\n**Clarity**: The paper effectively communicates its motivations, methods, and findings, providing a clear overview of neural fields and their challenges.\n\n**Significance**: The paper is important in the field as it addresses memory limitations and myopia issues, providing a scalable and efficient solution for large-scale neural field training. ", "weaknesses": "\n(1) **Clarity of motivation**:  The motivation behind choosing optimization-based meta-learning is not very clear.  Can the authors provide a more detailed explanation highlighting the advantages of optimization-based meta-learning for large-scale neural field training and its prior successful applications?\n\n(2) **Lack of comparative experiments**: The paper does not include comparative experiments with other optimization-based meta-learning methods, such as MAML. To enhance comparability, the authors should consider conducting comparative experiments to evaluate GradNCP's performance relative to these methods in terms of efficiency, memory utilization, and reconstruction performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 3CuA, \n\nWe sincerely appreciate your efforts and insightful comments to improve the manuscript. We respond to each of your comments one-by-one in what follows. \n\n---\n\n**[W1] Clarification on the motivation for using optimization-based meta-learning.**\n\nThe motivation for using optimization-based meta-learning is mainly two-fold; note that there mainly exist two meta-learning approaches for neural fields (NFs) training which are optimization-based and amortization-based meta-learning. \n\nFirst, optimization-based meta-learning is modality and model agnostic: it does not require any (domain-specific) encoder architectures, showing a versatile range of applications. On the other hand, amortization-based meta-learning (e.g., TransINR) often requires significant modification (or even limited) when using it for each modality and NF architecture. For instance, TransINR requires (a) tokenization, which is not straightforward to apply for spherical coordinate datasets (e.g., ERA5), and (b) notable modifications to the architecture when it consists of non-MLP layers (e.g., NeRV) as the framework is specific to MLPs (see Line 303).\n\nMoreover, optimization-based meta-learning is specialized for NF reconstruction as it directly optimizes the loss of the given context set. Meanwhile, amortization-based meta-learning is known to suffer from the context set underfitting [1,2] as it is difficult to predict the parameter that minimizes the loss at once, somewhat limiting the performance of the reconstruction. It is worth noting that MAML outperforms TransINR in most cases (see Table 1 in the main text).\n\nFinally, we remark prior works in meta-learning NFs, largely focus on using optimization-based meta-learning schemes due to these strengths [3,4,5]. Also, an emerging paradigm in NF research is to convert signals into NFs and then utilize them for downstream tasks (e.g., generation, compression, and classification [4,5,6]) where these works also leverage optimization-based meta-learning to train large-scale NFs.\n\n---\n\n**[W2] Comparison with other optimization-based meta-learning methods such as MAML.**\n\nWe carefully remark that we already extensively compared GradNCP with the optimization-based meta-learning method, i.e., MAML (note that Learnit and MAML are equivalent), under multiple evaluation metrics including, memory efficiency (see Figure 3 (a)), run-time efficiency (see Figure 3 (b)) and reconstruction performance (all tables in the main manuscript). Furthermore, we additionally compared with other optimization-based meta-learning schemes in Appendix D.4 (e.g., first-order MAML and Reptile); we will explicitly mention it in our final manuscript.\n\n---\n\n**[Q1] Question regarding the selection criteria.**\n\nWe focus each learning step on the subset of data with the highest expected immediate improvement in model performance, which is approximated efficiently as the gradient norm w.r.t the last layer (see the Equation after 154 in the main paper), i.e., we select the samples that have the higher gradient norm. We show that such criteria only require a single forward pass to compute (see Eq. (2)), thereby can be utilized in an *online* manner (i.e., iteratively use during training) due to its computational efficiency.\n\n---\n\n**[Q2] How Bootstrapped correction improves the performance.**\n\nBootstrapped correction improves the performance as it regularizes the parameter adapted with the pruned context $\\theta_K$ to be close to a better-performing parameter (which is the bootstrapped target $\\theta_{K+L}^{\\text{boot}}$). Note that bootstrapped target is further optimized $L$ steps with *the full context set*, hence, it is not only a better-performing parameter but also contains the information of pruned-out context points. We experimentally verify that the performance of bootstrapped target $\\theta_{K+L}^{\\text{boot}}$ is consistently better than $\\theta_K$ in Figure 5 (c), providing a well-conditioned target for the regularization (also see the effect of using the full context set in Appendix D.6).\n\n---\n\n**[Q3] Limitations or trade-offs of GradNCP.**\n\nWhile GradNCP is possible to meta-learn on the highest resolution signal than the prior works, still it might be challenging to meta-learn NFs on exceptionally high-resolution signals (e.g., long 8K movie) where even a *single* forward pass is even not possible. We believe that a great variety of techniques can be developed in this direction, e.g., iterative searching of coordinates with high gradient norms by starting from a low-dimension grid and incrementally increasing the resolution of the sampled area.\n\n---\n\n**Reference**\\\n[1] Attentive Neural Processes, ICLR 2019\\\n[2] Neural Processes with Stochastic Attention: Paying More Attention to the Context Dataset, ICLR 2021\\\n[3] Learned Initializations for Optimizing Coordinate-Based Neural Representations, CVPR 2021\\\n[4] COIN++: Neural Compression Across Modalities, TMLR 2022\\\n[5] Your data point is a function and you can treat it like one, ICML 2022\\\n[6] Modality-Agnostic Variational Compression of Implicit Neural Representations, ICML 2023\n\n"}, {"review_id": "11MR33Yr9D", "reviewer": "Reviewer_9F3K", "summary": "This paper proposes a way to sample the subset of the dataset to improve convergence speed and the eventual accuracy in the meta-learning framework, especially for neural fields. They also propose a bootstrap correction to resolve the well-known myopia issue. Moreover, gradient re-scaling during meta-testing also improved the performance. They have provided the experimental results on nine datasets, showing superior performance compared to the baseline methods.", "questions": "The questions are embedded in the weakness section.", "limitations": "I do not see any potential negative societal impact from this manuscript.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and easy to follow. The method is simple and easy to apply in the existing meta-learning frameworks, and the performance improvements are not marginal.\n\n2. It also provides very comprehensive experimental results, showing the effectiveness of each component very well (although I have some concerns, see the weakness section).\n", "weaknesses": "Here, I am writing both weaknesses and the questions in this section since both are quite related (but not necessarily all questions are weaknesses). Depending on the rebuttal and other reviewers' opinions, I would revise my score.\n\n1. Although the authors have provided extensive experimental results, I think the current setup would not clearly reveal the actual effects of the newly proposed components. First, the main contribution of this paper is ‘norm-based context pruning’, and I believe this paper should have compared against other data pruning or importance sampling methods (for example, you can refer to [1] for more related works on this). At least, the authors could have compared gradNCP to simple loss-based top-k sampling, which is much simpler than the proposed method. They are using the norm of the gradient of the last layer to measure the importance, and I think the higher loss, the higher the gradient norm. Hence, simple loss-based top-k sampling would also work well.\n\n2. I strongly encourage the authors to provide enough related works on data pruning or the importance sampling studies.\n\n3. The effect of ‘meta-training w/ gradNCP’ was not adequately tested. The proper setup would be (Random initialization + test-time gradNCP) vs. (MAML + test-time gradNCP) vs. the proposed method. In this way, the authors can convince that meta-training w/ gradNCP is really necessary and effective.\n\n4. In L145-146: I respectively disagree with this statement. The authors have only tested ‘reconstruction’ tasks, arguing that generalization performance is not important in neural fields. I think the most interesting applications of neural fields are related to generalization, such as novel view synthesis (NeRF) or 3d surface reconstructions (not just point-wise reconstructions). And IMHO, the current image and video reconstruction experiments done in this paper are very far from practical usage. The compression rate is nowhere close to the state-of-the-art image and video codecs, and the encoding time for a higher-compression rate is too slow to use in practice. \n\n5. Why only neural fields setup? The main claim of the paper is the effectiveness of the gradNCP in neural fields, but I do not see any specific treatment for neural fields. In other words, gradNCP can apply to any meta-learning setup? This question is not necessarily a criticism but more on the possibility of strengthening the paper.\n\n[1] Not All Samples Are Created Equal: Deep Learning with Importance Sampling, Katharopoulos et al., ICML 2018.\n\n-minor comments-\nL128: C^(i) was the set of coordinates of i-th signal, but here C^k denotes the set of coordinates of k-th step.\nL142: {x,y} -> ({x,y})\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 9F3K,\n\nWe sincerely appreciate your efforts and insightful comments to improve our manuscript. We respond to each of your comments one-by-one in what follows.\n\n---\n\n**[W1] Comparison with other data pruning. e.g., loss-based top-K selection.**\n\nFirst, we carefully remind the reviewer that we already compared with other data pruning methods such as SPG and TPG [1] (see Figure 3 (c)), where GradNCP shows almost the same performance while \bsignificantly reducing the time complexity for data selection.\n\nMoreover, we are very happy that the reviewer has pointed out the loss-based selection. During the initial development, we also tried the loss-based selection, showing a good performance as well, where GradNCP consistently exhibited better performance (see Table 4 in the attached pdf, e.g., 33.99$\\to$35.28 dB in UCF-101). We believe this is because the loss-based selection is an approximation of GradNCP and thus GradNCP can provide better data pruning results. Specifically, assuming that the penultimate feature has a bounded norm, i.e., $||[\\phi(.)~ 1]|| \\le M$ where $M\\in R^{+}$, the GradNCP score can be approximated to loss-based selection as follows: $||(y-f(x)) [\\phi(x)~ 1]^{T} || \\le ||y-f(x)|| ~ || [\\phi (x)~1] || \\le M ||y-f(x)||$. Since some setups consider SIREN (i.e., sine activation), the assumption holds for these setups, thereby showing similar performance in some cases, e.g., CelebA. We thank the comment and will describe the relationship between GradNCP and the loss-based selection in the revision.\n\n---\n\n**[W2] More related works on data pruning.**\n\nThank you for the comment. In addition to our related work section on data pruning in Appendix C, we will revise our manuscript with more comprehensive studies of data pruning literature, including the reference that you mentioned [2].\n\n---\n\n**[W3] Effect of meta-training with GradNCP is not tested due to test-time GradNCP: Use test-time GradNCP for MAML.**\n\nWe would like to clarify that the only difference between MAML and GradNCP for meta-testing is the gradient re-scaling which *has no effect for MAML*. Specifically, gradient re-scaling is proposed to match the gradient step size difference of meta-training and meta-testing (as GradNCP use pruned set for meta-training and a full set for meta-testing), while MAML uses the full set in both stages. Hence, the re-scaling does not affect any performance of MAML as the re-scaling constant is equal to 1.\n\n---\n \n**[W4] Clarification of reconstruction experiments and GradNCP for NeRFs.**\n\nFast reconstruction of NFs is an important basis for multiple applications in NF research. For example, a recently emerging paradigm in exploiting NFs, first converts signals into NFs by reconstructing them and casting them for various downstream tasks (e.g., generation or compression [3,4]). In particular, these approaches have recently shown great promise, e.g., [4] outperforms image, video, and audio codecs in compression, [5] achieves state-of-the-art results in 3D shape generation by reconstructing (not generalizing) each of 3D shape data and do generative modeling in this NF weight space, to name a few. Hence, prior works highlight the importance of the *fast* reconstruction of NFs [3,4,5], where we also followed the prior work’s experimental setup [6].\n\nNonetheless, we also agree with the reviewer that generalization is also an important application of NFs. To this end, we apply GradNCP for the NeRF experiment on ShapeNet Car to verify the generalization performance of GradNCP. Here, following the setup in [6], we train NeRFs with a given single view of the 3D scene and evaluate the performance on other unseen views. As shown in Table 2 in the attached pdf, GradNCP significantly outperforms other baselines including Learnit (e.g., 22.80 $\\to$ 24.06 dB), and even achieves better performance than TransINR which utilizes an additional (large) Transformer. We believe using GradNCP for large-scale generalization of NFs (e.g., NeRF for Google street view) will be an interesting future direction to explore, where we believe the efficiency of GradNCP can contribute to this field.\n\n---\n\n**[W5] GradNCP for general meta-learning setup.**\n\nThank you for your insightful comment! We primarily focused on resolving the memory issue of meta-learning NFs due to their unique aspects different from common meta-learning setups, i.e., training with extremely large context sets (can be >1 million) which suffers from severe memory burden. Note that this issue easily becomes more fatal since second-order gradients are necessary for meta-learning NFs [3] in contrast to other meta-learning problems that sometimes show comparable performance with memory-efficient first-order methods [7].\n\nNonetheless, we also agree that GradNCP can be used for general meta-learning setups. Following your suggestion, we consider a many-shot classification (e.g.,100shot) scenario on mini-ImageNet to verify the efficacy of GradNCP. As shown in Table 1 in the attached pdf, GradNCP significantly outperforms MAML under the given memory constraint, e.g., 66.03$\\to$73.45 \\%. It is worth noting that this also highlights the *generalization ability* of GradNCP and we believe exploring GradNCP for general meta-learning setups has great potential.\n\n---\n\n**Editorial comments.**\n\nThank you very much and we will update typos in the revision.\n\n---\n\n**Reference**\\\n[1] Automated Curriculum Learning for Neural Networks, ICML 2017\\\n[2] Not All Samples Are Created Equal: Deep Learning with Importance Sampling, ICML 2018\\\n[3] From Data to Functa: Your Data Point is a Function and You Can Treat it Like One, ICML 2022\\\n[4] Modality-Agnostic Variational Compression of Implicit Neural Representations, ICML 2023\\\n[5] HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion, ICCV 2023\\\n[6] Learned Initializations for Optimizing Coordinate-Based Neural Representations, CVPR 2021\\\n[7] On First-Order Meta-Learning Algorithms, arXiv 2018"}]

Author rebuttal:
Dear reviewers and AC,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript.

As reviewers highlighted, we believe our paper tackles an interesting and important problem (EMYi, 3CuA), and provide an efficient yet effective (all reviewers) framework for meta-learning NFs, validated with extensive evaluations (all reviewers) followed by a clear presentation (all reviewers).

We appreciate your constructive comments on our manuscript. In the attached pdf, we have run the following additional experiments to clarify the reviewer's comments:
- GradNCP for general meta-learning setup, i.e., many-shot classification (Table 1)
- NeRF experiment on ShapeNet Cars dataset (Table 2)
- Ablation study of $L$: number of gradient steps for generating bootstrapped target (Table 3)
- Comparison of loss-based and gradient norm-based top-K selection (Table 4)

We strongly believe that GradNCP can be a useful addition to the NeurIPS community, in particular, due to the enhanced manuscript by reviewers’ comments helping us better deliver the effectiveness of our method.

Thank you very much!\
Authors.


Meta-review: This paper uses meta-learning to improve training of neural fields.  The proposed technique incorporates automated curriculum selection based on examination of per-example gradient norms.  After the rebuttal and discussion phase, all reviewers favor accept, citing convincing empirical results and a clearly developed method.  The AC agrees with the reviewer consensus.
Decision: Accept (poster)