SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given both sets of parameters, the desired correction effect can be achieved by regularizing  $\theta_{K}$  to be close to the bootstrap target  $\theta_{K + L}^{\mathrm{boot}}$  according to a distance function  $\mu (\theta_K,\theta_{K + L}^{\mathrm{boot}})$  with  $\ell_2$  distance being a suitable choice. This correction has two effects: Firstly, it forces increased plasticity in the initial  $K$  steps as the only means to reduce the regularization loss to a target obtained using both additional information (i.e., access  $\mathcal{C}^{\mathrm{full}}$ ) and further optimization. This is possible as information about future learning dynamics is infused in the initial steps of learning through bootstrapping. In effect, the meta-learner teaches to improve itself. Secondly, as noted in the related work [17], bootstrapping reduces the phenomenon of myopia in meta-learning - a well-known issue significantly reducing the effectiveness of longer optimization loops at meta-test time (when no second-order optimization is needed and thus longer adaption procedures are possible). Flennerhag et al. [17] show that bootstrapping for this purpose leads to guaranteed performance improvements.
Overall meta-learning objective. In practice, we find that it is useful to calculate the final loss as a combination of the target correction term and the performance of  $\theta_{K}$  on the full context set. For a given hyper-parameter  $\lambda > 0$ , the meta-objective of GradNCP becomes:
$$
\mathcal {L} _ {\text {t o t a l}} \left(\theta_ {0}; \mathcal {C} _ {\text {f u l l}}\right) := \mathcal {L} _ {\text {M S E}} \left(\theta_ {K}; \mathcal {C} _ {\text {f u l l}}\right) + \lambda \mu \left(\theta_ {K}, \theta_ {K + L} ^ {\text {b o o t}}\right), \tag {3}
$$
a choice inspired by the common meta-learning practice of optimizing the meta-loss w.r.t the inner validation set performance, infusing further information into the initial  $K$  adaptation steps. The full meta-training and -testing procedures are shown in Algorithms 1 & 2.
Connection with target model regularization frameworks. Addressing the issue of information loss through a target model can be viewed as an application of prior methods that engage the target model in instructing the meta-learner. While prior works aim to adapt a better target by further optimization [17, 31], or adapting from a better initialization [59], we believe such methods can be further improved by utilizing the full context set. Importantly, to the best of our knowledge, this work is the first attempt of introducing bootstrapping as a means to correct the effects of data pruning.
2.4 Gradient Re-scaling enables Effective Meta-test Optimization
Relieved from the memory requirements of second-order optimization and the burden of meta-learning myopia, the use of the full context set at meta-test time appears a natural choice. The naive combination of these ideas may not lead to desired results however, as the norm of the inner-loop gradient significantly deviates between meta-training and testing by construction, i.e.,  $\| \nabla_{\theta}\mathcal{L}_{\mathrm{MSE}}(\theta ;\mathcal{C}_{\mathrm{high}})\| >\| \nabla_{\theta}\mathcal{L}_{\mathrm{MSE}}(\theta ;\mathcal{C}_{\mathrm{full1}})\|$ , in accordance with the importance score  $R_{k}^{\mathrm{GradNCP}}$ . Accordingly, we use a simple correction that ensures test-time gradient statistics are on the same order as statistics optimized for during meta-training. At step meta-test step  $t$  we correct as:
$$
g _ {t} ^ {\text {t e s t}} \leftarrow \frac {\| \nabla_ {\theta_ {t}} \mathcal {L} _ {\mathrm {M S E}} \left(\theta_ {t} ; \mathcal {C} _ {\text {h i g h}}\right) \|}{\| \nabla_ {\theta_ {t}} \mathcal {L} _ {\mathrm {M S E}} \left(\theta_ {t} ; \mathcal {C} _ {\text {f u l l}}\right) \|} \nabla_ {\theta_ {t}} \mathcal {L} _ {\mathrm {M S E}} \left(\theta_ {t}; \mathcal {C} _ {\text {f u l l}}\right), \tag {4}
$$
and then update  $\theta_t$  with  $g_t^{\mathrm{test}}$  instead of  $\nabla_{\theta_t}\mathcal{L}_{\mathrm{MSE}}(\theta_t;\mathcal{C}_{\mathrm{full}})$ . We observe this to be a simple and sufficient means to drastically improving test-time performance when using  $\mathcal{C}_{\mathrm{full}}$ .
We now provide an empirical evaluation of GradNCP, systematically verifying claims made throughout the manuscript and thus supporting the suitability of its constituent components. At this point, we remind the reader of the key questions raised thus far:
1. Efficiency and context scoring. Can GradNCP's online context selection mechanism reduce memory and compute requirements vis-à-vis standard meta-learning? If so, what is the quality of the introduced scoring function  $R_{k}^{\mathrm{GradNCP}}$  and how can its approximations be justified? Finally, do selected context points correspond to humanly interpretable features?
2. Bootstrap correction and myopia. What is the effect of bootstrap correction on any irreducible error introduced through context pruning? Do we observe improvements on the issue of myopia? How do we know that bootstrapping provides a well-conditioned target model?
3. Overall evaluation. How does GradNCP compare to state-of-the-art meta-learning techniques for NFs? What are the necessary conditions for the highest-quality results at test time? Do introduced efficiency techniques enable learning on more complex signals?
Before answering each question, we first lay out the experimental protocol used throughout:
Baselines. We compare GradNCP with existing meta-learning schemes for NFs, including the MAML [15] (known as Learnit [61] in the NF literature), transformer-based meta-learning (TransINR; [6]) and the recent Instant Pattern Composer (IPC; [29]). Where appropriate, we also show the effect of tablua rasa optimization from a random initialization (Random Init.) to highlight the need for accelerated learning. Presented TransINR and MAML/Learnit results have been obtained with reference to the official implementation (and recommended hyperparameters for TransINR) while we adjust MAML/Learnit's number of inner gradient steps for a fair comparison with GradNCP. IPC values correspond to results directly published by the authors.
(a) Memory consumption
(b) Runtime efficiency
Figure 3: Comparing the learning efficiency of GradNCP to standard meta-learning. (a) GradNCP drastically decreases memory usage when varying  $K$  at meta-training time. (b) GradNCP achieves the same performance in  $55\%$  of wall-clock time  $(1.8 \times \text{speedup})$ . (c) GradNCP scoring maintains equivalent performance while reducing runtime complexity:  $\mathcal{O}(M^2)$  to  $\mathcal{O}(M)$  relative to self-prediction gain (SPG) and target-prediction gain (TPG) [21] approximated with last layer update.
(c) Context scoring quality
NF architectures. We use SIREN [54] as the base architecture for all experiments, and additionally consider NeRV [5] in the video domain. We provide more comparisons on other architectures, such as Fourier feature networks (FFN) [60] in Appendix D.3.
Evaluation setup. To ensure a fair comparison, we control for the number of adaptation steps and/or memory requirements when comparing Learnit/MAML with GradNCP. Quantitative results are reported using Peak Signal-to-Noise Ratio (PSNR; higher is better) as well as perceptual similarity for image and video datasets by using LPIPS (lower is better) [69] and SSIM (higher is better) [65]. For video datasets, we evaluate these metrics in a frame-wise manner and average over the whole video following NeRV[5]. All ablation studies presented show results on CelebA  $(178\times 178)$  using  $K = 16$  inner gradient steps,  $L = 5$  bootstrap target construction steps and  $\gamma = 0.25$  for GradNCP. The MAML / Learnit configuration with equivalent memory use corresponds to  $K = 4$  while both methods use  $K_{\mathrm{test}} = 16$  test time adaptation steps unless otherwise stated.
Datasets. Highlighting the versatility of NFs, we consider four different modalities, namely images, videos, audio, and manifolds. For images, we follow Learnit and use CelebA [36], Imagenette [23], and pictures including Text [61] while additionally considering the high-resolution fine-grained datasets CelebA-HQ [26] and AFHQ [7]. To test under high visual diversity we also use all 1,000 classes of ImageNet [8]. Pushing memory requirements past current limits, we consider the video dataset UCF-101 [57] using resolutions  $(128\times 128\times 16,256\times 256\times 32)$  to demonstrate the scalability of GradNCP. Finally, we also test on audio using the speech dataset Librispeech [42] and on manifolds by considering climate data on the globe using ERA5 [22].
3.1 Online Context Pruning leads to Highly Efficient Meta-Learning
First turning to the primary motivation for our work, we show a comparison in terms of memory efficiency and runtime in Figures 3a and 3b, noting that GradNCP leads to both a significant improvement in the memory/performance trade-off. This enables either significant runtime speedups on the same machine or meta-learning on significantly higher dimensional signals (as we will show later). Given such encouraging results, how do we know whether approximations made for  $R_{k}^{\mathrm{GradNCP}}$  in Section 2.2 are justifiable? We attempt to answer this in Figure 3c by comparing to existing curriculum-learning metrics, noting (i) that approximating the effect of a full network update using the last layer only continues to consistently outperform random sub-sampling (ii) measuring the learning effect of a context point by the reduction in error on itself does not decrease performance (showing equivalent performance of SPG and TPG) and (iii) that our proposed first-order Taylor series approximation does successfully alleviates the need for an update without performance degradation.
What explains the robustness of both approximations? In Figure 4 we remarkably notice that when used in a meta-learning setting, the correlations between the last layer and full network gradient consistently improve with the number of inner adaptation steps (unlike for a random initialization). Remarkably, this suggests that meta-learning dynamics push parameters to a state where the approximation assumptions above are satisfied while simultaneously improving performance. Moreover, as we mentioned in Figure 1, a visualization of the context set  $\mathcal{C}_{\mathrm{high}}^k$  generally corresponds to interpretable features, with many more examples of this phenomenon shown in Appendix D.1.
(a) GradNCP component analysis
(b) Myopia
(c) Bootstrap target performance
Figure 5: Analysing the effects of bootstrapping. (a) Bootstrapping reduces error introduced by context pruning and stabilizes meta-training. (b) Bootstrap correction allows learning past the meta-training horizon ( $K = 16$ ), reducing myopia. (c) The bootstrap model achieves consistently higher performance, providing a well-conditioned target.
Figure 6: Qualitative comparison between GradNCP and baselines on AFHQ  $(512\times 512)$
3.2 Understanding the Effects of Bootstrap Correction
Given the high quality of our online-context point selection scheme, what additional contributions are made by the bootstrap correction (Section 2.3)?
We answer this question in Figure 5a, noting that bootstrapping indeed leads to consistent performance improvements while also stabilizing training. Note that this effect is distinct from any reduction in myopia (as we keep  $K = K_{\mathrm{test}}$  fixed). Myopia nevertheless exists, as we show in Figure 5b, noting that both Learnit / MAML and GradNCP w/o bootstrap correction result in reduced test-time adaptation results. It is noteworthy that the rate of improvement for each additional step drastically decreases for Learnit / MAML, showing the reduced plasticity effect we discussed throughout the manuscript.
One possible concern for bootstrapping in our setting is that the GradNCP loss (3) could be trivially minimized by finding a local minima that cannot be escaped in the  $L$  additional steps afforded to the target model, thus leading to a low regularization loss but also removing any positive effect of bootstrapping. Fortunately, this is not the case, as we show in Figure 5c, clearly highlighting that the  $K$  step meta-learner consistently trails an improved target model, providing a well-conditioned regularization target.
(a) Random init.
(b) Meta-learned init.
Figure 4: Meta-learning dynamics improve approximation quality over time. Shown is the correlation between the last layer and the entire network gradient.
3.3 GradNCP achieves State-of-the-Art Results in Meta-Learning Neural Fields
Having motivated and verified GradNCP's components, we now turn to our main results, presenting results for images in Table 1, videos in Table 2, as well as audio and manifolds in Table 3. Overall, GradNCP significantly and consistently outperforms state-of-the-art meta-learning methods by a large margin, leading to high-quality NFs as shown in Figure 6, exhibiting clear superiority over the baselines in capturing high-frequency components - a direct benefit of online context pruning, due to additional learning capacity dedicated to such high-frequency details in later adaptation steps. We provide more qualitative results in Appendix D.11.
Table 1: Reconstruction performance of SIREN on image datasets of various resolutions. We report results as PSNR [dB]  $(\uparrow)$  / SSIM  $(\uparrow)$  / LPIPS  $(\downarrow)$  using the same number of adaptation steps for all schemes. OOM denotes the out-of-memory on a single NVIDIA A100 40GB GPU. Here IPC indicates Instance Pattern Composers with results directly taken from [29].
CelebA (178 × 178)	Imagenette (178 × 178)	Text (178 × 178)
Random Init.	19.94 / 0.532 / 0.708	18.57 / 0.443 / 0.810	15.37 / 0.574 / 0.755
TransINR [6]	32.37 / 0.913 / 0.068	28.58 / 0.850 / 0.165	22.70 / 0.898 / 0.085
IPC [29]	35.93 / - / -	38.46 / - / -	- / - / -
Learnit / MAML [61]	38.28 / 0.964 / 0.010	35.66 / 0.950 / 0.014	30.31 / 0.956 / 0.018
GradNCP (Ours)	40.60 / 0.976 / 0.005	38.72 / 0.972 / 0.005	32.33 / 0.976 / 0.007
ImageNet (256 × 256)	AFHQ (512 × 512)	CelebA-HQ (1024 × 1024)
Random Init.	18.72 / 0.434 / 0.839	18.57 / 0.488 / 0.856	12.21 / 0.574 / 0.820
TransINR [6]	28.01 / 0.818 / 0.199	23.43 / 0.592 / 0.573	- OOM -
Learnit / MAML [61]	31.44 / 0.887 / 0.100	28.58 / 0.751 / 0.354	27.66 / 0.781 / 0.513
GradNCP (Ours)	32.52 / 0.898 / 0.068	29.61 / 0.786 / 0.286	28.90 / 0.789 / 0.438
Table 2: Reconstruction performance of meta-learned SIREN and NeRV. We consider two different resolutions of the UCF101 dataset. OOM denotes unavailable results due to running out-of-memory on a single NVIDIA A100 40GB GPU.
Resolution	Network	Method	PSNR (↑)	SSIM (↑)	LPIPS (↓)
128×128×16	SIREN	TransINR [6]	15.14	0.360	0.636
Learnit / MAML [61]	25.46	0.720	0.363
GradNCP (Ours)	26.92	0.781	0.223
NeRV	Learnit (MAML) [61]	28.86	0.871	0.140
GradNCP (Ours)	35.28	0.959	0.015
256×256×32	SIREN	TransINR [6]	——	OOM ——
Learnit / MAML [61]	——	OOM ——
GradNCP (Ours)	22.92	0.640	0.521
NeRV	Learnit / MAML [61]	23.75	0.659	0.422
GradNCP (Ours)	28.65	0.842	0.201
Table 3: PSNR of meta-learned SIREN on (a) Librispeech and (b) ERA5  $(181\times 360)$
Method	PSNR (↑)
1 sec	3 sec
TransINR [6]	39.22	33.17
IPC [29]	40.11	35.38
Learnit / MAML [61]	39.55	31.39
GradNCP (Ours)	43.25	36.24
(b) ERA5
Method	PSNR (↑)
Learnit / MAML [61]	64.91
GradNCP (Ours)	75.11
We remark that one of the nice properties of GradNCP is its modality- and model-agnosticism. On the other hand, we find TransINR struggles to generalize for some modalities and architectures: namely it requires (a) tokenization, which is not straightforward to apply for spherical coordinate datasets (e.g., ERA5), and may require the framework to deal with sequences of tokens that are too long (e.g., videos), and (b) notable modifications to the architecture when it consists of non-MLP layers (e.g., NeRV) as the framework is specific to MLPs.
High-resolution signals. One of the significant advantages of GradNCP is its exceptional memory efficiency, which allows us to meta-learn on large-scale high-resolution signals. As demonstrated in Table 1 and Table 2, GradNCP can even be trained on  $256 \times 256 \times 32$  resolution videos or  $1024 \times 1024$  resolution images, a scale previously not possible for prior work (even under the constraint of NVIDIA A100 40GB GPU) due to their intensive memory usage (see Figure 3a).
GradNCP for generalization tasks. To also verify that GradNCP can be used for generalization task where the meta-learning context and target sets are disjoint. To this end, we apply GradNCP for the NeRF [40] experiment on ShapeNet Car [4] to verify the generalization performance of GradNCP. Here, following the setup in Tancik et al. [61], we train NeRFs with a given single view of the 3D scene and evaluate the performance on other unseen views. As shown in Table 4, GradNCP significantly outperforms other baselines including Learnit / MAML (e.g.,  $22.80 \rightarrow 24.06$  dB), and even achieves better performance than TransINR which utilizes an additional (large) Transformer. We believe using GradNCP for large-scale generalization of NFs (e.g., NeRF for Google street view) will be an interesting future
Table 4: PSNR (dB) of unseen view synthesis of NeRF on ShapeNet Cars. We adapt NeRF with a single view of a 3D scene then evaluate the quality of unseen views.
(a) Librispeech
Method	PSNR
Learnit [61]	22.80
TransINR [6]	23.78
GradNCP (Ours)	24.06
direction to explore, where we believe the efficiency of GradNCP can contribute to this field.
Figure 7: Effect of meta-test time gradient re-scaling. We apply re-scaling when adapting with full context set on SIREN trained with GradNCP.
Figure 8: Gradient norm of the full context set  $\mathcal{C}_{\mathrm{full}}$  and the gradient norm-based pruned context set  $\mathcal{C}_{\mathrm{high}}^k$  at iteration  $k$ .
(a)  $k = 0$
(b)  $k = 15$
Table 5: Effect of context set selection ratio  $\gamma$  on SIREN meta-learned with CelebA (178 × 178).  $\gamma$  controls the performance and the memory trade-off when using GradNCP.
Selection ratio (γ)	0.05	0.10	0.15	0.20	0.25	0.5	0.75	1.0
PSNR (dB)	31.20	35.00	36.67	37.59	38.90	39.25	39.70	39.98
Memory (GB)	1.93	2.41	3.05	3.45	3.91	6.31	9.22	11.04
3.4 Additional Analysis
Gradient re-scaling analysis. Here, we investigate the importance of gradient re-scaling. To show the effect of gradient re-scaling at the meta-test time, we track the test PSNR during training SIREN with GradNCP (that does not use the Bootstrapped correction) on CelebA  $(178\times 178)$ . Here, we report the PSNR when the meta-initialization is adapted with the full context set on the meta-test stage. As shown in Figure 7, the gradient re-scaling significantly improves the performance.
To further investigate the necessity of gradient re-scaling, we additionally analyze the gradient norm during the (inner loop) adaptation phase of meta-training. Here, we plot the gradient norms at the adaptation step  $k$  measured by the full context set  $C_{\mathrm{full}}$ , and the gradient norm-based pruned context set  $C_{\mathrm{high}}^k$ , respectively. As shown in Figure 8, our results indicate that the norm of the gradients exhibits significant variations, with some steps exhibiting  $3 \times$  larger gradients when using the pruned context set. This suggests that the meta-learner employed a larger step size during training, which highlights the importance of test-time gradient re-scaling. We also find that re-scaling the gradient with the loss ratio of the full context set and the sampled context leads to similar performance improvements and may serve as a faster alternative, as it eliminates the need for gradient calculation twice, i.e., showing 40.53 in PSNR where the gradient re-scaling with gradient norms shows 40.80.
Selection ratio analysis. We also have investigated the tradeoff between performance and memory requirements of GradNCP by varying the value of the context set selection ratio  $\gamma$  in our meta-training process on CelebA  $(178 \times 178)$ . As shown in Table 5, the memory usage decreases proportionally as one decreases the selection ratio, while the performance is maintained fairly well even for relatively small values of  $\gamma = 0.15$ , i.e., 36.67 (dB). For the main experiment, we mainly use  $\gamma = 0.25$ , which shows high performance while significantly reducing the memory usage, e.g.,  $75\%$  of memory.
4 Discussion and Conclusion
We propose GradNCP, an efficient and effective method for fast and scalable neural field learning through online-pruning of the context set at each meta-training iteration, significantly reducing memory usage. We demonstrate that GradNCP improves performance over various modalities and, more importantly, exhibits superior memory efficiency allowing it to be the first method capable of meta-learning on exceptionally high-resolution signals.
Future work and limitation. We believe extending GradNCP to extreme high-resolution signals (e.g., a long 8K video), where even a single forward pass is not possible, would be an interesting future direction to explore. We believe that a great variety of techniques can be developed in this direction, e.g., iterative tree-search of high gradient norm samples by starting from a low-dimension grid and incrementally increasing the resolution of the sampled area.
We thank Kyuyoung Kim and Jongjin Park for providing helpful feedbacks and suggestions in preparing an earlier version of the manuscript. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST), and No.2022-0-00713, Meta-learning applicable to real-world problems) and National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No.RS-2023-00213710, Neural Network Optimization with Minimal Optimization Costs).