SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To this end, we measure the Pearson correlation coefficient between the entire gradient norm and the last layer gradient norm on individual signals of the CelebA test dataset (i.e., correlation computed on  $178 \times 178$  elements), then report the average correlation across signals. As shown in Figure 9, first the entire gradient norm and the last layer gradient norm are both highly correlated in all cases indicating that our approximation using the last layer only optimization is quite reasonable. Furthermore, remarkably, when computed on the meta-learned initialization, the correlations between the last layer and the entire network gradient norm consistently improve during the inner adaptation steps (unlike for the random initialization). This indicates that the meta-learning dynamics push parameters to a state where the approximation assumptions above are satisfied while simultaneously improving performance.
D.2 Ablation Study on Boostrapped Target Adaptation Steps
Table 6: PSNR (dB) of SIREN meta-learned on CelebA dataset  $(178\times 178)$  when increasing the number of gradient steps  $(L)$  for generating bootstrapped target. Note that '0' indicates no bootstrapped correction (i.e., only use the gradient norm-based pruning over MAML).
L	0	1	2	5	10
PSNR (dB)	37.49	37.78	38.20	38.90	38.99
We also perform an ablation study regarding the step for generating bootstrapped target  $\theta^{\mathrm{boot}}$ . As shown in Table 6, we found that the performance rapidly improves until  $L = 5$  and observed moderate improvement by further increasing  $L$ . During the development, we found that  $L = 5$  is already quite effective across all modalities and datasets, and we believe tuning the hyperparameter will improve the performance.
D.3 Comparison with Loss-based Selection
Table 7: Comparison of data selection criteria. We report the PSNR (dB) of meta-learned neural fields across various modalities when using loss-based (Loss) and gradient norm-based (Grad) top-K data selection. Bold indicates the best result of each group.
(a) Bounded activation
SIREN
CelebA	Imagenette	Text	AFHQ	CelebA-HQ	UCF-101 (128)	UCF-101 (256)	Libri (1sec)	Libri (3sec)	ERA5
Loss	40.54	37.71	33.11	29.37	28.89	26.59	22.76	43.40	36.45	74.10
Grad	40.60	38.72	32.33	29.61	28.90	26.92	22.92	43.25	36.24	75.11
(b) Non-bounded activation
NeRV	FFN
UCF-101 (128)	UCF-101 (256)	CelebA	Text	Libri (1sec)	Libri (3sec)
Loss	33.99	28.58	28.81	21.93	33.78	31.56
Grad	35.28	28.65	29.36	22.52	33.93	32.04
During the initial development, we also tried the loss-based (or error-based) selection [44], showing a good performance as well, where GradNCP consistently exhibited better performance (see Table 7, e.g.,  $33.99 \rightarrow 35.28$  dB in UCF-101). We believe this is because the loss-based selection is an approximation of GradNCP and thus GradNCP can provide better data pruning results. Specifically,
assuming that the penultimate feature has a bounded norm, i.e.,  $\| \phi (\cdot)\| \leq C$  where  $C\in \mathbb{R}^{+}$ , the GradNCP score can be approximated to loss-based selection as follows:  $\| (\mathbf{y} - f(\mathbf{x}))[\phi (\mathbf{x})\mathbf{1}]^{\top}\leq$ $\| \mathbf{y} - f(\mathbf{x})\| \| [\phi (\mathbf{x})\mathbf{1}]\| \leq C\| \mathbf{y} - f(\mathbf{x})\|$ . Since some setups consider SIREN (i.e., sine activation), the assumption holds for these setups, thereby showing similar performance. While this condition does not always hold for general NF architectures that use ReLU activations (e.g., NeRV and FFN), thereby showing a consistent improvement than loss-based selection under NF architectures with ReLU activations. Therefore, we emphasize that our gradient-based selection can provide more accurate results than the loss-based selection, regardless of NF architectures.
D.4 Last Layer Gradient Norm Statistic of Coordinates
(b)  $k = 15$
Figure 10: Last layer gradient norm statistic of coordinates where the indexes are sorted with the loss value and the highlighted region indicates the pruned context set (with gradient norm-based) by a given sampling ratio hyper-parameter  $\gamma \times 100$  (\%). The loss is measured on the meta-training set.  $k$  indicates the adaptation step number. We meta-learn SIREN on CelebA  $(178\times 178)$  with GradNCP.
To understand the behavior of GradNCP, we analyze the gradient norm statistics of the coordinates. Here, we visualize the last layer gradient norm of the given context set  $\mathcal{C}_{\mathrm{full}}$  at adaptation iteration  $k$ , namely  $\{R_k(\mathbf{x},\mathbf{y})|(x,y)\in \mathcal{C}_{\mathrm{full}}\}$ . Figure 10 shows the loss statistics where the indexes are sorted by the loss value. As shown in the figure, the distribution of the sample importance is quite similar to the Pareto distribution (or the long tail distribution; as in the main text), indicating that selecting the elements with high scores can represent the full context set.
D.5 Comparison with Memory-Efficient Meta-Learning Methods
Table 8: Comparison with first-order gradient-based meta-learning schemes on SIREN meta-learned under the CelebA (178×178) dataset.
Method	Ktrain	Ktest	PSNR
FOMAML [15]	16	16	25.43
Reptile [41]	16	16	32.98
GradNCP (Ours)	16	16	39.80
FOMAML [15]	128	128	37.27
Reptile [41]	128	128	39.87
GradNCP (Ours)	16	128	46.50
We also compare GradNCP with other efficient optimization-based meta-learning schemes, including first-order MAML (FOMAML) [15] and Reptile [41]. As shown in Table 8, GradNCP significantly outperforms the baselines by a large margin, even when we consider extremely long-horizon training for first-order meta-learning schemes, i.e., we consider first-order methods with 8 times longer adaptation during meta-training than GradNCP. This observation is consistent with prior works [11], which have also shown that first-order meta-learning schemes tend to struggle with NF learning tasks. Given this, we believe that efficient second-order meta-learning techniques such as GradNCP will be a promising direction in this field.
D.6 Context Set Selection for Bootstrapping
Table 9: Comparison of context set choice for generating the target model on SIREN meta-learned with CelebA (178×178). We consider no target correction (None), random sampling (Random), gradient norm-based pruning (Norm-based), and the full context (Full) when adapting the target.
Context set	PSNR (↑)	SSIM (↑)	LPIPS (↓)
None	37.49	0.952	0.017
Random	37.56	0.951	0.016
Norm-based	38.19	0.958	0.013
Full (Ours)	38.90	0.967	0.009
We performed an experiment to investigate whether the gain of the bootstrapped correction mainly comes from the information recovery (by using the full context set) or the longer horizon effect. To this end, we examined various selection schemes for adapting the full context target, including random context (online), gradient norm-based pruned context, and full context set. The results, presented in Table 9, indicate that adaptation using the full context set is indeed effective, and the improvement is attributed to the recovery of information by the target model. Note that even random sampling and gradient norm-based pruning also improve the performance as they additionally provide more samples but less so than the full context set.
D.7 Cross-domain Adaptation Experiments
Table 10: Cross-domain reconstruction performance (PSNR; dB) of meta-learned SIREN under UCF-101  $(128\times 128\times 16)$  dataset. We adapt the network to a different dataset and modalities.
Modality	Dataset	Method	PSNR (↑)
Image	CelebA	Learnit / MAML [61]	27.74
(128×128)	GradNCP (Ours)	28.57
Imagenette	Learnit / MAML [61]	25.18
(128×128)	GradNCP (Ours)	26.32
Video	Kinetics-400	Learnit / MAML [61]	26.42
(128×128×16)	GradNCP (Ours)	27.41
We also consider the cross-domain adaptation scenario: we adapt the meta-learned model on different datasets or even different modalities from the meta-training. In particular, we train our method on UCF-101 and adapt to two different image datasets (CelebA and Imagenette) and one video dataset (Kinetics-400) [28]. Table 10 summarizes the results: GradNCP significantly improves the performance over the baseline, indicating GradNCP has learned a transferable initialization from the diverse motion of UCF-101.
D.8 Additional Comparison with TransINR under Test-time Adaptation
Table 11: Comparison with TransINR under same test-time adaptation (TTA) steps on SIREN metalearned with CelebA  $(178\times 178)$  dataset. We further adapt the same adaptation steps (as GradNCP) from the predicted network from TransINR.
Method	PSNR (↑)	SSIM (↑)	LPIPS (↓)
TransINR [6]	32.37	0.913	0.068
TransINR [6] + TTA	34.12	0.932	0.046
GradNCP (Ours)	40.60	0.976	0.005
To further verify the superiority of GradNCP, we additionally compare with the test-time adaptation performance of TransINR. Here, we use SGD with the same adaptation steps (as GradNCP) to further optimize the predicted NF by TransINR. As shown in Table 11, GradNCP significantly shows better results even under this scenario. Still, note that such a comparison is not fair for GradNCP (in terms of # of parameters), as TransINR additionally uses a large Transformer encoder over GradNCP.
D.9 Effectiveness of Using Full Context Set for Meta-testing
Figure 11: Comparison of test reconstruction performance between the utilization of full context set (Full Context) and gradient norm-based pruned context set (Pruned Context) during meta-test. The experiment is conducted over a meta-learn SIREN on CelebA  $(178\times 178)$  dataset with GradNCP.
To verify that the utilization of a full context set for meta-testing is truly effective, we compare the adaptation performance when using full and the gradient norm-based pruned context set. Here, we only apply the gradient scaling when using the full context set, as pruned context set gradient norm mismatch does not occur between the meta-training and testing. As shown in Figure 11, using the full context set for the meta-test time is indeed effective and shows consistent improvement over the pruned context set. Note that using pruned context set is also quite effective as it shows better adaptation performance than Learnit on a long adaptation horizon.
D.10 GradNCP for General Meta-Learning Setup
Table 12: Many-shot in-domain adaptation accuracy (%) on mini-ImageNet datasets. We train Conv4 on 5-way and 10-way classification by using the same memory constraint and use the same number of inner-adaptation steps during meta-testing for a fair comparison. Reported results are averaged over three trials, subscripts denote the standard deviation, and bold denotes the best result.
Method	5-way 100-shot	10-way 50-shot
MAML [15]	66.03±0.82	48.95±0.52
GradNCP (Ours)	73.45±0.83	55.71±0.49
While we primarily focused on resolving the memory issue of meta-learning NFs due to their unique aspects different from common meta-learning setups (i.e., training with extremely large context sets which suffers from severe memory burden), we show that GradNCP can be used for general meta-learning setups that uses second-order gradient-based meta-learning. To this end, we consider a many-shot classification (e.g., 100-shot) scenario on mini-ImageNet to verify the efficacy of GradNCP. As shown in Table 12, GradNCP significantly outperforms MAML under the given memory constraint, e.g.,  $66.03 \rightarrow 73.45\%$ . Based on this, we believe exploring GradNCP for general meta-learning setups also has great potential for application that requires second-order gradient-based meta-learning, e.g., learning automatic reweighting [46, 24].
D.11 More Qualitative Comparison with Baselines on High-resolution Signals
Figure 12: Qualitative comparison between GradNCP and baselines on AFHQ  $(512\times 512)$
Figure 13: Qualitative comparison between GradNCP and baselines on CelebA-HQ (1024 × 1024). We compare GradNCP with Learnit as TransINR suffers from the out-of-memory issue.
(a) UCF-101  $(128\times 128\times 16)$  on SIREN
Figure 14: Qualitative comparison between GradNCP and Learnit on UCF-101 dataset.
E More Visualizations of Selected Context Points via GradNCP
(a) CelebA-HQ  $(1024\times 1024)$
(b) AFHQ  $(512\times 512)$
(c) Text  $(178\times 178)$
(d) ImageNet  $(256\times 256)$
Figure 15: Visualization of selected context points (top), residuals relative to the original signal (middle), and the reconstruction (bottom) of GradNCP trained on (a) CelebA-HQ, (b) AFHQ, (c) Text, and (4) ImageNet. Selected coordinates are highlighted in red (top) where our selection scheme focuses on only  $25\%$  of available points at each of the  $k$  adaptation steps. GradNCP first focuses on modeling global structure while subsequently learning high-frequency details.