SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. De Freitas. Learning to learn by gradient descent by gradient descent. Advances in Neural Information Processing Systems, 2016.
[2] M. Bauer, E. Dupont, A. Brock, D. Rosenbaum, J. Schwarz, and H. Kim. Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv:2302.03130, 2023.
[3] J. Bronskill, D. Massiceti, M. Patacchiola, K. Hofmann, S. Nowozin, and R. Turner. Memory efficient meta-learning with large images. In Advances in Neural Information Processing Systems, 2021.
[4] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.
[5] H. Chen, B. He, H. Wang, Y. Ren, S. N. Lim, and A. Shrivastava. NeRV: Neural representations for videos. In Advances in Neural Information Processing Systems, 2021.
[6] Y. Chen and X. Wang. Transformers as meta-learners for implicit neural representations. In European Conference on Computer Vision, 2022.
[7] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha. StarGAN v2: Diverse image synthesis for multiple domains. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[8] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, 2009.
[9] Y. Du, M. K. Collins, , B. J. Tenenbaum, and V. Sitzmann. Learning signal-agnostic manifolds of neural fields. In Advances in Neural Information Processing Systems, 2021.
[10] E. Dupont, A. Golinski, M. Alizadeh, Y. W. Teh, and A. Doucet. Coin: Compression with implicit neural representations. In ICLR Neural Compression: From Information Theory to Applications Workshop, 2021.
[11] E. Dupont, H. Kim, S. Eslami, D. Rezende, and D. Rosenbaum. From data to functa: Your data point is a function and you should treat it like one. In International Conference on Machine Learning, 2022.
[12] E. Dupont, H. Loya, M. Alizadeh, A. Golinski, Y. W. Teh, and A. Doucet. Coin++: Data agnostic neural compression. Transactions on Machine Learning Research, 2022.
[13] Z. A. S. Emam, H.-M. Chu, P.-Y. Chiang, W. Czaja, R. Leapman, M. Goldblum, and T. Goldstein. Active learning at the imagenet scale. arXiv preprint arXiv:2111.12880, 2021.
[14] V. Feldman and C. Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In Advances in Neural Information Processing Systems, 2020.
[15] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, 2017.
[16] S. Flennerhag, P. G. Moreno, N. D. Lawrence, and A. Damianou. Transferring knowledge across learning processes. arXiv preprint arXiv:1812.01054, 2018.
[56] J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
[57] K. Soomro, A. R. Zamir, and M. Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
[58] B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Advances in Neural Information Processing Systems, 2022.
[59] J. Tack, J. Park, H. Lee, J. Lee, and J. Shin. Meta-learning with self-improving momentum target. In Advances in Neural Information Processing Systems, 2022.
[60] M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems, 2020.
[61] M. Tancik, B. Mildenhall, T. Wang, D. Schmidt, P. P. Srinivasan, J. T. Barron, and R. Ng. Learned initializations for optimizing coordinate-based neural representations. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[62] M. K. Titsias, J. Schwarz, A. G. d. G. Matthews, R. Pascanu, and Y. W. Teh. Functional regularisation for continual learning with gaussian processes. In International Conference on Learning Representations, 2020.
[63] M. Toneva, A. Sordoni, R. T. d. Combes, A. Trischler, Y. Bengio, and G. J. Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019.
[64] T. Wang, J.-Y. Zhu, A. Torralba, and A. A. Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018.
[65] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing (TIP), 2004.
[66] Y. Wu, M. Ren, R. Liao, and R. Grosse. Understanding short-horizon bias in stochastic meta-optimization. In International Conference on Learning Representations, 2018.
[67] S. Xie, C. Sun, J. Huang, Z. Tu, and K. Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In European Conference on Computer Vision, 2018.
[68] S. Yu, J. Tack, S. Mo, H. Kim, J. Kim, J.-W. Ha, and J. Shin. Generating videos with dynamics-aware implicit generative adversarial networks. In International Conference on Learning Representations, 2022.
[69] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[70] L. Zintgraf, K. Shiarli, V. Kurin, K. Hofmann, and S. Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning, 2019.
In this section, we describe the experimental details of Section 3, including GradNCP and the baselines. We also provide the implementation of GradNCP at https://github.com/jihoontack/GradNCP.
CelebA. CelebA is a fine-grained dataset that consists of the face image of celebrities [36]. The dataset comprises 202,599 images, where we use 162K for training, 20K for validation, and 20K for testing. We resize the image into 178 and then center cropped of 178, which ends with a resolution of  $178 \times 178$ . We pre-process pixel coordinates into  $[-1,1]^2$  and signal values ranging from 0 to 1.
Imagenette. Imagenette is a 10-class subset of the ImageNet [8] dataset, which is comprised of 9,000 training and 4,000 test images [23]. We resize the image into 178 and then apply a center crop of 178, which ends with a resolution of  $178 \times 178$ . We pre-process pixel coordinates into  $[-1,1]^2$  and signal values ranging from 0 to 1.
Text. Text dataset consists of text image with a resolution of  $178 \times 178$  [61]. We pre-process the pixel coordinates into  $[-1, 1]^2$  and signal values ranging from 0 to 1.
ImageNet. ImageNet dataset is a large-scale natural image dataset consisting of 1,000 classes [8]. We use 1,281K images for training and evaluated the methods on 50K test images (known as the validation set of ImageNet). For image pre-processing, we resize the image into 256, then center cropped the image to get  $256 \times 256$  resolution image. For coordinate pre-processing, we use pixel coordinates of  $[-1,1]^2$  and signal values ranging from 0 to 1.
CelebA-HQ. CelebA-HQ is a high-resolution fine-grained dataset, which includes images of celebrities [26]. We divided the dataset into 27,000 training and 3,000 test samples and pre-processed the pixel coordinates into  $[-1,1]^2$  and signal values ranging from 0 to 1. We consider two resolutions, i.e.,  $512 \times 512$  and  $1024 \times 1024$ .
AFHQ. AFHQ is a high-resolution fine-grained dataset, which includes animal faces consisting of 15,000 images at  $512 \times 512$  resolution [7]. We divided the dataset into 14,336 training and 1,467 testing points and pre-processed the pixel coordinates into  $[-1,1]^2$  and signal values from 0 to 1.
UCF-101. UCF-101 is a video dataset comprising 13,320 videos (9,357 training and 3,963 test videos) with a resolution of  $320 \times 240$ , where the action classification consists of 101 classes [57]. Each video clip is center-cropped to  $240 \times 240$  and then resized into  $128 \times 128$  and  $256 \times 256$  with a video clip length of 16 and 32, respectively. We pre-process the pixel coordinates into  $[-1,1]^3$  and signal values ranging from 0 to 1.
Kinetics-400. For the cross-domain adaptation purpose in Appendix D.7, we use the mini-Kinetics200 dataset [67], which consists of 200 categories with the most training samples from the Kinetics400 dataset [28]. We center-crop the videos to the same height and width and then resize them into  $128 \times 128$  resolution, and use the frame of 16 clips. We pre-process the pixel coordinates into  $[-1, 1]^3$  and signal values ranging from 0 to 1.
ERA5. ERA5 is a dataset comprised of temperature observations on a global grid of equally spaced latitudes and longitudes [22]. By following [11], we use the grid resolution of  $181 \times 360$  by resizing the grid. We interpret each time step as an independent signal, and the dataset comprises 9,676 training points and 2,420 test points. For the input pre-processing, given latitudes  $\rho$  and longitudes  $\varphi$  are transformed into 3D Cartesian coordinates  $\mathbf{c} = (\cos \rho \cos \varphi, \cos \rho \sin \varphi, \sin \rho)$  where latitudes  $\rho$  are equally spaced between  $-\frac{\pi}{2}$  and  $\frac{\pi}{2}$  and longitudes  $\varphi$  are equally spaced between 0 and  $\frac{2\pi(n-1)}{n}$  where  $n$  the number of distinct values of longitude (360).
LibriSpeech. LibriSpeech is an English speech recording collection at a  $16\mathrm{kHz}$  sampling rate [42]. By following Dupont et al. [12], we use the train-clean-100 split for training and test-clean split for testing, i.e., 28,539 training and 2,620 test samples. For the main experiments, we use the first 1 second and 3 seconds of each example (1 second contains 16,000 coordinates). For the pre-processing, we scale the coordinates into  $[-50,50]$ .
A.2 Training and Evaluation Details
Network architecture details. For the main experiment, we mainly use SIREN, a multi-layer perception (MLP) with sinusoidal activation functions [54], i.e.,  $\mathbf{x} \mapsto \sin(\omega_0(W\mathbf{x} + b))$  where  $W, b$  are weight and biases of the MLP layer and  $\omega_0$  is the fixed hyperparameter. For image, audio, and manifold datasets, we use SIREN with 5 layers with 256 hidden dimensions, and for video, we use 7 layers with the same hidden dimension. We used  $\omega_0 = 50$  for the manifold dataset and use  $\omega_0 = 30$  for the rest. We additionally consider NeRV [5] for the video dataset. For the UCF-101 dataset of  $128 \times 128 \times 16$ , we use 4 NeRV blocks, and for  $256 \times 256 \times 32$ , we use 5 NeRV blocks. Finally, we consider the Fourier feature network (FFN) in Appendix D.3, where we use the same network size as SIREN and the use Fourier feature scale  $\sigma = 10$ .
Training details. For all dataset, we use Adam optimizer [32] for the outer loop. We use the outer step of 150,000, except for learning the ImageNet dataset where we use 500,000 steps. For SIREN, we use the outer learning rate of  $\beta = 3.0 \times 10^{-6}$  for Librispeech and use  $\beta = 1.0 \times 10^{-5}$  for the rest. For NeRV, we use the outer learning rate of  $\beta = 1.0 \times 10^{-4}$ . As for the inner loop learning rate, we use  $\alpha = 1.0 \times 10^{-2}$ , and  $\alpha = 1.0 \times 10^{-1}$  for SIREN and NeRV, respectively. For inner step number  $K$ , GradNCP is trained on a longer horizon than Learnit by multiplying  $1 / \gamma$  (which uses the same memory usage). For Learnit, we mainly use  $K = 4$  for the main table, where we use  $K = 1$  for CelebA-HQ (1024  $\times$  1024),  $K = 5$  on UCF-101 (256  $\times$  256  $\times$  32) on NeRV, and  $K = 20$  on UCF-101 (128  $\times$  128  $\times$  16) on NeRV. We use the same batch size for Learnit and GradNCP to fairly use the memory where the size was selected differently across the dataset (e.g., under the given GPU memory budget).
Hyperparameter details for GradNCP. We find that the hyperparameter introduced by GradNCP is not sensitive across datasets and architectures. For the context set selection ratio  $\gamma$ , i.e., the ratio of retaining coordinates, we use 0.25 for most of the dataset except for Librispeech and ERA5, where we used 0.5 (as we do not need to prune the context much for these low-resolution signals), and 0.5, 0.2 when training NeRV on UCF-101 on 128 and 256 resolution, respectively. We found that for most of the datasets, the performance does not significantly decrease until  $\gamma = 0.2$  while significantly reducing the memory, i.e., about 5 times. For bootstrap target correction hyperparameters, we used  $L = 5$ , and  $\lambda = 100$ , where we believe tuning these hyperparameters will indeed improve the performance much more (we did not tune extensively).
Evaluation details. For the evaluation, to fairly compare with the baseline, we use the same test-time adaptation step for Learnit and GradNCP (e.g., for CelebA experiments we use  $K_{\mathrm{test}} = 16$ ) which is the same step number that is used by GradNCP on meta-training. We additionally compare the results of test-time adaptation of TransINR in Section D.8.
NeRF details. By following prior works [61, 6], we use the simplified NeRF model [40] which uses a single network rather two networks (coarse and fine), and do not use the view direction for the network. We use a MLP with 6 layers with 256 hidden dimension with ReLU activation. We follow the same training detail of Tancik et al. [61], except for inner loop steps where we use 16 steps (rather than 100 steps) and first random sample 512 rays then select the important 128 rays by using  $\gamma = 0.25$  (same ray size as the prior works).
Resource details. For the main development, we mainly use Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz and a single RTX 3090 24GB GPU, except for high-resolution signals including CelebA-HQ of  $1024 \times 1024$  and UCF-101 of  $256 \times 256 \times 32$ , where we use AMD EPYC 7542 32 Core Processor and a single NVIDIA A100 SXM4 40GB.
A.3 Baseline Details
In this section, we explain the meta-learning baselines we used for evaluating GradNCP. We note that we additionally compare with memory-efficient meta-learning schemes in Appendix D.5.
- Learnit [61] mainly uses the second-order gradient version of MAML [15] for learning NFs.
- TransINR [6] utilizes Transformer as a meta-learner to predict the NF parameter with a given context set, and additionally proposed a parameter-efficient NF architecture specialized for MLP.
- IPC [29] denotes instant pattern composer which is a modulation technique for meta-learning NFs. This modulation uses a low-rank parametrization of NF weights where the authors used TransINR as a meta-learning approach to train these modulations.
B Derivation of Our Data Importance Score
In this section, we prove our sample importance score (shown in the following equation) well approximates the updated loss value when adapting the network using the given sample.
$$
R _ {k} ^ {\operatorname {G r a d N C P}} (\mathbf {x}, \mathbf {y}) := \left\| \left(\mathbf {y} - f _ {\theta_ {k}} (\mathbf {x})\right) \left[ \phi_ {\theta_ {k} ^ {\text {b a s e}}} (\mathbf {x}), \mathbf {1} \right] ^ {\top} \right\|. \tag {5}
$$
We first review the derivation in the main text. Let  $\theta_{k}^{\mathrm{base}}$  be the set of parameters excluding the final layer and  $g_{k} = [0, \nabla_{W_{k}}\ell, \nabla_{b_{k}}\ell]$  a padded gradient involving zeros for all but the last layer's weights  $W_{k}$  and biases  $b_{k}$ , we start from the SPG metric:
$$
\begin{array}{l} \ell \left(\theta_ {k}; \{(\mathbf {x}, \mathbf {y}) \}\right) - \ell \left(\theta_ {k} ^ {\prime}; \{(\mathbf {x}, \mathbf {y}) \}\right) \\ \approx \ell \left(\theta_ {k}; \left\{\left(\mathbf {x}, \mathbf {y}\right) \right\}\right) - \ell \left(\theta_ {k} - \alpha g _ {k}; \left\{\left(\mathbf {x}, \mathbf {y}\right) \right\}\right) \quad (\text {L a s t}) \\ \approx \ell \left(\theta_ {k}; \left\{\left(\mathbf {x}, \mathbf {y}\right) \right\}\right) - \left(\ell \left(\theta_ {k}; \left\{\left(\mathbf {x}, \mathbf {y}\right) \right\}\right) - \alpha g _ {k} ^ {\top} \nabla_ {\theta_ {k}} \ell \left(\theta_ {k}; \left\{\left(\mathbf {x}, \mathbf {y}\right) \right\}\right)\right) \quad (T a y l o r \\ = \alpha g _ {k} ^ {\top} \nabla_ {\theta_ {k}} \ell \left(\theta_ {k}; \left\{\left(\mathbf {x}, \mathbf {y}\right) \right\}\right) = \alpha \| g _ {k} \| _ {2} ^ {2}. \\ \end{array}
$$
where  $\alpha > 0$  is the step size of the optimization. We remove  $\alpha$ , as the sample importance ranking does not change as  $\alpha$  is a positive value. Since NF training considers MSE loss for  $\ell$ , the gradient of the last layer weight and bias are as:
$$
\begin{array}{l} \left. \nabla_ {W _ {k}} \ell \left(\theta_ {k}; \left\{\left(\mathbf {x}, \mathbf {y}\right) \right\}\right) = \nabla_ {W _ {k}} \left\| \left(\mathbf {y} - \left(W _ {k} \phi_ {\theta_ {k} ^ {\text {b a s e}}} (\mathbf {x}) + b _ {k}\right)\right) \right\| _ {2} ^ {2} \right. (6) \\ = - 2 \left(\mathbf {y} - \left(W _ {k} \phi_ {\theta_ {k} ^ {\text {b a s e}}} (\mathbf {x}) + b _ {k}\right)\right) \phi_ {\theta_ {k} ^ {\text {b a s e}}} (\mathbf {x}) ^ {\top} (7) \\ = - 2 \left(\mathbf {y} - f _ {\theta_ {k}} (\mathbf {x})\right) \phi_ {\theta_ {k} ^ {\text {b a s e}}} (\mathbf {x}) ^ {\top} (8) \\ \end{array}
$$
$$
\begin{array}{l} \left. \nabla_ {b _ {k}} \ell \left(\theta_ {k}; \left\{(\mathbf {x}, \mathbf {y}) \right\}\right) = \nabla_ {b _ {k}} \left\| \left(\mathbf {y} - \left(W _ {k} \phi_ {\theta_ {k} ^ {\text {b a s e}}} (\mathbf {x}) + b _ {k}\right)\right) \right\| _ {2} ^ {2} \right. (9) \\ = - 2 \left(\mathbf {y} - \left(W _ {k} \phi_ {\theta_ {k} ^ {\text {b a s e}}} (\mathbf {x}) + b _ {k}\right)\right) (10) \\ = - 2 \left(\mathbf {y} - f _ {\theta_ {k}} (\mathbf {x})\right) (11) \\ \end{array}
$$
Given the gradient of the last layer in Eq. (8) and (11), the final score is derived as follows:
$$
\begin{array}{l} R _ {k} ^ {\operatorname {G r a d N C P}} (\mathbf {x}, \mathbf {y}) := \| g _ {k} \| _ {2} ^ {2} = \left\| \left[ \nabla_ {W _ {k}} \ell \left(\theta_ {k}; \{(\mathbf {x}, \mathbf {y}) \}\right) \nabla_ {b _ {k}} \ell \left(\theta_ {k}; \{(x, y) \}\right) \right] \right\| _ {2} ^ {2} (12) \\ = 2 \left\| \left(\mathbf {y} - f _ {\theta_ {k}} (\mathbf {x})\right) \left[ \phi_ {\theta_ {k} ^ {\text {b a s e}}} (\mathbf {x}), \mathbf {1} \right] ^ {\mathsf {T}} \right\|. (13) \\ \end{array}
$$
Since the order (or the ranking) of the gradient norm value does not change by multiplying the positive value, we simply removed 2 from the final score. For NF with non-linear final layer (e.g., NeRV), one can easily derive the following GradNCP criteria:  $\left\| \left(\mathbf{y} - f_{\theta_k}(\mathbf{x})\right)\sigma '\left(W_k\phi_{\theta_k^{\mathrm{base}}}\left(\mathbf{x}\right) + b_k\right)\left[\phi_{\theta_k^{\mathrm{base}}}\left(\mathbf{x}\right),\mathbf{1}\right]^{\mathsf{T}}\right\|$  where the output of NF is  $f_{\theta_k}(\mathbf{x}) = \sigma \bigl (W_k\phi_{\theta_k^{\mathrm{base}}}(\mathbf{x}) + b_k\bigr)$ ,  $\sigma (\cdot)$  is the element-wise non-linear operator (e.g., Sigmoid) and  $\sigma^\prime$  is the derivative of  $\sigma$ .
Neural fields (NFs). NFs, also referred to as implicit neural representations (INRs), have emerged as a new paradigm for representing complex, continuous signals [43, 60]. This approach encodes the signal with a neural network, typically a multi-layer perceptron (MLP) in conjunction with sinusoidal activations [54] or positional encoding [40]. They have become popular as their number of parameters do not strictly scale with the resolution of the signal [39, 52], easing modeling of multi-modal signals [9, 37] and showing potential for new approaches to prominent applications and downstream tasks, including data compression [10, 49], classification [11, 2], and generative modeling [55, 68]. However, fitting NFs is quite costly, especially for high-resolution signals [30]. To tackle this issue, we develop an efficient meta-learning framework for large-scale NF training which significantly reduce the memory usages of meta-learning.
Efficient meta-learning. There have been several works on developing (memory) efficient algorithms in the field of meta-learning. Typically, such algorithms have been explored in amortization-based (or encoder-based) schemes, e.g., Prototypical Networks [56, 3] and Neural Processes [20, 19, 18]. Unlike optimization-based meta-learning, however, these methods are somewhat limited in applicability to diverse modalities and NF architectures (requiring modality and model-specific design).
In the optimization-based regime, several memory efficient schemes have been introduced, including first-order MAML [15], Reptile [41], Implicit MAML [45] and continual trajectory shifting [51], that do not use the second-order gradient adaptation. However, recent works have shown that first-order optimization-based schemes can underperform for NFs [11, 12]. We also observed a similar result as summarized in Table 8 of Appendix D.5. Recently, network sparsity has been combined with meta-learning NFs [34, 48], which may help reduce computation at inference time but still requires some memory usage when meta-learning and more importantly can reduce the network expressive power. In this paper, we focus on developing a memory-efficient meta-learning framework that is built upon second-order gradient-based schemes for effective NF learning.
Sparse data selection. GradNCP is related to areas of machine learning which focus on identifying subsets of a dataset. For instance, data pruning focuses on designing data importance metrics to reduce the dataset without comprising the performance [63, 14, 27, 58], memory-based techniques of continual learning use subsets of past tasks to prevent catastrophic forgetting [62, 47], dataset distillation proposes the use of optimized few synthetic data for the same purpose [64], and active learning study way to identify and label data points to facilitate efficient learning progress during subsequent online updates [50, 13]. In this paper, we develop an efficient online context pruning scheme for meta-learning NFs by drawing inspiration from a recent work of curriculum learning to estimate the sample importance [21].
D.1 Gradient Norm Correlation
(a) Random init / Adapt with the full context set
Figure 9: Correlation between the gradient norm of the entire network and the last layer network. We measure the Pearson correlation coefficient between the entire gradient norm and the last layer gradient norm on individual signals of the CelebA test dataset (i.e., correlation computed on  $178 \times 178$  elements), then report the average correlation across signals. Here the pruned set indicates the context pruning with GradNCP.
(b) Random init / Adapt with the pruned set
(c) Meta-learned init / Adapt with the full context set (d) Meta-learned init / Adapt with the pruned set
In this section, we compare the correlation between the gradient norms of the entire network and the last layer to verify that our last layer gradient norm is a reasonable approximation. We first note that the entire network gradient norm is an approximation for the SPG score (i.e., by using the first-order Taylor series on SPG) without the last layer gradient approximation, hence, serves as a reasonable measure for the sample importance. It is worth noting that, while the entire gradient norm is a reasonable score, this is still computationally too heavy to use online, as it requires computing the gradient w.r.t to the entire network parameter with  $O(M)$  time complexity where our last layer gradient norm only needs a forward pass of  $O(M)$  (remark that  $M$  is the context set size).