SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhijie Deng $^{1*}$ , Peng Cui $^{2*}$ , Jun Zhu $^{2\dagger}$
$^{1}$ Qing Yuan Research Institute, Shanghai Jiao Tong University
$^{2}$ Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center,
Tsinghua-Bosch Joint ML Center, THBI Lab, Tsinghua University, Beijing, 100084 China  
zhijied@sjtu.edu.cn, xpeng.cui@gmail.com, dcszj@tsinghua.edu.cn
Mislabeled, duplicated, or biased data in real-world scenarios can lead to prolonged training and even hinder model convergence. Traditional solutions prioritizing easy or hard samples lack the flexibility to handle such a variety simultaneously. Recent work has proposed a more reasonable data selection principle by examining the data's impact on the model's generalization loss. However, its practical adoption relies on less principled approximations and additional holdout data. This work solves these problems by leveraging a lightweight Bayesian treatment and incorporating off-the-shelf zero-shot predictors built on large-scale pre-trained models. The resulting algorithm is efficient and easy to implement. We perform extensive empirical studies on challenging benchmarks with considerable data noise and imbalance in the online batch selection scenario, and observe superior training efficiency over competitive baselines. Notably, on the challenging WebVision benchmark, our method can achieve similar predictive performance with significantly fewer training iterations than leading data selection methods.
The past year has witnessed significant breakthroughs in deep learning research and applications, with Stable Diffusion [38], ChatGPT [32], and SAM [22] as representative examples. Practitioners have realized that the quality of data used to fuel AI systems is critical in unlocking their full potential. Unfortunately, real-world scenarios often present mistrabeled, duplicated, or biased data. As a result, it is paramount to develop methods that can prioritize valuable training data to enable more efficient model training and even improved model convergence.
Data selection for accelerating the training of deep models is gaining increasing interest. Some studies, such as curriculum learning, advocate prioritizing easy samples in the early training stages [1], but these samples quickly become redundant once they have been learned, making continued training on them a waste of time. On the other hand, online batch selection methods [26, 19, 17] prioritize hard samples with high training loss or gradient norm to avoid duplicate training. Nevertheless, in practice, the hardness of samples often arises from pathologies such as improper annotations, inherent ambiguity, or unusual patterns, rendering it problematic to prioritize such samples [31].
The reducible hold-out loss selection (RHO-LOSS) approach [31] addresses these issues by quantifying the usefulness of a sample based on its marginal influence on the model's generalization loss, forming a theoretically grounded and universal objective for data selection. However, the estimation of this objective is non-trivial, and RHO-LOSS has to rely on less principled approximations for
practical adoption. Besides, RHO-LOSS hinges on a considerable number of holdout data to train an auxiliary validation model, which can be costly and should be performed repeatedly for new tasks.
This paper aims to bridge this gap to make the generalization loss-based data selection principle more accessible to a broader audience. We establish a more reasonable approximation of the original objective than RHO-LOSS while eliminating the need for holdout data. To achieve this, we derive a lower bound of the objective to separate the posterior predictive defined on the training data from that defined on the holdout data. Afterward, we propose to use off-the-shelf zero-shot predictors, built upon large-scale pre-trained models [35, 41], as a proxy for the latter, since these models often contain generally applicable information for solving specific downstream tasks.
We maintain a Bayesian treatment of the training model to ensure an accurate estimation of the original objective. Bearing in mind that our original goal is to accelerate the training of a deterministic model, we adopt the simple and effective Laplace approximation [29, 37, 8] for Bayesian inference. It effortlessly converts point-estimate parameters to a Gaussian posterior. We further introduce Kronecker-factored (KFAC) [30] and last-layer [23] approximations to accelerate the processing of modern neural networks (NNs), resulting in an efficient and easy-to-implement algorithm.
We conduct comprehensive empirical studies on various benchmarks to evaluate the effectiveness of our method. The experiments on standard image recognition tasks demonstrate that our approach can outperform various baselines in aspects of training speed and final accuracy. This conclusion also holds for learning with label noise and class imbalance. On the challenging WebVision [25] dataset, which contains plenty of noisy labels and ambiguous images collected from the internet, our method significantly reduces the number of training steps needed to reach the target accuracy and achieves up to  $19\%$  higher final accuracy than prior arts (see Table 3). These results highlight the practical value of our approach. In addition, we conduct informative ablation studies to gain a better understanding of the behavior of our method.
In this section, we briefly revisit the concept of online batch selection [26] and the data selection principle defined with the model's generalization loss in [31].
Consider training a  $\theta$ -parameterized deep model  $f_{\theta}:\mathcal{X}\to \mathbb{R}^{k}$  on a dataset  $\mathcal{D} = \{(x_i,y_i)\}_{i = 1}^n$  using stochastic gradient descent (SGD). The model likelihood is formally  $p(y|x,\theta) = p(y|f_{\theta}(x))$ . At each training step  $t$ , we can access a data batch  $B_{t}$  of size  $n_B$  from  $\mathcal{D}$ . In online batch selection, we need to compute statistics of the samples in  $B_{t}$  and select only those that meet certain requirements to update the model. This filtering process aims to remove samples that are deemed less valuable.
Let  $\mathcal{D}_{t-1}$  denote the set of data observed before  $t$  and  $(x', y')$  a sample from  $B_t$ . If we accept selecting  $(x', y')$ , the updated predictive distribution, in a Bayesian view, will be  $p(y|x, \mathcal{D}_{t-1}, \{(x', y')\}) = \mathbb{E}_{p(\theta | \mathcal{D}_{t-1}, \{(x', y')\})} p(y|x, \theta)$ . The question arises how to estimate the quality of this distribution to determine which sample  $(x', y') \in B_t$  should be selected. A natural tactic is to compare this distribution to the ground truth data-generating distribution  $\dot{p}(x, y)$ . That said, the typical KL divergence can be introduced, and our goal becomes solving the following problem:
$$
\min  _ {(x ^ {\prime}, y ^ {\prime}) \in B _ {t}} \mathbb {E} _ {\dot {p} (x)} \left(D _ {\mathrm {K L}} \left[ \dot {p} (y | x) \| p (y | x, \mathcal {D} _ {t - 1}, \{(x ^ {\prime}, y ^ {\prime}) \}) \right]\right) = \text {c o n s t .} - \mathbb {E} _ {\dot {p} (x, y)} \left[ \log p (y | x, \mathcal {D} _ {t - 1}, \{(x ^ {\prime}, y ^ {\prime}) \}) \right], \tag {1}
$$
where const. denotes a constant agnostic to the object to optimize. By applying Monte Carlo (MC) estimation using extra holdout samples  $\mathcal{D}^{*} = \{(\tilde{x}_{i},\tilde{y}_{i})\}_{i = 1}^{m}$  from  $\dot{p} (x,y)$ , we arrive at the following optimization problem:
$$
\max  _ {\left(x ^ {\prime}, y ^ {\prime}\right) \in B _ {t}} \frac {1}{m} \sum_ {i = 1} ^ {m} \left[ \log p \left(\tilde {y} _ {i} \mid \tilde {x} _ {i}, \mathcal {D} _ {t - 1}, \left\{\left(x ^ {\prime}, y ^ {\prime}\right) \right\}\right) \right] \Longleftrightarrow \max  _ {\left(x ^ {\prime}, y ^ {\prime}\right) \in B _ {t}} \log p \left(\mathcal {D} ^ {*} \mid \mathcal {D} _ {t - 1}, \left\{\left(x ^ {\prime}, y ^ {\prime}\right) \right\}\right). \tag {2}
$$
This objective corresponds to the model's generalization loss instead of the fitness of training data.
By Bayes' rule, there is:
$$
p \left(\mathcal {D} ^ {*} \mid \mathcal {D} _ {t - 1}, \left\{\left(x ^ {\prime}, y ^ {\prime}\right) \right\}\right) = \frac {p \left(y ^ {\prime} \mid x ^ {\prime} , \mathcal {D} ^ {*} , \mathcal {D} _ {t - 1}\right) p \left(x ^ {\prime} , \mathcal {D} ^ {*} , \mathcal {D} _ {t - 1}\right)}{p \left(y ^ {\prime} \mid x ^ {\prime} , \mathcal {D} _ {t - 1}\right) p \left(x ^ {\prime} , \mathcal {D} _ {t - 1}\right)} = \frac {p \left(y ^ {\prime} \mid x ^ {\prime} , \mathcal {D} ^ {*} , \mathcal {D} _ {t - 1}\right)}{p \left(y ^ {\prime} \mid x ^ {\prime} , \mathcal {D} _ {t - 1}\right)} \cdot p \left(\mathcal {D} ^ {*} \mid \mathcal {D} _ {t - 1}, x ^ {\prime}\right), \tag {3}
$$
2We assume selecting one single sample per time for simplicity. Multi-sample selection is viable.
where the item  $p(\mathcal{D}^*|\mathcal{D}_{t-1}, x')$  actually equals to the constant  $p(\mathcal{D}^*|\mathcal{D}_{t-1})$  because  $x'$  alone cannot incur model update. Plugging this back into Equation (2), we arrive at the final objective for data selection:
$$
\max  _ {(x, y) \in B _ {t}} \log p (y | x, \mathcal {D} ^ {*}, \mathcal {D} _ {t - 1}) - \log p (y | x, \mathcal {D} _ {t - 1}), \tag {4}
$$
where we omit constants and abuse  $(x,y)$  to represent  $(x^{\prime},y^{\prime})$  hereinafter if there is no misleading.
Although the above selection principle is theoretically sound and universally applicable, estimating it accurately is challenging. In particular, it is difficult to estimate the posterior predictive distribution defined with the combination of training and holdout data in a computationally efficient manner. To address this issue, RHO-LOSS proposes approximating  $\log p(y|x,\mathcal{D}^*,\mathcal{D}_{t - 1})$  with  $\log p(y|x,\mathcal{D}^*)$  and approximating the posterior predictive with the point-estimate model' predictions [31]. However, these approximations compromise the method's theoretical groundness, and access to holdout data can often be infeasible in practice. Our approach overcomes these limitations by utilizing a lightweight Bayesian treatment and incorporating off-the-shelf zero-shot predictors built on large-scale pre-trained models, as detailed below.
In this section, we demonstrate that the data selection principle discussed earlier can be lower-bounded by more easily computable quantities. We then leverage these insights to develop a Bayesian data selection approach to accelerating the training of deterministic deep models.
3.1 The Lower Bound
As previously discussed, we need to estimate two log-probabilities,  $\log p(y|x,\mathcal{D}^*,\mathcal{D}_{t - 1})$  and  $\log p(y|x,\mathcal{D}_{t - 1})$ , for each sample in  $B_{t}$  to identify the most useful one and update the model accordingly. However, as mentioned, evaluating the former is challenging. To address this issue, we derive a lower bound that allows for a more feasible estimation.
We first unfold  $\log p(y|x,\mathcal{D}^*,\mathcal{D}_{t - 1})$  as the combination of a posterior  $p(\theta |\mathcal{D}^*,\mathcal{D}_{t - 1})$  and the model likelihood  $p(y|x,\theta)$ , detailed below (we defer the derivation to Appendix):
$$
\log p (y | x, \mathcal {D} ^ {*}, \mathcal {D} _ {t - 1}) = \log \int p (\mathcal {D} ^ {*} | \theta) p (\theta | \mathcal {D} _ {t - 1}) p (y | x, \theta) d \theta - \log p (\mathcal {D} ^ {*} | \mathcal {D} _ {t - 1}). \tag {5}
$$
Then, by Jensen's inequality, there is (derivation is deferred to Appendix)
$$
\log p (y | x, \mathcal {D} ^ {*}, \mathcal {D} _ {t - 1}) \geq \mathbb {E} _ {p (\theta | \mathcal {D} _ {t - 1})} \log p (y | x, \theta) + \mathbb {E} _ {p (\theta | \mathcal {D} _ {t - 1})} \log p (\mathcal {D} ^ {*} | \theta) - \log p (\mathcal {D} ^ {*} | \mathcal {D} _ {t - 1}). \tag {6}
$$
Notably, the last two terms are independent of  $(x,y)$  and can be excluded from the optimization.
We can similarly derive another lower bound:
$$
\begin{array}{l} \log p (y | x, \mathcal {D} ^ {*}, \mathcal {D} _ {t - 1}) = \log \int p (\mathcal {D} _ {t - 1} | \theta) p (\theta | \mathcal {D} ^ {*}) p (y | x, \theta) d \theta - \log p (\mathcal {D} _ {t - 1} | \mathcal {D} ^ {*}) \tag {7} \\ \geq \mathbb {E} _ {p (\theta | \mathcal {D} ^ {*})} \log p (y | x, \theta) + \mathbb {E} _ {p (\theta | \mathcal {D} ^ {*})} \log p (\mathcal {D} _ {t - 1} | \theta) - \log p (\mathcal {D} _ {t - 1} | \mathcal {D} ^ {*}), \\ \end{array}
$$
where the last two terms are also agnostic to  $(x,y)$  and hence can be omitted.
Combining Equation (6) and Equation (7), there is
$$
\log p \left(y | x, \mathcal {D} ^ {*}, \mathcal {D} _ {t - 1}\right) \geq \alpha \mathbb {E} _ {p \left(\theta \mid \mathcal {D} _ {t - 1}\right)} \log p (y | x, \theta) + (1 - \alpha) \mathbb {E} _ {p \left(\theta \mid \mathcal {D} ^ {*}\right)} \log p (y | x, \theta) + \text {c o n s t .}, \tag {8}
$$
where  $\alpha \in [0,1]$  represents a trade-off coefficient. This way, we disentangle the posterior associated with the training data  $\mathcal{D}_{t-1}$  from that associated with the holdout data  $\mathcal{D}^*$ .
Rewriting  $p(y|x, \mathcal{D}_{t-1})$  as  $\mathbb{E}_{p(\theta | \mathcal{D}_{t-1})} p(y|x, \theta)$ , we can subsequently convert Equation (4) to the following maximization problem:
$$
\max  _ {(x, y) \in B _ {t}} \alpha \mathbb {E} _ {p (\theta | \mathcal {D} _ {t - 1})} \log p (y | x, \theta) + (1 - \alpha) \mathbb {E} _ {p (\theta | \mathcal {D} ^ {*})} \log p (y | x, \theta) - \log \mathbb {E} _ {p (\theta | \mathcal {D} _ {t - 1})} p (y | x, \theta). \tag {9}
$$
The presented objective is intriguing for several reasons. Firstly, the first term and  $\alpha$  times the third term perform expectation and logarithm operations in reverse order, resulting in a quantity similar to
the uncertainty estimates in Bayesian deep learning [39] (nonetheless, our objective involves data labels). Additionally, the third term helps to prevent the selection of redundant samples. And, the second term prioritizes points with high semantic alignment with their annotations. These three forces are integrated adaptively to accelerate model training across various stages.
3.2 Zero-shot Predictor as the Validation Model
Collecting extra holdout data to train an auxiliary validation model can be costly and should be performed repeatedly for new tasks. To address this issue, we propose to use zero-shot predictors built upon large-scale pre-trained models [35, 41] as a proxy for the validation model, based on the observation that they usually exhibit promising transfer performance across a broad range of downstream applications.
Formally, we make the following approximation:
$$
\mathbb {E} _ {p \left(\theta \mid \mathcal {D} ^ {*}\right)} \log p (y \mid x, \theta) \approx \log p (y | \tilde {f} (x)), \tag {10}
$$
where  $\tilde{f}:\mathcal{X}\to \mathbb{R}^k$  denotes the zero-shot predictor used.
We can think of the pre-trained model as a universal validation model trained on an extensive dataset, leading to the Bayesian posterior collapsing to a point estimate. Although its training data may not precisely follow the data-generating distribution for the current task, they share fundamental patterns with the data in our problem, making the above approximation reasonable. Notably, as shown in Section 4, our trained model eventually performs much better than the zero-shot predictor.
3.3 Lightweight Bayesian Treatment of the Training Model
The first and third terms in Equation (9) raise the requirement of maintaining a Bayesian posterior  $p(\theta | \mathcal{D}_{t-1})$  during training. Due to the high nonlinearity of deep NNs, the analytical posterior is intractable. By convention, we can introduce an approximate posterior  $q(\theta | \mathcal{D}_{t-1})$  and tune it by approximate Bayesian inference methods like MCMC [42, 5, 45], variational inference [2, 15, 28, 44, 21], Laplace approximation [29, 37], etc. Recalling that, despite relying on a Bayesian treatment, our final goal is to accelerate and improve the training of a deterministic model, Laplace approximation is well suited to our setting—it can convert point-estimate parameters to a Gaussian posterior effortlessly, thus lifting the unnecessary burden of learning and maintaining a posterior explicitly. We clarify that, technically, other Bayesian inference methods are also compatible with our approach.
Although Laplace approximation is typically used for maximum a posteriori estimation, it can be effectively adapted to the online learning cases [36]. Specifically, consider an isotropic Gaussian prior with precision  $\tau_0$  over parameters  $\theta$ . Let  $\mathcal{D}_{t-1} \coloneqq b_1 \cup b_2 \cup \dots \cup b_{t-1}$  denote all selected training samples before time  $t$ . The parameters of our deterministic model at time  $t-1$ , dubbed as  $\theta_{t-1}$ , are likely to approach a mode of the true posterior  $p(\theta | \mathcal{D}_{t-1})$  in the presence of a proper weight decay regularizer in stochastic optimization [10]. The online Laplace approximation then deploys the following approximate posterior:
$$
q \left(\theta \mid \mathcal {D} _ {t - 1}\right) = \mathcal {N} \left(\theta_ {t - 1}, H _ {t - 1} ^ {- 1}\right), H _ {t - 1} = \tau_ {0} I + \sum_ {i = 1} ^ {t - 1} \left(\sum_ {x, y \in b _ {i}} \frac {\partial^ {2} [ - \log p (y \mid x , \theta) ]}{\partial \theta^ {2}} \Big | _ {\theta = \theta_ {i}}\right). \tag {11}
$$
The Gaussian covariance should be positive semi-definite, but as  $\theta_{t - 1}$  cannot be guaranteed to be exactly the posterior mode, we would better replace the Hessian matrix with the generalized Gauss-Newton (GGN) matrix to avoid ill-posed approximation, resulting in:
$$
q \left(\theta \mid \mathcal {D} _ {t - 1}\right) = \mathcal {N} \left(\theta_ {t - 1}, G _ {t - 1} ^ {- 1}\right), G _ {t - 1} = \tau_ {0} I + \sum_ {i = 1} ^ {t - 1} \left(\sum_ {x, y \in b _ {i}} J _ {\theta_ {i}} (x) ^ {\top} \Lambda_ {\theta_ {i}} (x, y) J _ {\theta_ {i}} (x)\right), \tag {12}
$$
where  $J_{\theta_i}(x)\coloneqq \nabla_\theta f_\theta (x)|_{\theta = \theta_i}$  and  $\Lambda_{\theta_i}(x,y)\coloneqq \nabla_f^2 [-\log p(y|f)]|_{f = f_{\theta_i}(x)}$
Practical acceleration. Modern neural networks usually contain millions of parameters, making the matrix  $G_{t-1}$  too large to fit into CPU/GPU memories. To address this, a common practice is to sparsify the matrix using diagonal or Kronecker-factored (KFAC) approximations [30]. This work
${}^{3}{b}_{i} \subset  {B}_{i}$  represents the set of selected samples at time step  $i$  .
Algorithm 1 Bayesian data selection to accelerate the training of deterministic deep models.
1: Input: Number of iterations  $T$ , dataset  $\mathcal{D}$ , prior precision  $\tau_0$ , number of effective data  $n_e$ , batch size  $n_B$ , number of selections  $n_b$ , zero-shot predictor  $\tilde{f}$ , deterministic model with parameters  $\theta$ .
2: Initialize  $\theta_0, A_0 \gets 0, G_0 \gets 0$ ;
3: for  $t$  in  $1, \ldots, T$  do
4: Draw a mini-batch  $B_{t}$  from  $\mathcal{D}$ ;
5:  $V_{t - 1}\leftarrow \sqrt{n_e} A_{t - 1} + \sqrt{\tau_0} I,U_{t - 1}\leftarrow \sqrt{n_e} G_{t - 1} + \sqrt{\tau_0} I;$
6: Estimate the objective in Equation (16) for every sample in  $B_{t}$  and select the top-  $n_b$  ones to form  $b_t$ ;
7: Perform back-propagation with  $\sum_{x,y\in b_t}\log p(y|f_{\theta_{t - 1}}(x));$
8: Apply weight decay regularization and do gradient ascent to obtain  $\theta_{t}$
9: Use the last-layer features and softmax gradients to update  $A_{t}$  and  $G_{t}$  with exponential moving average;
10: end for
prefers KFAC as it preserves the correlations between parameters within the same layers. Besides, a recent surprising finding is that we can even apply Laplace approximation to only the last layer of a deep NN, leaving the other parameters point-estimate, to conjoin efficiency and calibrated uncertainty estimates [23, 8]. As a result, we consider combining last-layer and KFAC approximations to reduce the burden caused by the Bayesian treatment, with the details presented below.
Let us decompose  $f_{\theta_i}$  as a feature extractor  $h_{\theta_i}$  and a linear layer with weights  $\theta_i^{(l)} \in \mathbb{R}^{d \times k}$ , i.e.,  $f_{\theta_i}(x) := h_{\theta_i}(x)^\top \theta_i^{(l)}$ . We adapt the GGN matrix associated with the last-layer weights  $\theta_i^{(l)}$  derived in [37, 23] to the following formula:
$$
G _ {t - 1} ^ {(l)} \approx V _ {t - 1} \otimes U _ {t - 1} := \left(\sqrt {\left| \mathcal {D} _ {t - 1} \right|} A _ {t - 1} + \sqrt {\tau_ {0}} I\right) \otimes \left(\sqrt {\left| \mathcal {D} _ {t - 1} \right|} G _ {t - 1} + \sqrt {\tau_ {0}} I\right), \tag {13}
$$
where  $\otimes$  denotes the Kronecker product and
$$
A _ {t - 1} := \frac {1}{\left| \mathcal {D} _ {t - 1} \right|} \sum_ {i = 1} ^ {t - 1} \sum_ {x, y \in b _ {i}} h _ {\theta_ {i}} (x) h _ {\theta_ {i}} (x) ^ {\top}, \tag {14}
$$
$$
G _ {t - 1} := \frac {1}{| \mathcal {D} _ {t - 1} |} \sum_ {i = 1} ^ {t - 1} \sum_ {x, y \in b _ {i}} [ \nabla_ {f} \log p (y | f) | _ {f = f _ {\theta_ {i}} (x)} ] [ \nabla_ {f} \log p (y | f) | _ {f = f _ {\theta_ {i}} (x)} ] ^ {\top}.
$$
Of note that the matrices  $A_{t - 1}\in \mathbb{R}^{d\times d}$  and  $G_{t - 1}\in \mathbb{R}^{k\times k}$  raise only minimal extra storage cost. The approximate posterior can be formulated as a matrix-variate Gaussian [12]:  $q(\theta^{(l)}|\mathcal{D}_{t - 1}) = \mathcal{MN}(\theta^{(l)}|\theta_{t - 1}^{(l)},U_{t - 1}^{-1},V_{t - 1}^{-1})$ . Given the linear nature, it is straightforward to get the distribution over the model output  $f_{x}$  for input  $x$  (derivation is deferred to Appendix):
$$
q \left(f _ {x} \mid \mathcal {D} _ {t - 1}\right) = \mathcal {N} \left(f _ {\theta_ {t - 1}} (x), \left(h _ {\theta_ {t - 1}} (x) ^ {\top} V _ {t - 1} ^ {- 1} h _ {\theta_ {t - 1}} (x)\right) U _ {t - 1} ^ {- 1}\right). \tag {15}
$$
The final data selection objective. With the above equation, the selection objective boils down to
$$
\max  _ {(x, y) \in B _ {t}} \alpha \left[ \frac {1}{S} \sum_ {s = 1} ^ {S} \log p \left(y \mid f _ {x} ^ {(s)}\right) \right] + (1 - \alpha) \log p (y | \tilde {f} (x)) - \log \left[ \frac {1}{S} \sum_ {s = 1} ^ {S} p \left(y \mid f _ {x} ^ {(s)}\right) \right], \tag {16}
$$
where  $f_{x}^{(s)} \sim q(f_{x}|\mathcal{D}_{t - 1}), s = 1, \dots, S$  are MC samples. Compared to the non-last-layer Laplace approximation, which involves sampling over the parameter space and performing MC integration, the last-layer one enjoys a much faster evaluation procedure. It also enables the use of a large  $S$ . Our method can also abandon the KFAC approximation when the linear head is small.
The algorithm. We present the training procedure in Algorithm 1. To avoid too large  $|\mathcal{D}_{t-1}|$  and hence too sharpened approximate posterior, we use a tunable hyper-parameter  $n_e$  to replace  $|\mathcal{D}_{t-1}|$  in the computation of  $V_{t-1}$  and  $U_{t-1}$ . For implemental simplicity, we use the exponential moving average technique to update  $A_t$  and  $G_t$ . For models equipped with batch normalization layers [16], we perform an extra forward propagation for the selected data  $b_t$  to obtain proper batch statistics for model update, following [31]. Our method takes a similar amount of time per iteration as [31]. The primary difference is that we use a zero-shot predictor based on pre-trained models to compute the second term in Equation (16), whereas [31] uses a validation model. Computing the Gaussian covariance in Equation (15), MC sampling, and updating  $A_t$  and  $G_t$  consume negligible resources.