SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zige Wang $^{1,2}$  Yonggang Zhang $^{2}$  Zhen Fang $^{3}$  Long Lan $^{4}$  Wenjing Yang $^{4*}$  Bo Han $^{2}$
$^{1}$ School of Computer Science, Peking University  $^{2}$ Hong Kong Baptist University  $^{3}$ University of Technology Sydney  $^{4}$ National University of Defense Technology
Adapting models deployed to test distributions can mitigate the performance degradation caused by distribution shifts. However, privacy concerns may render model parameters inaccessible. One promising approach involves utilizing zeroth-order optimization (ZOO) to train a data adaptor to adapt the test data to fit the deployed models. Nevertheless, the data adaptor trained with ZOO typically brings restricted improvements due to the potential corruption of data features caused by the data adaptor. To address this issue, we revisit ZOO in the context of test-time data adaptation. We find that the issue directly stems from the unreliable estimation of the gradients used to optimize the data adaptor, which is inherently due to the unreliable nature of the pseudo-labels assigned to the test data. Based on this observation, we propose pseudo-label-robust data adaptation (SODA) to improve the performance of data adaptation. Specifically, SODA leverages high-confidence predicted labels as reliable labels to optimize the data adaptor with ZOO for label prediction. For data with low-confidence predictions, SODA encourages the adaptor to preserve data information to mitigate data corruption. Empirical results indicate that SODA can significantly enhance the performance of deployed models in the presence of distribution shifts without requiring access to model parameters.
Deep neural networks have emerged as a dominant tool in solving artificial intelligence problems due to their exceptional performance across various tasks [19, 20, 42], thus being deployed to various environments. However, in practice, these models typically suffer performance degradation due to distribution discrepancies between training and test data [1, 13, 47, 35]. To mitigate this issue, Test-Time Adaptation (TTA) is proposed as a promising solution, where unlabeled test data are leveraged to modify the parameters of deployed models to alleviate performance degradation [7, 21, 39].
In practice, the parameters of deployed models may be unmodifiable and inaccessible in many applications due to intellectual property protection, misuse prevention, or privacy concerns in healthcare and finance [22, 46]. The difficulty in modifying model parameters has hindered previous efforts aimed at adapting deployed models to the test distribution [15, 23, 29]. It is shown that training a data adaptor to modify test data offers an alternative solution to mitigate the performance degradation caused by distribution discrepancy [18, 37]. However, the difficulty in accessing model parameters makes gradient computation a challenging task, hindering these data adaptation methods.
One straightforward approach involves utilizing zeroth-order optimization (ZOO) [25] to train the data adaptor to adapt test data to fit deployed models. In particular, ZOO can be employed to estimate gradients for the optimization of data adaptors without modifying and accessing the parameters of deployed models. Therefore, test-time data adaptation with ZOO makes it possible to improve the performance of deployed models in many practical scenarios. However, introducing data adaptors
*Corresponding Author: Wenjing Yang(wenjing.yang@nudt.edu.cn)
Figure 1: Demonstration of corrupted data and adapted data. The left part shows examples of the original test data from CIFAR-10-C, the corrupted data generated by the data adaptor trained with unreliable pseudo-labels, and the adapted data generated by the data adaptor in SODA. The right part shows the corresponding accuracy of test data, corrupted data, and adapted data on CIFAR-10/100-C.
trained with ZOO brings limited improvement. To endow data adaptors with reliability in improving model performance, we revisit ZOO in the context of test-time data adaptation. Training of the data adaptor depends heavily on the predicted labels. Thus, unreliable predicted labels will lead to unreliable gradient estimations in ZOO, which makes data features corrupted rather than adapted to deployed models. This is consistent with our observations as depicted in Figure 1.
Built upon the observation, we propose pseudo-label-robust data adaptation (SODA) to enhance deployed models with inaccessible parameters. SODA robustifies the training of data adaptors by splitting the test dataset into two subsets. One subset contains data with high-confidence predictions and is regarded as a reliable dataset. The other contains the remaining data points, considered to be an unreliable subset. For data in the reliable dataset, SODA trains the data adaptor using ZOO in a supervised manner. In the meantime, SODA encourages the data adaptor to preserve input information in an unsupervised manner for data corruption mitigation over unreliable data. To verify the efficacy of SODA, we evaluate it on three widely used benchmark datasets under various settings. Our experimental results demonstrate that SODA can effectively mitigate the performance degradation of deployed models in the presence of distribution shifts.
To sum up, our main contributions are presented as follows:
- To enhance deployed models with unmodifiable and inaccessible parameters, we propose to shift from model adaptation to data adaptation, where zeroth-order optimization is employed to estimate gradients for the training of the data adaptor. Unfortunately, test-time data adaptation causes corruption of data features, leading to limited performance improvements.
- To exploit the potential of test-time data adaptation, we find that the limited improvement issue stems from the unreliable nature of the predicted labels used in ZOO. Thus, we propose SODA to robustify the training of test-time data adaptation, where the data adaptor is trained to preserve input information for data with unreliable predictions.
- We verify the effectiveness of SODA under various settings. Our experimental results demonstrate that SODA can directly enhance the deployed model under distribution shifts without accessing and modifying the model parameters.
Test-time adaptation. Test-time adaptation is a machine learning technique that addresses the distribution shift problem using only unlabeled test data before making predictions. To mitigate the distribution discrepancy, most previous works adapt the pre-trained model to test data by modifying
the full or part of the model parameters. Some advanced works [7, 21] adapt the feature extractor to obtain more efficient feature representations of test data, while others [16, 17, 45] modify or replace the last linear classification layer to improve prediction over the extracted features. Batch normalization calibration [23, 34, 43, 49] is also exploited to adjust the statistics and affine parameters in batch normalization layers. Unlike previous works, our work focuses on situations where model parameters are unmodifiable and adapts test data to the pre-trained model as an alternative solution.
White-box input adaptation. Recently, several works have put effort into input-level optimization in test-time adaptation by changing the input data or features. Auxiliary auto-encoders [12, 18], amplitude features and Fourier-style calibration [50], label-preserving features along with a generative model [31], and generative diffusion model [10] are utilized to achieve input-level adaptation. These works require either modification of the model training process or gradients from the deployed model at test time. In contrast, our work focuses on test-time data adaptation without accessing the training process and gradients of the deployed model, making it more practical and broadly applicable.
Domain adaptation of black-box predictors. Domain Adaptation of Black-Box Predictors (DABP) is a subcategory of unsupervised domain adaptation [9, 30] that solves a more restricted and practical application setting. Under this setting, the pre-trained model is treated as a black box, and only the model's output is available during adaptation. Few works [22, 32, 46] have proposed solutions to this challenge using knowledge distillation to transfer knowledge from the black-box model to target models. In this work, we address the black-box setting in the context of test-time adaptation and propose to perform data adaptation without knowledge transfer. Besides, while previous DABP works require training on the entire unlabeled test dataset, our proposed method can also deal with online black-box settings where the test data arrive sequentially.
Zeroth-order optimization. Zeroth-order optimization (ZOO) [25] is a family of optimization methods that do not require gradient information to search for the optimal solution. Instead, it explores the searching space using various techniques [6, 24] to estimate the optimization direction. Although it can be less efficient than first-order optimization methods, ZOO methods can be helpful in scenarios where gradient information is unavailable or expensive to compute. For instance, recent studies have applied ZOO to perform adversarial attacks to black-box neural networks [48], hyperparameter optimization in federated learning without gradient descent [51], and transfer learning on black-box models [40]. In this work, we propose to leverage ZOO for test-time data adaptation on deployed models with inaccessible gradient information.
We mainly focus on the  $C$ -way image classification task with a distribution shift between the training and test data, following previous works [26, 39, 43]. Given a deployed model  $\mathbf{M}$  with inaccessible parameters, our purpose is to improve its prediction accuracy on unlabeled test data  $\mathbf{X} = \{\mathbf{x}_1,\dots,\mathbf{x}_n\}$ . Since the parameters and inner structures of  $\mathbf{M}$  are unknown, only the output prediction probabilities are available from  $\mathbf{M}$  during the entire adaptation process. Namely, the proposed method pseudolabel-robust data adaptation (SODA) aims to adapt  $\mathbf{X}$  to  $\mathbf{M}$  without requiring access to the parameters of  $\mathbf{M}$ . The overall framework of SODA is shown in Figure 2.
Before elaborating on our proposed framework, we introduce zeroth-order optimization (ZOO). ZOO is a gradient-free alternative of first-order optimization (FOO), e.g., SGD, SCD, and Adam. Most ZOO methods follow the structure of FOO and consist of three fundamental steps [25]: gradient estimation, descent direction computation, and point updating. They utilize function-value-based gradient estimations to approximate the full or stochastic gradients computed in FOO.
One commonly used ZOO gradient estimation strategy is multi-point estimation [6, 24]. Given a continuously differentiable objective function  $f(\pmb{\theta})$  on a  $d$ -dimensional variable  $\pmb{\theta} \in \mathbb{R}^d$ , multi-point estimation computes directional derivative approximation as follows:
$$
\widehat {\nabla} _ {\boldsymbol {\theta}} f (\boldsymbol {\theta}) := \frac {1}{\mu q} \sum_ {i = 1} ^ {q} \left[ \left(f \left(\boldsymbol {\theta} + \mu \mathbf {u} _ {i}\right) - f (\boldsymbol {\theta})\right) \mathbf {u} _ {i} \right], \tag {1}
$$
Figure 2: The overall framework of SODA. (a) Before adaptation, SODA first performs reliable pseudo-label selection according to prediction confidence. (b) During adaptation, the data adaptor with parameter  $\theta$  is trained over the test data with reliable pseudo-labels using cross-entropy loss and those with unreliable pseudo-labels using mutual information maximization. The gradient is estimated using (c) zeroth-order optimization.
where  $\mathbf{u}_1, \dots, \mathbf{u}_q$  are  $q$  random direction vectors typically drawn from the standard multivariate normal distribution  $\mathcal{N}(\mathbf{0}, \mathbf{I})$ , and  $\mu$  is the smoothing parameter. On a mini-batch of data points  $\mathbf{x}_1, \dots, \mathbf{x}_l$ , the estimated gradient  $\hat{\delta}$  is the average of the multi-point estimations to all data points  $\mathbf{x}_1, \dots, \mathbf{x}_l$ :
$$
\hat {\boldsymbol {\delta}} = \frac {1}{l} \sum_ {i = 1} ^ {l} \widehat {\nabla} _ {\boldsymbol {\theta}} f (\boldsymbol {\theta}; \mathbf {x} _ {i}). \tag {2}
$$
Various strategies are adopted to compute the descent direction. For zeroth-order stochastic gradient descent (ZO-SGD) utilized in our work, the descent direction is set as the current gradient estimation  $\hat{\delta}$ . The point updating rule is the same as in traditional SGD: with learning rate  $\eta$ ,
$$
\boldsymbol {\theta} = \boldsymbol {\theta} - \eta \hat {\boldsymbol {\delta}}. \tag {3}
$$
3.2 Zeroth-Order Optimization in Test-Time Data Adaptation
Let  $\mathbf{G}$  with parameters  $\theta$  be the data adaptor for test-time data adaptation. For each test data point  $\mathbf{x}_i$ $(i = 1,\dots,n)$ ,  $\mathbf{G}$  transforms it to form the adapted data  $\mathbf{x}_i^\theta$  for inference as follows:
$$
\mathbf {x} _ {i} ^ {\boldsymbol {\theta}} = \mathbf {G} (\mathbf {x} _ {i}; \boldsymbol {\theta}). \tag {4}
$$
Ideally, the data adaptor  $\mathbf{G}$  should be trained by minimizing the KL divergence between the predicted probabilities of the adapted data and the true labels of test data. Typically, the training process requires back-propagating the gradients from the deployed model to the data adaptor. A challenge arises as gradient computation is infeasible for the deployed model  $\mathbf{M}$  with inaccessible parameters. In this regard, ZOO provides an effective approach to estimating gradients, as discussed in the previous section. Considering the parameters  $\theta$  of the data adaptor  $\mathbf{G}$  as the variables to be optimized, utilizing ZOO in test-time data adaptation is to replace the objective function  $f(\theta)$  with the training objective function used to train the data adaptor  $\mathbf{G}$ . Assuming that the true one-hot label  $\mathbf{y}_i$  of a test data point  $\mathbf{x}_i$  is given, and replacing the function  $f(\theta)$  in Eq. (1) with KL divergence loss  $\mathcal{L}(\cdot ,\cdot)\coloneqq \mathrm{KL}(\cdot \| \cdot)$ , the directional derivative approximation w.r.t.  $\mathcal{L}(\cdot ,\cdot)$  and  $(\mathbf{x}_i,\mathbf{y}_i)$  is
$$
\widehat {\nabla} _ {\boldsymbol {\theta}} \mathcal {L} _ {i} = \frac {1}{\mu q} \sum_ {j = 1} ^ {q} \left[ \left(\mathcal {L} \left(\mathbf {y} _ {i}, \mathbf {M} \circ \mathbf {G} \left(\mathbf {x} _ {i}; \boldsymbol {\theta} + \mu \mathbf {u} _ {j}\right)\right) - \mathcal {L} \left(\mathbf {y} _ {i}, \mathbf {M} \circ \mathbf {G} \left(\mathbf {x} _ {i}; \boldsymbol {\theta}\right)\right)\right) \mathbf {u} _ {j} \right]. \tag {5}
$$
In test-time adaptation, the true labels of test data are obviously unknown. A common strategy is to use the predicted pseudo-label  $\hat{\mathbf{y}}_i$  as the substitute of the true label  $\mathbf{y}_i$ .
However, the pseudo-labels are unreliable due to the inaccurate model prediction under distribution shifts, causing the corrupted data features depicted in Figure 1. Let  $\sigma_{i}$  denote the disturbance of pseudo-label  $\hat{\mathbf{y}}_i$ , i.e.,  $\hat{\mathbf{y}}_i = \boldsymbol{\sigma}_i + \mathbf{y}_i$ , and  $\hat{\mathbf{p}}_i^\theta = \mathbf{M} \circ \mathbf{G}(\mathbf{x}_i; \boldsymbol{\theta})$  denote the predicted probability of the adapted data point  $\mathbf{x}_i^\theta = \mathbf{G}(\mathbf{x}_i; \boldsymbol{\theta})$ , the KL divergence loss at test point  $\mathbf{x}_i$  becomes:
$$
\mathcal {L} _ {i} = - H \left(\mathbf {y} _ {i} + \boldsymbol {\sigma} _ {i}\right) + \mathcal {L} _ {\mathrm {c e}} \left(\mathbf {y} _ {i}, \hat {\mathbf {p}} _ {i} ^ {\boldsymbol {\theta}}\right) - \boldsymbol {\sigma} _ {i} \log \hat {\mathbf {p}} _ {i} ^ {\boldsymbol {\theta}}, \tag {6}
$$
where  $\mathcal{L}_{\mathrm{ce}}(\cdot ,\cdot)$  is the cross-entropy loss. Then, replacing  $\mathbf{y}_i$  with  $\hat{\mathbf{y}}_i$  in Eq. (5), the directional derivative approximation becomes
$$
\widehat {\nabla} _ {\boldsymbol {\theta}} \check {\mathcal {L}} _ {i} = \widehat {\nabla} _ {\boldsymbol {\theta}} \mathcal {L} _ {\mathrm {c e}} + \frac {\sigma_ {i}}{\mu q} \sum_ {j = 1} ^ {q} \log \frac {\hat {\mathbf {p}} _ {i} ^ {\boldsymbol {\theta}}}{\hat {\mathbf {p}} _ {i} ^ {\boldsymbol {\theta} + \mu \mathbf {u} _ {j}}} \mathbf {u} _ {j}, \tag {7}
$$
where  $\widehat{\nabla}_{\theta}\mathcal{L}_{\mathrm{ce}} = \frac{1}{\mu q}\sum_{j = 1}^{q}\left[\left(\mathcal{L}_{\mathrm{ce}}(\mathbf{y}_i,\hat{\mathbf{p}}_i^{\boldsymbol{\theta} + \mu \mathbf{u}_j}) - \mathcal{L}_{\mathrm{ce}}(\mathbf{y}_i,\hat{\mathbf{p}}_i^\boldsymbol {\theta})\right)\mathbf{u}_j\right]$  is the ideal directional derivative approximation. The derivations of Eq. (6) and Eq. (7) are deferred to Appendix A. In Eq. (7), the last term is the disturbing term directly introduced by  $\sigma_{i}$ , causing the difference between  $\widehat{\nabla}_{\theta}\mathcal{L}_{\mathrm{ce}}$  and the unreliable directional derivative approximation  $\widehat{\nabla}_{\theta}\check{\mathcal{L}}_i$ .  $\widehat{\nabla}_{\theta}\check{\mathcal{L}}_i$  further leads to unreliable estimated gradients in Eq. (2), which hinders the optimization of  $\theta$  and the training of  $\mathbf{G}$ .
3.3 Pseudo-Label-Robust Training
A direct strategy to alleviate the impact of the disturbing term in Eq. (7) is to select pseudo-labels with small  $\sigma_{i}$ . The selected pseudo-labels form reliable pseudo-label set  $\hat{\mathbf{Y}}_r$  to train the data adaptor in a supervised manner. In particular, two basic criteria for reliable pseudo-label selection are adopted in SODA: 1) the prediction confidence should be higher than a threshold  $\tau$ , indicating the selected pseudo-labels have small disturbances; 2) the number of selected reliable pseudo-labels for each class should be less than  $(1 - \rho)n / C$  to maintain the balance among classes, where  $\rho$  is the noise ratio, and  $C$  is the number of classes. The test data points corresponding to the selected reliable pseudo-labels are considered as reliable data set  $\mathbf{X}_r$ , which is trained over with cross-entropy loss  $\mathcal{L}_{\mathrm{ce}}$  and  $\hat{\mathbf{Y}}_r$ .
To mitigate the data corruption over the remaining test data points  $\mathbf{X}_u$  with unreliable pseudo-labels, SODA trains over them in an unsupervised manner. Following previous works [21, 22], mutual information maximization [4, 36, 41] is a widely-used unsupervised loss that can encourage both global diversity and local certainty of model predictions by maximizing the mutual information between the input data sample and the predicted probabilities. Thus, it is adopted to preserve input information in  $\mathbf{X}_u$  as shown in Eq. (8), where  $\mathbf{x}_i^\theta = \mathbf{G}(\mathbf{x}_i; \boldsymbol{\theta})$  and  $\hat{\mathbf{p}}_i = \mathbf{M} \circ \mathbf{G}(\mathbf{x}_i; \boldsymbol{\theta})$ .
$$
\mathcal {L} _ {\mathrm {i m}} \left(\mathbf {X} _ {u} ^ {\boldsymbol {\theta}}\right) = \mathbb {E} _ {\mathbf {x} _ {i} ^ {\boldsymbol {\theta}} \in \mathbf {X} _ {u} ^ {\boldsymbol {\theta}}} \left[ \sum_ {k = 1} ^ {C} \hat {\mathbf {p}} _ {i k} \log \hat {\mathbf {p}} _ {i k} \right] - \sum_ {k = 1} ^ {C} \mathbb {E} _ {\mathbf {x} _ {i} ^ {\boldsymbol {\theta}} \in \mathbf {X} _ {u} ^ {\boldsymbol {\theta}}} \hat {\mathbf {p}} _ {i k} \log \mathbb {E} _ {\mathbf {x} _ {i} ^ {\boldsymbol {\theta}} \in \mathbf {X} _ {u} ^ {\boldsymbol {\theta}}} \hat {\mathbf {p}} _ {i k}. \tag {8}
$$
To theoretically show the effectiveness of the proposed pseudo-label-robust training strategy, we analyze the expected gradient estimation error [5, 24] in the training of test-time data adaptor with zeroth-order optimization. For simplicity, we consider the special case where the estimated gradient equals directional derivative approximation with mini-batch size equal to 1. The expected gradient estimation error  $\mathcal{R}_{\mathbf{X}}$  between the estimated gradient  $\widehat{\nabla}_{\theta}\check{\mathcal{L}}_i$  and the true gradient  $\nabla_{\theta}\mathcal{L}_i$  w.r.t. the whole test dataset  $\mathbf{X}$  is:
$$
\mathcal {R} _ {\mathbf {X}} = \mathbb {E} _ {\mathbf {X}} \left[ \mathbb {E} [ \| \hat {\nabla} _ {\boldsymbol {\theta}} \tilde {\mathcal {L}} _ {i} - \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {i} \| _ {2} ] \right]. \tag {9}
$$
Denoting  $h(\mathbf{x}_i) = -\sigma_i \log \hat{\mathbf{p}}_i^\theta$  in Eq. (6), the gradient of the KL divergence loss is  $\nabla_{\theta} \mathcal{L}_i = \nabla_{\theta} \mathcal{L}_{\mathrm{ce}} + \nabla_{\theta} h$ . Accordingly, the estimated gradient of the KL divergence loss is  $\widehat{\nabla}_{\theta} \check{\mathcal{L}}_i = \widehat{\nabla}_{\theta} \mathcal{L}_{\mathrm{ce}} + \widehat{\nabla}_{\theta} h$ . Then, before applying pseudo-label-robust data adaptation, the upper bound of expected gradient estimation error is:
$$
\mathcal {R} _ {\mathbf {X}} \leq \mathbb {E} _ {\mathbf {X}} \left[ \mathbb {E} [ \| \hat {\nabla} _ {\boldsymbol {\theta}} \check {\mathcal {L}} _ {\mathrm {c e}} - \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathrm {c e}} \| _ {2} ] + \mathbb {E} [ \| \hat {\nabla} _ {\boldsymbol {\theta}} h - \nabla_ {\boldsymbol {\theta}} h \| _ {2} ] \right]. \tag {10}
$$
Algorithm 1 SODA framework
Input: test data  $\mathbf{X}$  , data adaptor  $\mathbf{G}$  with parameters  $\theta$  , deployed model M, adaptation epochs  $N,$  learning rate  $\eta ,\alpha ,q$    
STEP 1: Before adaptation:   
Compute  $\hat{\mathbf{p}} = \mathbf{M}(\mathbf{X})$    
Select  $\hat{\mathbf{Y}}_r$ $\mathbf{X}_r$  and  $\mathbf{X}_u$  as described in Section 3.3   
STEP 2: During adaptation:   
for 1 to  $N$  do for mini-batch  $\mathbf{X_b} = \{\mathbf{x_{b_1}},\mathbf{x_{b_2}},\dots,\mathbf{x_{b_l}}\}$  in  $\mathbf{X}$  do Compute  $\hat{\mathbf{Y}}_{\mathrm{b}} = \{\hat{\mathbf{y}}_{\mathrm{b}_{1}},\hat{\mathbf{y}}_{\mathrm{b}_{2}},\dots,\hat{\mathbf{y}}_{\mathrm{b}_{l}}\}$  , where  $\hat{\mathbf{y}}_{\mathrm{bi}} = \mathbf{M}(\mathbf{x}_{\mathrm{bi}})$  Select  $\hat{\mathbf{Y}}_{r_b} = \hat{\mathbf{Y}}_r\cap \hat{\mathbf{Y}}_b = \{\hat{\mathbf{y}}_{\mathrm{b}_1},\hat{\mathbf{y}}_{\mathrm{b}_2},\dots,\hat{\mathbf{y}}_{\mathrm{b}_{l_r}}\}$  and their corresponding data  $\mathbf{X}_{r_b} =$ $\{\mathbf{x}_{r_1},\mathbf{x}_{r_2},\dots,\mathbf{x}_{r_{l_r}}\}$  , the remaining data as  $\mathbf{X}_{u_b} = \{\mathbf{x}_{u_1},\mathbf{x}_{u_2},\dots,\mathbf{x}_{u_{l_u}}\}$ $\{\mathbf{X}_{r_b}\cup \mathbf{X}_{u_b} = \mathbf{X}_b\}$  Compute  ${\bf x}_{r_i}^{\theta}$  and  ${\bf x}_{u_i}^{\theta}$  using Eq. (5) for each data point in  $\mathbf{X}_{r_b}$  and  $\mathbf{X}_{u_b}$    
Compute  $\mathcal{L}_{\mathrm{ce}}$  for  $\mathbf{X}_{rb}^{\theta} = \{\mathbf{x}_{r_1}^{\theta},\mathbf{x}_{r_2}^{\theta},\dots,\mathbf{x}_{r_{l_r}}^{\theta}\}$  using  $\hat{\mathbf{Y}}_{r_b}$    
Compute  $\mathcal{L}_{\mathrm{im}}$  for  $\mathbf{X}_{ub}^{\theta} = \{\mathbf{x}_{u_1}^{\theta},\mathbf{x}_{u_2}^{\theta},\dots,\mathbf{x}_{u_{l_u}}^{\theta}\}$  using Eq. (8)   
Compute estimated gradients  $\hat{\delta}_{\mathrm{ce}}$  and  $\hat{\delta}_{\mathrm{im}}$  w.r.t.  $\mathcal{L}_{\mathrm{ce}}$  and  $\mathcal{L}_{\mathrm{im}}$  using Eq. (5) and Eq. (2)  $\theta \gets \theta -\eta (\hat{\delta}_{\mathrm{im}} + \alpha \hat{\delta}_{\mathrm{ce}})$    
end for   
end for
In pseudo-label-robust training strategy, the test dataset  $\mathbf{X}$  is separated into reliable set  $\mathbf{X}_r$  learned by cross-entropy loss with pseudo-labels, and unreliable set  $\mathbf{X}_u$  learned by mutual information loss, the expected gradient estimation error w.r.t.  $\mathbf{X}$  becomes: