SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\widetilde {\mathcal {R}} _ {\mathbf {X}} = \mathbb {E} _ {\mathbf {X} _ {r}} \left[ \mathbb {E} [ \| \hat {\nabla} _ {\boldsymbol {\theta}} \mathcal {L} _ {\mathrm {c e}} - \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathrm {c e}} + \hat {\nabla} _ {\boldsymbol {\theta}} h - \nabla_ {\boldsymbol {\theta}} h \| _ {2} ] \right] + \mathbb {E} _ {\mathbf {X} _ {u}} \left[ \mathbb {E} [ \| \hat {\nabla} _ {\boldsymbol {\theta}} \mathcal {L} _ {\mathrm {i m}} - \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathrm {i m}} \| _ {2} ] \right]. \tag {11}
$$
According to the previous study [3], minimizing the cross entropy loss  $\mathcal{L}_{\mathrm{ce}}(\mathbf{y}_i, \hat{\mathbf{p}}_i^\theta)$  is equivalent to maximizing the mutual information  $\mathcal{L}_{\mathrm{im}}$ . Then, the upper bound of the expected gradient estimation error w.r.t.  $\mathbf{X}$  is:
$$
\widetilde {\mathcal {R}} _ {\mathbf {X}} \leq \mathbb {E} _ {\mathbf {X} _ {r}} \left[ \mathbb {E} [ \| \hat {\nabla} _ {\boldsymbol {\theta}} \mathcal {L} _ {\mathrm {c e}} - \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathrm {c e}} \| _ {2} ] + \mathbb {E} [ \| \hat {\nabla} _ {\boldsymbol {\theta}} h - \nabla_ {\boldsymbol {\theta}} h \| _ {2} ] \right] + \mathbb {E} _ {\mathbf {X} _ {u}} \left[ \mathbb {E} [ \| \hat {\nabla} _ {\boldsymbol {\theta}} \mathcal {L} _ {\mathrm {c e}} - \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathrm {c e}} \| _ {2} ] \right]. \tag {12}
$$
Comparing Eq. (10) and Eq. (12), the upper bound of the expected gradient estimation error is tightened after applying the pseudo-label-robust training strategy, justifying that the proposed pseudolabel-robust training strategy can alleviate the disturbance caused by the unreliable pseudo-labels in the gradient estimation in zeroth-order optimization of the test-time data adaptor.
3.5 SODA Framework
Up to this point, we have discussed zeroth-order optimization in test-time data adaptation and our proposed pseudo-label-robust training strategy. The overall training objective is shown in Eq. (13),
$$
\mathcal {L} _ {\text {a l l}} \left(\mathbf {X}, \hat {\mathbf {Y}} _ {r}\right) = - \mathcal {L} _ {\text {i m}} \left(\mathbf {X} _ {u}\right) + \alpha \mathcal {L} _ {\mathrm {c e}} \left(\mathbf {X} _ {r}, \hat {\mathbf {Y}} _ {r}\right), \tag {13}
$$
where  $\alpha$  is the balancing hyperparameter between mutual information maximization loss  $\mathcal{L}_{\mathrm{im}}$  and cross entropy loss  $\mathcal{L}_{\mathrm{ce}}$ . The training algorithm of SODA is summarized in Algorithm 1.
Datasets. We first evaluate our proposed framework SODA on two widely used out-of-distribution benchmarks, namely CIFAR-10-C and CIFAR-100-C [14], each containing 10,000 CIFAR-10/100 test images corrupted by 19 kinds of corruptions with 5 severity levels. We further evaluate SODA on a large scale dataset ImageNet-C [14] with 50,000 ImageNet validation images corrupted by the same corruptions as CIFAR10/100-C. We conduct our experiments on the highest level of each corruption and report the averaged accuracies over 19 corruptions.
Table 1: Average accuracies (%) on CIFAR-10-C (C10-C), CIFAR-100-C(C100-C) and ImageNet-C (IN-C). FO Grad. means the requirement of first-order gradient from the target model or deployed model. Model Mod. means the requirement of modifying the parameters of deployed models. Distill. indicates methods using knowledge distillation to learn target models. DA indicates methods using test-time data adaptation. MA indicates methods using model adaptation.
Categories	Methods	FO Grad.	Model Mod.	C10-C	C100-C	IN-C
-	Deployed	-	-	72.39	41.41	31.36
Distill.	DINE	✓	✗	73.86	40.52	-
BETA	✓	✗	75.71	39.62	-
DA	DA-PGD	✗	✗	24.63	4.15	14.39
DA-ZOO-Input	✗	✗	68.70	31.53	17.57
DA-Direct	✗	✗	70.48	37.67	29.37
DA-PL	✗	✗	72.93	41.44	31.91
SODA (Ours)	✗	✗	82.55	52.41	42.14
SODA-R (Ours)	✓	✗	88.39	60.31	48.70
MA	MA-SO	✓	✓	86.54	62.02	56.90
Baselines. Deployed is the deployed model without adaptation. We compare our proposed SODA framework with two DABP baselines utilizing knowledge distillation. DINE [22] distills the knowledge from the deployed model to a target model by minimizing the KL divergence between smoothed pseudo-labels and target model predictions. BETA [46] divides the target domain into easy- and hard-to-adapt subdomains, then mutually distills twin networks with weak-strong augmentation on two subdomains. We further implement four vanilla baselines of test-time data adaptation using ZOO. DA-Direct directly generates adapted data instead of perturbations using the same network structure and initial pseudo-labels. DA-PGD adopts PGD [27] to directly generate perturbations using estimated gradients computed by ZOO and the training objective of SODA. DA-ZOO-Input uses the same data adaptor and training objective as SODA, except that the variables optimized in ZOO are the input data of the deployed model, i.e., the output adapted data of the data adaptor. The gradients of data adaptor parameters are computed based on the estimated gradients w.r.t. the adapted data. DA-PL trains the same data adaptor as SODA using initial pseudo-labels. Assuming gradient information is accessible, we also implement SODA-R using first-order gradients computed from the deployed model. To compare SODA with model adaptation, we implement MA-SO that modifies model parameters (except the last linear layer) using the same training objective as SODA.
Deployed model settings. For all experiments regarding CIFAR-10/100-C tasks, we adopt the CIFAR-10/100 pretrained ResNet-50 [11] model used in [26] and [38] as the deployed model. For experiments regarding the ImageNet-C task, we adopt the ImageNet pre-trained ResNet-50 model provided by TorchVision [28] as the deployed model. Except for SODA-R and MA-SO, the deployed model is frozen with only output probabilities accessible. For SODA-R, the deployed model is also frozen, but gradients can be computed and back-propagated through the model. For MA-SO, the deployed model is set in training mode, and all parameters can be modified.
Implementation details. For SODA, the data adaptor uses a small network with two convolutional layers and an instance normalization layer in between to generate perturbations added to the original test data. For SODA-R in the CIFAR-10/100-C tasks, two ResNet blocks as in [33] are inserted between the convolutional layers to form a larger data adaptor. For SODA-R in the ImageNet-C task, one ResNet block and a couple of downsampling/upsampling layers are inserted instead. Detailed data adaptor structure is described in Appendix B.3. For all methods except DINE, BETA, DA-PGD, and SODA-R, the data adaptor/model is optimized using SGD with learning rate  $= 1\mathrm{e} - 3$ , momentum  $= 0.9$ , and weight decay  $= 1\mathrm{e} - 5$ . For DINE and BETA, we follow the same settings as in [22] and [46], and the target models are both ResNet-50 initialized with ImageNet-pretrained weights downloaded from TorchVision [28]. For DA-PGD, the step size is also set to be 1e-3. SODA-R is optimized using Adam with learning rate  $= 1\mathrm{e} - 3$  and weight decay  $= 1\mathrm{e} - 5$ . Batch size  $= 256$  is fixed for all methods. The number of training epochs  $= 150$  for all baselines except DINE and BETA. For DINE and BETA, we train them for 120 epochs and fine-tune them for 30 epochs. For all methods using ZOO, the query number  $q = 5$  for CIFAR-10-C and ImageNet-C and  $q = 10$  for CIFAR-100-C, smoothing parameter
Table 2: Comparison of SODA using ZOO with different query numbers and SODA using first-order gradients. The network structure, training objective, and training strategy are the same. Averaged accuracies  $(\%)$  over 19 corruptions are reported.
Query Numbers	2	5	10	20	50	SODA-FO
CIFAR-10-C	82.43	82.55	82.57	82.59	82.53	85.97
CIFAR-100-C	51.03	52.19	52.41	52.97	52.97	54.32
$\mu = 1\mathrm{e} - 3$ . All experiments are repeated with three random seeds. The code implementation can be found at https://github.com/tmlr-group/SODA.
Effectiveness of SODA. Table 1 shows the accuracies on CIFAR-10-C, CIFAR-100-C and ImageNet-C averaged over 19 corruptions. In the mostly restricted setting where first-order gradient computation and model modification are both not allowed, SODA improves the deployed model prediction accuracy by a large margin,  $10\%$  on CIFAR-10-C,  $11\%$  on CIFAR-100-C and  $11\%$  on ImageNet-C. SODA also outperforms the DABP baselines on CIFAR10-C and CIFAR100-C tasks. Note that DINE and BETA are excluded in experiments on ImageNet-C because they are required to initialize their target models by ImageNet pre-trained weights, which violates the TTA setting. Especially on CIFAR-100-C and ImageNet-C task with much lower initial prediction accuracy, all baselines except MA-SO fail to improve, while SODA and SODA-R make significant improvement.
SODA v.s. model adaptation. Further relaxing the restriction on the deployed model, when parameters of the deployed model are frozen, but gradient computation is feasible, SODA-R achieves comparable accuracy with completely unrestricted MA-SO on CIFAR-100-C and even better accuracy on CIFAR-10-C. It shows that data adaptation can be as effective as model adaptation for deployed models with inaccessible parameters. On ImageNet-C, SODA-R performs worse than MA-SO, suggesting that improvement is still needed on large-scale datasets.
Effect of pseudo-label-robust training strategy in SODA. The failures of DA-Direct show that directly generating data using unreliable pseudo-labels fails to adapt data to the deployed model due to the corrupted data features. By adding perturbations to the original test data, DA-PL improves from DA-Direct to a small degree but still fails to enhance the deployed model. With the same data adaptor as DA-PL, SODA can improve the model prediction to a large degree, indicating the effectiveness of the proposed pseudo-label-robust training strategy.
Effect of data adaptor parameter ZOO in SODA. Both using a perturbation generation strategy, DA-PGD fails by directly generating perturbations using estimated gradients without learning a data adaptor, and DA-ZOO-Input achieves worse results by directly optimizing the model input using ZOO. With the same training objective, the significant improvements made by SODA indicate the effectiveness of the data adaptor parameter ZOO adopted in SODA.
Effect of query number in ZOO. To analyze the relation between query number  $q$  and adaptation accuracy in SODA, we conduct experiments with  $q \in \{2, 5, 10, 20, 50\}$ . The results in Table 2 show that SODA is not sensitive to query number used in ZOO. Especially on CIFAR-10-C, only a slight performance drop is observed when  $q = 2$ . On CIFAR-100-C, a larger query number contributes to accuracy improvement more observably, but when  $q > 10$ , the contribution becomes less efficient, as the computation cost is also increased. Balancing the trade-off between accuracy and computation costs, we finally choose  $q = 5$  for CIFAR-10-C and 10 for CIFAR-100-C.
Zeroth-order optimization v.s. first-order optimization. To better illustrate the effect of ZOO, we implement a comparing baseline as SODA-FO that shares the same setting with SODA but uses the first-order gradients back-propagated from the deployed model. Note that SODA-FO and SODA-R differ in data adaptor network structure, optimizer, and other strategies as discussed in Appendix C.1. As shown in Table 2, although SODA does not achieve the same accuracies as SODA-FO due to the unavoidable gradient estimation error, it still indicates competitive performance when gradient computation is infeasible. In Figure 3, SODA using ZOO has the slowest convergence speeds
Figure 3: Accuracy convergence on Gaussian noise in (a) CIFAR-10-C and (b) CIFAR-100-C.
Figure 4: Comparison of SODA trained over different numbers of test samples.
Table 3: Comparison of SODA and SODA-R using data adaptor with different numbers of ResNet blocks. Averaged accuracies  $(\%)$  over 19 corruptions on CIFAR-10-C are reported.
#blocks	0	1	2	3
SODA	82.55	81.91	80.56	78.65
SODA-R	86.07	88.07	88.39	88.51
Table 4: Average accuracies  $(\%)$  on CIFAR-10-C (C10-C) and CIFAR-100-C (C100-C) under online setting with difference batch size (epochs = 10). D is the deployed model, BS is batch size.
Methods	D	SODA-O
BS	-	32	64	128	256
C10-C	72.39	78.01	78.66	78.79	77.03
C100-C	41.41	45.82	47.11	47.26	45.93
and lowest accuracies on both datasets. SODA-FO has similar convergence speeds as SODA-R but converges to lower accuracies, indicating that the strategies used in SODA-R with first-order optimization, i.e., deeper network and Adam optimizer, can boost the training of the data adaptor.
Effect of data adaptor network complexity. We also explore the effect of data adaptor network complexity on SODA and SODA-R. To increase network complexity, we change the number of ResNet blocks in the data adaptor. Results of data adaptor with  $\{0,1,2,3\}$  ResNet blocks are reported in Table 3. For SODA, accuracy decreases as network complexity increases, indicating that complex networks hinder data adaptation using ZOO. However, the accuracy of SODA-R increases along with network complexity. This contrast illustrates that a more complex data adaptor can achieve higher accuracy but is restricted and even encumbered by zeroth-order gradient estimation.
SODA with fewer test data points. We further explore the effectiveness of SODA with fewer test data points. We randomly choose  $\{50, 100, 500, 1000, 5000, 10000\}$  test data points evenly distributed across 10 classes in CIFAR-10-C for each corruption as smaller test datasets. The averaged accuracies are reported in Figure 4. The performance of SODA is better when training data adaptor over more test data points. Nevertheless, training over 5,000 data points achieves comparable accuracy with training over 10,000 data points, showing that SODA does not require an extremely large number of data points to achieve good performance. Besides, SODA can still improve with less than 500 test data points, providing promising insights for test-time data adaption with smaller test datasets.
Hyperparameter sensitivity. We conduct sensitivity analysis regarding the noise ratio  $\rho$ , the confidence threshold  $\tau$  and the balancing parameter  $\alpha$ . The detailed results illustrated in Appendix C.3 show that SODA is robust to different combinations of hyperparameters, but adapting an adapted threshold instead of a fixed threshold might further improve the performance of SODA.
4.4 SODA for Online Test-Time Adaptation
More practically, test data is not entirely available but arrives sequentially, i.e., online test-time adaptation. Hence, we further implement SODA-O as a variant of SODA under online settings. Given mini-batches of test data arrived sequentially  $\{\mathbf{X}_1,\dots,\mathbf{X}_T\}$ , SODA-O adapts one mini-batch at a time before processing the next mini-batch with knowledge accumulated from the previous mini-batches.
The main difference between SODA-O and SODA lies in the reliable pseudo-label selection. Without access to the test data before adaptation, SODA-O maintains an ordered queue  $\mathbf{Q}$  with maximum size  $S$  to store the selected reliable pseudo-labels  $\mathbf{Y}_r$  and their corresponding data points  $\mathbf{X}_r$ . Specifically,
Table 5: Average accuracies (%) on CIFAR-10-C and CIFAR-100-C under online setting with different number of epochs per batch (batch size = 256).
Methods	Deployed	SODA-O	SODA
Epochs/Batch	-	5	10	30	50	100	150	150*
CIFAR-10-C	72.39	75.22	77.03	79.63	80.38	81.33	81.71	82.55
CIFAR-100-C	41.41	43.59	45.81	48.56	49.26	50.04	50.12	52.41
\*SODA is trained over the entire test dataset for 150 epochs
for a mini-batch  $\mathbf{X}_t$  arrives at time  $t$ , reliable pseudo-labels  $\mathbf{Y}_{r_t}$  with prediction confidence higher than  $\tau$  are selected and pushed into  $\mathbf{Q}$  along with their corresponding test data points  $\mathbf{X}_{r_t}$ . To maintain the class balance in  $\mathbf{Q}$ , the pseudo-labels with the smallest confidence for class  $k$  will be popped out once the number of pseudo-labels for  $k$  in  $\mathbf{Q}$  is larger than  $S/C$ . Then, the remaining data points  $\mathbf{X}_{u_t}$  in  $\mathbf{X}_t$  are considered as  $\mathbf{X}_{u_t}'$ , all pseudo-labels and data points stored in  $\mathbf{Q}$  are considered as  $\mathbf{Y}_{r_t}'$  and  $\mathbf{X}_{r_t}'$ .  $\mathbf{X}_{u_t}'$  and  $\mathbf{X}_{r_t}'$  form a small test dataset to train the data adaptor as in SODA, i.e. Step 2 in Algorithm 1. After adaptation, the inference of  $\mathbf{X}_t$  is given by the current data adaptor and the deployed model. Note that the optimization in SODA-O is not repeated after reaching the entire test dataset but only repeats for the current test data batch and the cached queue. During the adaptation of the current test data batch, the previous data batches are no longer available except for those saved in the queue. The data adaptor, hyperparameter setting, and training strategy of SODA-O are the same as SODA. The queue size  $S = 1,000$  for both CIFAR-10-C and CIFAR-100-C.
Effect of epochs per mini-batch. We fix batch size  $= 256$  and conduct experiments with different numbers of epochs per mini-batch, i.e.,  $\{5,10,30,50,100,150\}$  epochs/batch. As shown in Table 5, SODA-O is effective under online setting. As epochs/batch increase, the accuracy of SODA-O also increases and approaches the accuracy of SODA. But more training epochs means more processing time for each mini-batch, leading to a time-accuracy trade-off. With 10 epochs/batch, SODA-O can still improve the deployed model by  $5\%$  and  $4\%$  on CIFAR-10-C and CIFAR-100-C.
Effect of batch size. Fixing epochs/batch  $= 10$ , we also conduct experiments with different batch sizes,  $\{32, 64, 128, 256\}$ . The averaged accuracies are shown in Table 4. The results show that the performance of SODA-O is stable across different batch sizes. The best performance is achieved with batch size  $= 128$ . Only a slight performance drop is observed when batch size is reduced to 32, indicating that SODA-O can handle relatively small batch size.
While providing valuable insights into robust test-time data adaptation for deployed models with inaccessible parameters, it is also essential to consider the limitations that may have affected the efficiency and effectiveness of our proposed framework. The main limitation of SODA is the time consumption brought by i) multiple queries required for gradient estimation in ZOO and ii) multiple passes required for training the random initialized data adaptor. In the future, one solution to alleviate this issue could be leveraging ZOO methods with higher query efficiency and less estimation error. Moreover, it is unclear whether SODA designed for OOD generalization can benefit OOD detection [8, 44]. Thus, we leave the exploration for OOD detection as our future work.
This paper focuses on two major challenges in adapting deployed machine learning models with inaccessible parameters at test-time: unmodifiable model parameters and infeasible gradient computation. Without modifying the model parameters, a data adaptor is adopted to adapt test data to the deployed model. Zeroth-order optimization is further leveraged to train the data adaptor with estimated gradients. Revisiting ZOO in test-time data adaptation, we discover that the unreliable gradient estimation in ZOO is due to the unreliable pseudo-labels assigned to test data. The proposed pseudo-label-robust data adaptation (SODA) addresses this issue with reliable pseudo-label selection and input information maximization. Our experiments on three widely used out-of-distribution benchmarks demonstrate the effectiveness of SODA in both offline and online settings.
This work was partially supported by the National Natural Science Foundation of China (No. 62372459, No. 62376282).
[37] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In Advances in Neural Information Processing Systems, 2022.
[38] Yongyi Su, Xun Xu, and Kui Jia. Revisiting realistic test-time training: Sequential inference and adaptation by anchored clustering. In Advances in Neural Information Processing Systems, 2022.
[39] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning, 2020.
[40] Yun-Yun Tsai, Pin-Yu Chen, and Tsung-Yi Ho. Transfer learning without knowing: Reprogramming black-box machine learning models with scarce data and limited resources. In International Conference on Machine Learning, 2020.
[41] Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference on Learning Representations, 2020.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
[43] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2020.
[44] Qizhou Wang, Feng Liu, Yonggang Zhang, Jing Zhang, Chen Gong, Tongliang Liu, and Bo Han. Watermarking for out-of-distribution detection. In Advances in Neural Information Processing Systems, 2022.
[45] Zehao Xiao, Xiantong Zhen, Ling Shao, and Cees GM Snoek. Learning to generalize across domains on single test samples. In International Conference on Learning Representations, 2021.
[46] Jianfei Yang, Xiangyu Peng, Kai Wang, Zheng Zhu, Jiashi Feng, Lihua Xie, and Yang You. Divide to adapt: Mitigating confirmation bias for domain adaptation of black-box predictors. In International Conference on Learning Representations, 2023.
[47] Yonggang Zhang, Mingming Gong, Tongliang Liu, Gang Niu, Xinmei Tian, Bo Han, Bernhard Schölkopf, and Kun Zhang. Adversarial robustness through the lens of causality. In International Conference on Learning Representations, 2022.
[48] Yonggang Zhang, Ya Li, Tongliang Liu, and Xinmei Tian. Dual-path distillation: A unified framework to improve black-box attacks. In International Conference on Machine Learning, 2020.
[49] Bowen Zhao, Chen Chen, and Shu-Tao Xia. Delta: degradation-free fully test-time adaptation. In International Conference on Learning Representations, 2023.
[50] Xingchen Zhao, Chang Liu, Anthony Sicilia, Seong Jae Hwang, and Yun Fu. Test-time fourier style calibration for domain generalization. In International Joint Conference on Artificial Intelligence, 2022.
[51] Yi Zhou, Parikshit Ram, Theodoros Salonidis, Nathalie Baracaldo, Horst Samulowitz, and Heiko Ludwig. Flora: Single-shot hyper-parameter optimization for federated learning. arXiv preprint arXiv:2112.08524, 2021.
A Derivation of Directional Derivative Approximation in SODA
In Section 3.2, given a deployed model  $\mathbf{M}$ , the ideal objective function of training the data adaptor  $\mathbf{G}$  with parameters  $\theta$  in SODA is the KL divergence between the predicted probability  $\hat{\mathbf{p}}_i^\theta = \mathbf{M} \circ \mathbf{G}(\mathbf{x}_i; \theta)$  of the adapted data point  $\mathbf{x}_i^\theta = \mathbf{G}(\mathbf{x}_i; \theta)$  and the true label  $\mathbf{y}_i$  of the original data point  $\mathbf{x}_i$ . Because  $\mathbf{y}_i$  is not available at test time, pseudo-label  $\hat{\mathbf{y}}_i$  predicted by  $\mathbf{M}$  is adopted as a substitute of  $\mathbf{y}_i$ . Due to the inaccurate model prediction under distribution shifts, there is a disturbance  $\sigma_i$  in  $\hat{\mathbf{y}}_i$  compared to  $\mathbf{y}_i$ , i.e.  $\hat{\mathbf{y}}_i = \sigma_i + \mathbf{y}_i$ . Hence, the KL divergence loss  $\mathcal{L}(\cdot, \cdot) := \mathrm{KL}(\cdot \| \cdot)$  at test data point  $\mathbf{x}_i$  is