SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\widetilde {L} _ {t} ^ {\psi} (r) = \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ \partial \psi (r (\mathbf {x})) r (\mathbf {x}) - \psi (r (\mathbf {x})) \right] - \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {t} (\mathbf {x})} \left[ \partial \psi (r (\mathbf {x})) \right] \tag {5}
$$
is the empirical approximation of the expected loss  $L_{t}^{\psi}$  using the data  $S_{0}$ .
Remark 1 (assumptions on  $\psi$ ). We imposed certain assumptions on the divergence function  $\psi$  in Theorem 1. One can check these conditions hold for commonly used  $\psi$  in density ratio estimation, including all the divergence functions mentioned earlier ( $\psi_{\mathrm{LR}}$  and  $\psi_{\mathrm{KL}}$  are strongly convex when the inputs are upper bounded, which can be satisfied with suitable choices of  $\mathcal{H}_{\theta}$ ). Section 4 will present an example with  $\psi_{\mathrm{LR}}$  to show how the conditions are satisfied.
Theorem 1 shows that we can immediately obtain a sequence of high-quality density ratio estimators  $\{r_t^*\}_{t=1}^T$  by minimizing the dynamic regret with respect to  $\{\widetilde{L}_t^\psi\}_{t=1}^T$ ,
$$
\operatorname {R e g} _ {T} ^ {\mathbf {d}} \left(\left\{\widetilde {L} _ {t} ^ {\psi}, r _ {t} ^ {*} \right\} _ {t = 1} ^ {T}\right) = \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} ^ {\psi} (\widehat {r} _ {t}) - \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} ^ {\psi} \left(r _ {t} ^ {*}\right). \tag {6}
$$
One caveat is that the second term of  $\widetilde{L}_t^\psi$  in the definition (5) requires the knowledge of underlying distribution  $\mathcal{D}_t$ , which is unavailable. Empirically, we can only observe  $\widehat{L}_t^\psi$  defined below, building upon the empirical observations  $S_t \sim \mathcal{D}_t$ ,
$$
\widehat {L} _ {t} ^ {\psi} (r) = \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ \partial \psi (r (\mathbf {x})) r (\mathbf {x}) - \psi (r (\mathbf {x})) \right] - \mathbb {E} _ {\mathbf {x} \sim S _ {t}} \left[ \partial \psi (r (\mathbf {x})) \right]. \tag {7}
$$
That said, we need to design an online optimization process to minimize the dynamic regret (6) defined over  $\widetilde{L}_t^\psi$  based on the observed loss  $\{\widehat{L}_t^\psi\}_{t=1}^T$ . Since the time-varying comparator  $r_t^*$  in (6)
is not the minimizer of the observed loss  $\widehat{L}_t^\psi$  (but rather the minimizer of the expected loss  $L_t^\psi$  defined in (4)), directly minimizing the empirical loss  $\widehat{L}_t^\psi$  will lead to a high estimation error. In the next section, we introduce how to optimize the dynamic regret with the online ensemble framework developed in recent studies of non-stationary online convex optimization [15, 16, 19].
Remark 2 (comparison with previous work). For the continuous label shift [18], the online ensemble framework was employed to train the predictor  $\mathbf{w}_t$ . However, the previous method crucially relies on the construction of an unbiased importance ratio estimator satisfying  $\mathbb{E}_{\mathcal{D}_t}[\widehat{r}_t(\mathbf{x})] = r_t^* (\mathbf{x})$ . Such a favorable property is hard to be satisfied in the covariate shift case. For example, in the Bregman divergence density ratio matching framework, one can only observe an unbiased loss  $\widehat{L}_t^\psi (r)$ , whose minimizer  $\widehat{r}_t = \arg \min_r\widehat{L}_t^\psi (r)$  is not unbiased to the true value  $r_t^*$  in general. To this end, we disentangle the model training and importance weight estimation process in this paper. Our reduction holds for the general Bergman divergence matching framework and thus can be initiated with different models (not necessarily unbiased). It is possible to extend our framework to continuous label shift problem with other importance weight estimators besides the unbiased one [40] used in [18].
4 Instantiation: Logistic Regression Model
When the loss function  $\widehat{L}_t^\psi$  is non-convex, it is generally intractable to conduct the online optimization, regardless of minimizing the standard regret or the strengthened dynamic regret. Fortunately, the attained loss functions are convex or enjoy even stronger curvature with the properly chosen hypothesis space and divergence function. In this section, we instantiate our framework with the logistic regression model. A corresponding online DRE method is presented with dynamic regret guarantees.
Example 1 (Logistic regression model). Consider the function  $\psi = \psi_{\mathrm{LR}} \triangleq t \log t - (t + 1) \log (t + 1)$  and the hypothesis space  $H_{\theta}^{\mathsf{LR}} = \{\mathbf{x} \mapsto \exp(-\theta^\top \phi(\mathbf{x})) \mid \| \theta \|_2 \leq S\}$ . Here,  $\phi: \mathcal{X} \mapsto \mathbb{R}^d$  represents a specific basis function with bounded norm  $\| \phi(\mathbf{x}) \|_2 \leq R$ , which could be, for instance, the feature representation extractor from a deep neural network. Then, the loss function  $\widehat{L}_t^\psi(\theta)$  as per (7) becomes
$$
\widehat {L} _ {t} ^ {\psi} (\theta) = \frac {1}{2} \Bigl (\mathbb {E} _ {S _ {0}} [ \log (1 + e ^ {- \phi (\mathbf {x}) ^ {\top} \theta}) ] + \mathbb {E} _ {S _ {t}} [ \log (1 + e ^ {\phi (\mathbf {x}) ^ {\top} \theta}) ] \Bigr).
$$
Let  $\beta = \exp (SR)$ . Then, the output of any density ratio function  $r\in \mathcal{H}_{\theta}$  is bounded by  $[1 / \beta ,\beta ]$ . It can be validated that  $\psi_{\mathrm{LR}}$  is  $1 / (\beta +\beta^{2})$ -strongly convex and satisfies the condition  $\partial^3\psi_{\mathrm{LR}}(z)\leq 0$  required by Theorem 1. Besides, the logistic loss is a  $1 / (1(1 + \beta))$  -expconcave function and  $R^2 /2$  smooth function, presenting a favorable function properties for online convex optimization [13].
We note that the other two divergence functions  $\psi_{\mathrm{LS}}$  and  $\psi_{\mathrm{KL}}$  also exhibit desirable function properties as  $\psi_{\mathrm{LR}}$ . When choosing the hypothesis space  $H_{\theta} = \{\mathbf{x}\mapsto \theta^{\top}\phi (\mathbf{x})\mid \theta \in \Theta \}$ , the methods recover the uLISF [28] and UKL [37] density ratio estimators equipped with the generalized linear model. Our analysis is also applicable in the two cases. More details are provided in Appendix D.7.
4.1 Density Ratio Estimation via Online Ensemble
In this part, we introduce our online ensemble method for online DRE with the logistic regression model. Since the logistic regression loss is exp-concave, [19] shows that one can employ the following-the-leading-history (FLH) method [41] to minimize the dynamic regret. However, a caveat is that the observed loss  $\widehat{L}_t$  is only an empirical estimation of  $\widetilde{L}_t$  established on few online data  $S_t$ . The result of [19] only implies an expected bound for our problem.
We twisted the FLH algorithm to achieve a high probability bound. As shown in Figure 1, our algorithm maintains multiple base-learners, each learning over different intervals of the time-horizon, and then employs a meta-learner to aggregate their predictions. We modify the meta-learner in FLH from Hedge [42] to Adapt-ML-Prod [43], which ensures that the meta-learner can track each base-learner on the associated time interval with high probability. This strategy allows us to selectively reuse the historical information to handle the non-stationary environments. We introduce ingredients of our algorithms as follows. A more detailed algorithm describes and comparison with the dynamic regret minimization literature in OCO can be found in Appendices D.1 and B respectively.
Base-learner. Our algorithm runs multiple base-learners that are active on different intervals of the time horizon. Let  $\mathcal{C} = \{I_i = [s_i, e_i] \subseteq [T]\}$  be a interval set. Each base-learner  $\mathcal{E}_i$  will only
Figure 1: An illustration of our online ensemble method, where we employ a meta-learner to aggregate the predictions from base-learners running over different intervals of the time horizon.
update and submit the model  $\widehat{\theta}_{t,i}$  to the meta-learner if  $t\in I_i$ . We use the online Newton step (ONS) for the model update. For an exp-concave loss function, the method ensures the base-learner's model is comparable with the best fixed model over the interval up to a logarithmic factor in  $T$  [44]. Specifically, for each base-learner  $\mathcal{E}_i$  running on the interval  $[s_i,e_i]$ , ONS updates the model by
$$
\widehat {\theta} _ {t + 1, i} = \Pi_ {\Theta} ^ {A _ {t, i}} \left[ \widehat {\theta} _ {t, i} - \gamma A _ {t, i} ^ {- 1} \nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t, i}\right) \right], \forall t \in \left[ s _ {i}, e _ {i} \right], \tag {8}
$$
where the matrix  $A_{t,i} = \lambda I + \sum_{\tau = s_i}^t\nabla \widehat{L}_\tau (\widehat{\theta}_{\tau ,i})\nabla \widehat{L}_s(\widehat{\theta}_{\tau ,i})^\top$ . In the above, the projection function is defined as  $\Pi_{\Theta}^{A_{t,i}}[\theta_1] = \arg \min_{\theta \in \Theta}\| \theta -\theta_1\|_{A_{t,i}}$  and  $\Theta = \{\theta \in \mathbb{R}^d\mid \| \theta \| _2\leq S\}$  is the parameter space. The constant  $\lambda >0$  and  $\eta >0$  are the regularizer parameter and step size to be specified latter.
Meta-learner. As shown by Figure 1, the meta-learner's role is to aggregate the models produced by the base-learners using a weighting scheme. At each iteration  $t$ , the meta-learner will maintain a weight  $p_{t,i}$  for each "active" base-learner  $\mathcal{E}_i$ , defined as the one whose associated interval contains  $t$ . We update the weights for active base-learners based on the Adapt-ML-Prod method [43]. Specifically, for every active base-learner  $\mathcal{E}_i$ , our algorithm maintains a "potential"  $v_{t,i} \in \mathbb{R}_{+}$  at iteration  $t$ , which reflects the historical performance of the base-learner until time  $t$ . Then, denoting by  $\mathcal{A}_t$  the index set of the active base-learners at time  $t$ , their weights and the output model are obtained by
$$
p _ {t, i} \propto \varepsilon_ {t - 1, i} v _ {t - 1, i} \text {f o r a l l} i \in \mathcal {A} _ {t} \quad \text {a n d} \quad \widehat {\theta} _ {t + 1} = \sum_ {i \in \mathcal {A} _ {t + 1}} p _ {t + 1, i} \widehat {\theta} _ {t + 1, i}. \tag {9}
$$
In the above  $\varepsilon_{t,i} > 0$  is a step size that can be automatically tuned along the learning process. A noteworthy ingredient of our algorithm is the construction of the potential  $v_{t,i}$ , using the linearized loss  $\langle \nabla \widehat{L}_t(\widehat{\theta}_t),\widehat{\theta}_t - \widehat{\theta}_{t,i}\rangle /(SR)$ . Then, the generalization gap between  $\widehat{L}_t$  and  $\widetilde{L}_t$  can be controlled with the negative term introduced by the exp-concave loss function. As a result, we can establish a high probability bound to ensure that  $\widehat{\theta}_t$  is competitive with any base-learner's model  $\widehat{\theta}_{t,i}$  on the corresponding interval  $I_{i}$ . The detailed configurations of  $v_{t,i}$  and  $\varepsilon_{t,i}$  are deferred to Appendix D.1.
Schedule of Intervals. As shown in Figure 1, we specify the intervals with the geometric covering scheme [41], where the interval set is defined as  $\mathcal{C} = \bigcup_{k\in \mathbb{N}\cup \{0\}}\mathcal{C}_k$  with  $\mathcal{C}_k = \left\{\left[i\cdot 2^k,(i + 1)\cdot 2^k -1\right]\mid i\in \mathbb{N}\text{and} i\cdot 2^k\leq T\right\}$ . One can check that  $|\mathcal{C}|$  is at most  $T$ , and the number of active base-learner is bounded as  $|\mathcal{A}_t|\leq \lceil \log t\rceil$ . Thus, we only need to maintain at most  $\mathcal{O}(\log t)$  base-learners at time  $t$ . The intervals specified by the geometric covering are informative to capture the non-stationarity of the environments, leading to the following dynamic regret guarantees.
4.2 Theoretical Guarantees
This part presents the theoretical guarantees of our method. The estimator  $\widehat{r}_t(\mathbf{x}) = \exp (-\phi (\mathbf{x})^\top \widehat{\theta}_t)$  established on  $\widehat{\theta}_t$  returned by the online ensemble method (9) achieves the dynamic regret guarantee.
Theorem 2. Assume the true density ratio  $r_t^*(\mathbf{x}) = \mathcal{D}_t(\mathbf{x}) / \mathcal{D}_0(\mathbf{x})$  is contained in the hypothesis space as  $r_t^* \in H_\theta^{\mathrm{LR}} \triangleq \{\mathbf{x} \mapsto \exp(-\phi(\mathbf{x})^\top \theta) \mid \theta \in \Theta\}$  for any  $t \in [T]$ . Then, with probability at least  $1 - \delta$ , the dynamic regret of the density ratio estimator  $\widehat{r}_t(\mathbf{x}) = \exp(-\phi(\mathbf{x})^\top \widehat{\theta}_t)$  is bounded by
$$
\mathbf {R e g} _ {T} ^ {\mathbf {d}} \left(\left\{\widetilde {L} _ {t}, r _ {t} ^ {*} \right\} _ {t = 1} ^ {T}\right) \leq \widetilde {\mathcal {O}} \left(\max  \left\{T ^ {\frac {1}{3}} V _ {T} ^ {\frac {2}{3}}, 1 \right\} + T / N _ {0}\right),
$$
when the parameters are set as  $\gamma = 3(1 + \beta)$  and  $\lambda = 1$ . In the above,  $V_{T} = \sum_{t=2}^{T} \| \mathcal{D}_{t}(\mathbf{x}) - \mathcal{D}_{t-1}(\mathbf{x}) \|_{1}$  measures the variation of input densities.
Theorem 2 imposes a realizable assumption for the true density ratio function such that  $r_t^* \in H_\theta^{\mathrm{LR}}$ . Such an assumption is required since we train the density ratio estimator on a given hypothesis space while the true density ratio function could be arbitrary. We note that the realizability assumption is frequently used in the analysis for the density ratio estimation problem [29, 45] and other related topics, including active learning [46] and contextual bandit [47], where the density ratio (or density) estimation is required. A possible direction to relax the assumption is to consider a richer function class, e.g., a neural network or a non-parametric model. We leave this as a future work.
The  $\widetilde{\mathcal{O}}(T^{\frac{1}{3}}V_T^{\frac{2}{3}})$  rate in our bound exhibits the same rate as the minimax optimal dynamic regret bound for the squared loss function with the noisy feedback [48]. Since the squared loss function enjoys even stronger curvature than the logistic loss, our result is hard to be improved. To achieve this fast-rate result, the key was to show a "squared" formulation  $\mathcal{O}\big(\max\{1, |I|V_I^2\}\big)$  of the dynamic regret for the base-learner on each interval  $I$  (Lemma 3 in Appendix D.2). [19] achieved this by a complicated analysis with the KKT condition to capture the structure of the comparators, while we greatly simplified the analysis by exploiting the structure that the comparator  $\theta_t^*$  is essentially the minimizers of expected functions  $L_t$  (hence avoiding analyzing the KKT condition). Our analysis is applicable to the case where the minimiers lie in the interior of the decision set and only requires the smoothness of the loss function, which can be of independent interest in online convex optimization.
Averaged Excess Risk Bound for Continuous Covariate Shift. After obtaining the density ratio estimator, we can train the predictive model by IwERM (2), leading to the following guarantee.
Theorem 3. Under the same condition as Theorem 2 and letting  $V_{T} = \sum_{t=2}^{T} \| \mathcal{D}_{t}(\mathbf{x}) - \mathcal{D}_{t-1}(\mathbf{x}) \|_{1}$ , running IwERM (2) with the estimated density ratio function  $\widehat{r}_{t}(\mathbf{x}) = \exp(-\phi(\mathbf{x})^{\top} \widehat{\theta}_{t})$  yields
$$
\mathfrak {R} _ {T} \left(\left\{\widehat {\mathbf {w}} _ {t} \right\} _ {t = 1} ^ {T}\right) \leq \widetilde {\mathcal {O}} \left(N _ {0} ^ {- \frac {1}{2}} + \max  \left\{T ^ {- \frac {1}{3}} V _ {T} ^ {\frac {1}{3}}, T ^ {- \frac {1}{2}} \right\}\right).
$$
Theorem 3 shows that our learned predictor  $\widehat{\mathbf{w}}_t$  adapts to the environment with a converged average excess risk when compared with the per-round best predictor  $\mathbf{w}_t^*$ . By the discussion below Proposition 1, the  $\widetilde{\mathcal{O}}(N_0^{-1/2})$  generalization error over the offline data  $S_0$  is hard to improve. We focus on the error in the online learning part. When the environment is nearly-stationary, i.e.,  $V_T \leq \mathcal{O}(T^{-1/2})$ , our bound implies an  $\widetilde{\mathcal{O}}(N_0^{-1/2} + T^{-1/2})$  average excess risk, matching the same rate for the one-step adaptation [49], even if the unlabeled data appear sequentially and the comparator  $\mathbf{w}_t^*$  could change over time. When the environments shift quickly, the  $\mathcal{O}(T^{-1/3} V_T^{1/3})$  rate still exhibits a diminishing error for covariate shift adaptation once  $V_T = o(T)$ . Indeed, our result has the same rate as that for continuous label shift [18] (with a slightly different definition of  $V_T$  though). In Appendix D.6, we further provide evidence to show that the  $\mathcal{O}(\max\{T^{-1/3} V_T^{1/3}, T^{-1/2}\})$  rate can hardly be improved, even if one can receive labels of the testing stream after prediction.
Discussion on Assumptions. We end this section by a discussion on possible future directions to relax the linear model assumption for DRE and covariate shift assumption. Since our analysis is based on the online convex optimization framework, we employed the (generalized) linear model for DRE to ensure the convexity. To go beyond the linear model while still having theoretical guarantees, one might extend the online ensemble framework to learn within the Reproducing Kernel Hilbert Space, leveraging advances in online kernel learning [50]. As for the step towards handling the general distribution shift, it is possible to consider the sparse joint shift model [51] where both covariate and label distribution could shift. Besides, studying how to handle the joint distribution shift with few online labeled data also presents an interesting future direction. Our research on the covariate shift might serve as a basic step towards addressing more complex real-world distribution shifts.
Setup. We generate continuous covariate shift by combining two fixed distributions with a time-varying mixture proportion  $\alpha_{t} \in [0,1]$ . Specifically, given two fixed distributions  $\mathcal{D}'(\mathbf{x})$  and  $\mathcal{D}''(\mathbf{x})$ , we generate samples from  $\mathcal{D}_t(\mathbf{x}) = (1 - \alpha_t)\mathcal{D}'(\mathbf{x}) + \alpha_t\mathcal{D}''(\mathbf{x})$  at each round. The mixture proportion  $\alpha_{t}$  shifts in four patterns: in Sin Shift and Squ Shift,  $\alpha_{t}$  changes periodically, following sine and square waves; in Lin Shift, the environment changes slowly from  $\alpha_{1} = 1$  to  $\alpha_{T} = 0$  linearly over  $T$  rounds, while in Ber Shift, the proportion  $\alpha_{t}$  flips quickly between 0 and 1 with a certain probability. For parameterization in the Accous implementation, we set  $R$  by directly calculating the data norm
Figure 2: Performance comparison on the four kinds of shifts.
Figure 3: Weight  $(\%)$  heatmap of base-learners in squared shift.
Figure 4: Average error and estimator loss in squared shift.
and set  $S = d / 2$  for all experiments. We repeat all experiments five times and evaluate the algorithms by the average classification error over the unlabeled test stream for 10,000 rounds.
Contenders. We compare our method with six algorithms, which can be divided into three groups. The first is a baseline approach that predicts directly using the model trained on initial offline data (FIX). The second group consists of the one-step covariate shift adaptation methods that do not reuse historical information: DANN [52] handles the shifts by learning an invariant feature space, IW-KMM [26], IW-KEIEP [25] and IW-uLSIF [28] equip the one-step IwERM method (2) with different density ratio estimators. The third is OLRE, which serves as an ablation study for our Accous algorithm. The OLRE algorithm estimates the density ratio by running an ONS (8) with all historical data and then performs the IwERM at each round. All algorithm parameters are set by their default.
In the following, we focus on the results of empirical studies. Detailed configurations for the datasets, simulated shifts, and setups for the Accous algorithm and contenders can be found in Appendix A.
5.1 Illustrations on Synthetic Data
Average Classification Error Comparison. We summarize the comparison results on synthetic data with different types of covariate shifts with all contenders in Figure 2. The Accous algorithm outperforms almost all other methods in the four shift patterns. We observe that the offline model FIX performs poorly compared to the online methods. The best result of DANN, IW-KMM, IW-KLIEP and IW-uLSIF is called One-Step for comparison. By reusing historical data, the Accous algorithm achieves a lower average classification error compared to these One-Step competitors that use only one round of online data. The Accous also outperforms OLRE, especially when the environments change relatively quickly (Squ, Sin, and Ber). These results justify that performing density ratio estimation with either one round of online data or all historical data is inappropriate for non-stationary environments. Moreover, when the number of unlabeled data per round changes from  $N_{t} = 5$  to  $N_{t} = 1$ , the one-step methods suffer severe performance degradation. In contrast, our Accous algorithm still performs relatively well, highlighting the need for selective reuse of historical data.
Effectiveness of Online Density Ratio Estimation. We then take a closer look at the core component of our algorithm, the online density ratio estimator, in the Squ shift where the distribution of the online data shifts every  $M$  rounds. Figure 3 shows the weight assignment for base-learners with different interval lengths, averaged over their active period. The result shows that our meta-learner successfully assigns the largest weight to the base-learner whose interval length matches the switching period  $M$ , which ensures that the right amount of historical data is reused. Figure 4 also shows the average error of Accous and OLRE over 1,000 iterations. The results show that Accous (red line) can quickly adapt to new distributions after the covariate shift occurs. In contrast, OLRE (blue line) struggles because it uses all the historical data generated from different distributions. Furthermore, we denote the loss of the density ratio estimator  $\widehat{L}_t(\widehat{\theta}_t)$  by the green line, which is quickly minimized after the covariate shift occurs. The similar tendency between the average error (red line) and the loss of the estimator (green line) confirms our theoretical finding in Theorem 1: we can minimize the excess risk of the predictor by minimizing the dynamic regret of the density ratio estimator.
5.2 Comparison on Real-world Data
Performance Comparison on Benchmarks. In Table 1 and Table 2, we present the results of the average classification error comparison with the contenders on four benchmark datasets. Our
Table 1: Average classification error (\%) of different algorithms on various real-world datasets with Lin and Ber shifts. We report the mean accuracy and standard deviation over five runs. The best algorithms are emphasized in bold. The number of data in each round is set as  $N_{t} = 5$ .
Lin	Ber
FIX	DANN	KMM	KLIEP	uLSIF	LR	OLRE	Accous	FIX	DANN	KMM	KLIEP	uLSIF	LR	OLRE	Accous
Diabetes	39.55	46.88	43.67	37.85	38.32	39.17	37.40	35.92	40.90	49.03	46.80	42.98	43.22	40.28	42.05	39.14
±6.58	±7.09	±6.23	±6.15	±6.00	±7.10	±6.56	±6.44	±6.90	±6.54	±6.67	±6.52	±6.78	±6.25	±6.22	±4.51
Breast	10.25	16.03	8.26	6.90	11.07	7.33	3.59	4.60	11.68	15.34	10.12	7.13	9.01	8.14	5.27	3.89
±3.00	±4.79	±0.31	±0.77	±2.34	±1.59	±0.25	±0.16	±4.16	±5.23	±0.60	±1.30	±1.91	±2.02	±0.44	±0.37
MNIST-SVHN	8.19	13.37	6.24	7.52	6.38	6.81	6.65	5.93	11.33	16.54	10.81	11.50	12.19	11.18	13.42	10.31
±1.39	±0.97	±1.52	±0.33	±0.76	±0.84	±0.93	±0.72	±0.83	±1.10	±0.63	±0.97	±1.20	±0.50	±0.49	±0.51
CIFAR10-CNIC10	29.24	31.59	29.60	29.33	29.18	29.75	28.17	27.80	33.16	42.50	33.49	31.44	32.98	33.03	31.22	30.69
±0.51	±2.18	±1.43	±0.97	±1.50	±0.48	±1.24	±0.83	±0.94	±1.29	±0.83	±1.57	±0.67	±1.26	±1.01	±0.62