SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Consider the first example. The minimizer of  $R(f)$  can be any Bayes-optimal classifier, i.e., any  $f$  such that  $f(\cdot, +) > 0$  and  $f(\cdot, -) < 0$ . Next, on the top-left square, we have  $p_{\mathrm{te}}(\pmb{x}, 1) = 1/4$ ,  $p_{\mathrm{te}}(\pmb{x}, 0) = 0$ ,  $p_{\mathrm{tr}}(\pmb{x}, 1) = 1/2$ , and  $p_{\mathrm{tr}}(\pmb{x}, 0) = 0$ , and thus  $w^{*}(\pmb{x}, y) = 1/2$ . Likewise, on the bottom-left square, we have  $w^{*}(\pmb{x}, y) = 1/2$ . As a result,  $J(f) = \frac{1}{2}\mathbb{E}_{p_{\mathrm{tr}}(\pmb{x}, y)}[\ell(\pmb{f}(\pmb{x}), y)]$ , meaning that the minimizer of  $J(f)$  can be any Bayes-optimal classifier on  $S_{\mathrm{tr}}$ , i.e., any  $f$  such that  $f(-, +) > 0$  and  $f(-, -) < 0$ . The simplest manner for  $f$  to transfer its knowledge from  $p_{\mathrm{tr}}$  to  $p_{\mathrm{te}}$  is to have a linear decision boundary and extend it to  $S_{\mathrm{te}} \setminus S_{\mathrm{tr}}$ , so that  $f(\cdot, +) > 0$  and  $f(\cdot, -) < 0$  on  $S_{\mathrm{te}}$ . We can see that the set of minimizers is shared and thus IW is classifier-consistent.
For any possible implementation, as long as it honestly implements the objective of IW, the quality of the objective will be reflected in the quality of the implementation.
As the classifier  $\pmb{f}$  can only access the information on  $S_{\mathrm{tr}}$  but must predict the class label on the whole  $S_{\mathrm{te}}$ , we assume that  $\pmb{f}$  would transfer its knowledge about  $p_{\mathrm{tr}}(\pmb{x},y)$  to  $p_{\mathrm{te}}(\pmb{x},y)$  in the simplest manner. Furthermore, we simplify  $\pmb{f}(\pmb{x})$  into  $f(x^{(1)},x^{(2)})$ , since it is binary classification where  $\pmb{x} \in \mathbb{R}^2$ .
The cross-entropy loss is confidence-calibrated and thus classification-calibrated (Sugiyama et al., 2022).
Consider the second example. The minimizer of  $J(f)$  is still the same while the minimizer of  $R(f)$  significantly changes to any  $f$  such that  $f(-, +) > 0$ ,  $f(-, -) < 0$ ,  $f(+, +) < 0$ , and  $f(+, -) > 0$ . This non-linear decision boundary is a checkerboard where any two adjacent squares have opposite predictions. It is easy to see that IW is classifier-inconsistent and its test accuracy is 0.5. For binary classification with balanced classes, this accuracy is as poor as random guessing.
3 Generalized importance weighting (GIW)
We have seen two examples where IW is as good/bad as possible in case (iii). In practice, we cannot rely on the luck and hope that IW would work. In this section, we propose generalized importance weighting (GIW), which is still IW in cases (i) and (ii) and is better than IW in cases (iii) and (iv).
3.1 Expected objective of GIW
The key idea of GIW is to split the test support  $S_{\mathrm{te}}$  into the in-training (IT) part  $S_{\mathrm{te}} \cap S_{\mathrm{tr}}$  and the out-of-training (OOT) part  $S_{\mathrm{te}} \setminus S_{\mathrm{tr}}$ . More specifically, we introduce a third random variable, the support-splitting variable  $s \in \{0,1\}$ , such that  $s$  takes 1 on  $S_{\mathrm{tr}}$  and 0 on  $S_{\mathrm{te}} \setminus S_{\mathrm{tr}}$ . As a result, the underlying joint density  $p(\boldsymbol{x},y,s)$  can be defined by  $p_{\mathrm{te}}(\boldsymbol{x},y)$  as<sup>10</sup>
$$
p (\boldsymbol {x}, y, s) = \left\{ \begin{array}{l l} p _ {\mathrm {t e}} (\boldsymbol {x}, y) & \text {i f} (\boldsymbol {x}, y) \in \mathcal {S} _ {\mathrm {t r}} \text {a n d} s = 1, \text {o r} (\boldsymbol {x}, y) \in \mathcal {S} _ {\mathrm {t e}} \backslash \mathcal {S} _ {\mathrm {t r}} \text {a n d} s = 0, \\ 0 & \text {i f} (\boldsymbol {x}, y) \in \mathcal {S} _ {\mathrm {t r}} \text {a n d} s = 0, \text {o r} (\boldsymbol {x}, y) \in \mathcal {S} _ {\mathrm {t e}} \backslash \mathcal {S} _ {\mathrm {t r}} \text {a n d} s = 1. \end{array} \right. \tag {4}
$$
Let  $\alpha = p(s = 1)$ . Then, the expected objective of GIW is defined as
$$
J _ {\mathrm {G}} (\boldsymbol {f}) = \alpha \mathbb {E} _ {p _ {\operatorname {t r}} (\boldsymbol {x}, y)} [ w ^ {*} (\boldsymbol {x}, y) \ell (\boldsymbol {f} (\boldsymbol {x}), y) ] + (1 - \alpha) \mathbb {E} _ {p (\boldsymbol {x}, y | s = 0)} [ \ell (\boldsymbol {f} (\boldsymbol {x}), y) ]. \tag {5}
$$
The corresponding empirical version  $\hat{J}_{\mathrm{G}}(\pmb {f})$  will be derived in the next subsection. Before proceeding to the empirical objective of GIW, we establish risk consistency of GIW.
Theorem 4. GIW is always risk-consistent for distribution shift problems.
Proof. Let us work on the first term of  $J_{\mathrm{G}}(\pmb {f})$  in (5). When  $(\pmb {x},y)\in S_{\mathrm{tr}}$
$$
\alpha w ^ {*} (\boldsymbol {x}, y) p _ {\mathrm {t r}} (\boldsymbol {x}, y) = \alpha p _ {\mathrm {t e}} (\boldsymbol {x}, y) = p (s = 1) p (\boldsymbol {x}, y \mid s = 1) = p (\boldsymbol {x}, y, s = 1),
$$
where  $p_{\mathrm{te}}(\pmb{x},y) = p(\pmb{x},y \mid s = 1)$  given  $(\pmb{x},y) \in S_{\mathrm{tr}}$  according to (4). Since  $p(\pmb{x},y,s = 1) = 0$  on  $S_{\mathrm{te}} \setminus S_{\mathrm{tr}}$ , we have
$$
\alpha \mathbb {E} _ {p _ {\mathrm {t r}} (\boldsymbol {x}, y)} [ w ^ {*} (\boldsymbol {x}, y) \ell (\boldsymbol {f} (\boldsymbol {x}), y) ] = \sum_ {y = 1} ^ {C} \int_ {\{\boldsymbol {x}: (\boldsymbol {x}, y) \in \mathcal {S} _ {\mathrm {t e}} \}} \ell (\boldsymbol {f} (\boldsymbol {x}), y) p (\boldsymbol {x}, y, s = 1) \mathrm {d} \boldsymbol {x}. \tag {6}
$$
Next, for the second term of  $J_{\mathrm{G}}(\pmb {f})$ , since  $(1 - \alpha)p(\pmb {x},y\mid s = 0) = p(\pmb {x},y,s = 0)$ , we have
$$
(1 - \alpha) \mathbb {E} _ {p (\boldsymbol {x}, y | s = 0)} [ \ell (\boldsymbol {f} (\boldsymbol {x}), y) ] = \sum_ {y = 1} ^ {C} \int_ {\{\boldsymbol {x}: (\boldsymbol {x}, y) \in S _ {\mathrm {t e}} \}} \ell (\boldsymbol {f} (\boldsymbol {x}), y) p (\boldsymbol {x}, y, s = 0) \mathrm {d} \boldsymbol {x}. \tag {7}
$$
Note that  $p(\pmb{x}, y, s = 1) + p(\pmb{x}, y, s = 0) = p_{\mathrm{te}}(\pmb{x}, y)$  according to (4). By adding (6) and (7), we can obtain that  $J_{\mathrm{G}}(\pmb{f}) = R(\pmb{f})$ . This conclusion holds in all the four cases.
Theorem 4 is the main theorem of this paper. It implies that the objective of GIW can always be a good objective. Recall that IW is also risk-consistent in cases (i) and (ii), and it is interesting to see how IW and GIW are connected. By definition, given fixed  $p_{\mathrm{tr}}(\boldsymbol{x}, \boldsymbol{y})$  and  $p_{\mathrm{te}}(\boldsymbol{x}, \boldsymbol{y})$ , if there exists a risk-consistent objective, it is unique. Indeed, in cases (i) and (ii), GIW is reduced to IW, simply due to that  $\alpha = 1$  and  $J_{\mathrm{G}}(\boldsymbol{f}) = J(\boldsymbol{f})$  for any  $\boldsymbol{f}$ .
3.2 Empirical objective and practical implementation of GIW
Approximating  $J_{\mathrm{G}}(\pmb {f})$  in (5) is more involved than approximating  $J(\pmb {f})$  in (2). Following (3), we need an empirical version  $\widehat{w} (\pmb {x},y)$ , and we need further to split the validation data  $\mathcal{D}_{\mathrm{v}}$  into two sets and estimate  $\alpha$ . Obviously, how to accurately split the validation data is the most challenging part. After splitting  $\mathcal{D}_{\mathrm{v}}$  and obtaining an estimate  $\widehat{\alpha}$ , the empirical objective will have two terms, where
10Here,  $p(\pmb {x},y,s)$  accepts  $(\pmb {x},\pmb {y})\in S_{\mathrm{te}}\cup S_{\mathrm{tr}}$  rather than  $(\pmb {x},\pmb {y})\in S_{\mathrm{te}}$  , and  $s$  actually splits  $\mathcal{S}_{\mathrm{te}}\cup \mathcal{S}_{\mathrm{tr}}$  into  $\mathcal{S}_{\mathrm{tr}}$  and  $\mathcal{S}_{\mathrm{te}}\setminus \mathcal{S}_{\mathrm{tr}}$  . This is exactly what we want, since  $p_{\mathrm{te}}(\pmb {x},\pmb {y}) = 0$  outside  $\mathcal{S}_{\mathrm{te}}$
Algorithm 1 Generalized importance weighting.
Require: model  $f_{\theta}$  parameterized by  $\theta$  training data set  $\mathcal{D}_{\mathrm{tr}}$  2: for  $t = 1$  to  $T$  do validation data set  $\mathcal{D}_{\mathrm{v}}$  3: sample  $S_{\mathrm{tr}}$  of size  $m$  from  $\mathcal{D}_{\mathrm{tr}}$  batch sizes  $m,n_{1}$  ,and  $n_2$  4: sample  $S_{\mathrm{v1}}$  of size  $n_1$  from  $\mathcal{D}_{\mathrm{v1}}$  number of iterations  $T$  5: sample  $S_{\mathrm{v2}}$  of size  $n_2$  from  $\mathcal{D}_{\mathrm{v2}}$  6: forward the instances of  $S_{\mathrm{tr}}$  &  $S_{\mathrm{v1}}$    
2: pretrain  $\pmb {f}_{\theta}$  on  $\mathcal{D}_{\mathrm{tr}}$  7: compute the loss values as  $L_{\mathrm{tr}}$  &  $L_{\mathrm{v1}}$    
3: forward the instances of  $\mathcal{D}_{\mathrm{tr}}$  &  $\mathcal{D}_{\mathrm{v}}$  8: match the distributions of  $L_{\mathrm{tr}}$  and  $L_{\mathrm{v1}}$  to obtain an estimated  $\widehat{w} (\boldsymbol {x},y)$    
5: train an O-SVM on  $Z_{\mathrm{tr}}$  as  $g(z)$  9: weight  $L_{\mathrm{tr}}$  with the estimated  $\widehat{w} (\boldsymbol {x},y)$    
6: compute  $g(z)$  for every  $z$  in  $Z_{\mathrm{v}}$  10: forward the instances of  $S_{\mathrm{v2}}$    
7: partition  $\mathcal{D}_{\mathrm{v}}$  into  $\mathcal{D}_{\mathrm{v1}}$  &  $\mathcal{D}_{\mathrm{v2}}$  11: backward  $\widehat{\mathcal{J}}_{\mathrm{G}}(\pmb {f}_{\theta})$  , and update  $\theta$    
8: estimate  $\hat{\alpha} = |\mathcal{D}_{\mathrm{v1}}| / |\mathcal{D}_{\mathrm{v}}|$  12: end for   
9: return  $\mathcal{D}_{\mathrm{v1}},\mathcal{D}_{\mathrm{v2}},\hat{\alpha}$  13: return the final  $\pmb {f}_{\theta}$    
10: end procedure 14: end procedure
the first term can be handled by any IW algorithm given training data and IT validation data, and the second term just involves OOT validation data.
To split  $\mathcal{D}_{\mathrm{v}}$  and estimate  $\alpha$ , we employ the one-class support vector machine (O-SVM) (Scholkopf et al., 1999). Firstly, we pretrain a deep network for classification on the training data  $\mathcal{D}_{\mathrm{tr}}$  a little bit and obtain a feature extractor from the pretrained deep network. Secondly, we apply the feature extractor on the instances in  $\mathcal{D}_{\mathrm{tr}}$  and train an O-SVM based on the latent representation of these instances, giving us a score function  $g(z)$  that could predict whether  $p_{\mathrm{tr}}(\boldsymbol{x}) > 0$  or not, where  $z$  is the latent representation of  $\boldsymbol{x}$ . Thirdly, we apply the feature extractor on the instances in  $\mathcal{D}_{\mathrm{v}}$  and then employ  $g(z)$  to obtain the IT validation data  $\mathcal{D}_{\mathrm{v1}} = \{(x_i^{\mathrm{v1}}, y_i^{\mathrm{v1}})\}_{i=1}^{n_{\mathrm{v1}}}$  and the OOT validation data  $\mathcal{D}_{\mathrm{v2}} = \{(x_i^{\mathrm{v2}}, y_i^{\mathrm{v2}})\}_{i=1}^{n_{\mathrm{v2}}}$ . Finally,  $\alpha$  can be naturally estimated as  $\hat{\alpha} = n_{\mathrm{v1}} / n_{\mathrm{v}}$ .
We have two comments on the split of  $\mathcal{D}_{\mathrm{v}}$ . The O-SVM  $g(z)$  predicts whether  $p_{\mathrm{tr}}(x) > 0$  or not rather than whether  $p_{\mathrm{tr}}(x,y) > 0$  or not. This is because the  $x$ -support change is often sufficiently informative in practice: when the  $x$ -support changes, O-SVM can detect it; when the  $(x,y)$ -support changes without changing the  $x$ -support, it will be very difficult to train an O-SVM based on the loss-value representation of  $(x,y)$  to detect it, but such changes are very rare. The other comment is about the choice of the O-SVM. While there are more advanced one-class classification methods (Hido et al., 2011; Zaheer et al., 2020; Hu et al., 2020; Goldwasser et al., 2020) (see Perera et al. (2021) for a survey), the O-SVM is already good enough for the purpose (see Appendix C.1).
Subsequently,  $\mathcal{D}_{\mathrm{v1}}$  can be viewed as being drawn from  $p(\boldsymbol{x}, y \mid s = 1)$ , and  $\mathcal{D}_{\mathrm{v2}}$  can be viewed as being drawn from  $p(\boldsymbol{x}, y \mid s = 0)$ . Based on  $\mathcal{D}_{\mathrm{tr}}$  and  $\mathcal{D}_{\mathrm{v1}}$ , we can obtain either  $\hat{w}(\boldsymbol{x}, y)$  or  $\hat{w}_i$  for each  $(x_i^{\mathrm{tr}}, y_i^{\mathrm{tr}})$  by IW. IW has no problem here since the split of  $\mathcal{D}_{\mathrm{v}}$  can reduce case (iii) to case (i) and case (iv) to case (ii). In the implementation, we employ DIW (Fang et al., 2020) because it is friendly to deep learning and it is a state-of-the-art IW method. Finally, the empirical objective of GIW can be expressed as
$$
\widehat {J} _ {\mathrm {G}} (\boldsymbol {f}) = \frac {n _ {\mathrm {v} 1}}{n _ {\mathrm {v}} n _ {\mathrm {t r}}} \sum_ {i = 1} ^ {n _ {\mathrm {t r}}} \widehat {w} \left(\boldsymbol {x} _ {i} ^ {\mathrm {t r}}, y _ {i} ^ {\mathrm {t r}}\right) \ell \left(\boldsymbol {f} \left(\boldsymbol {x} _ {i} ^ {\mathrm {t r}}\right), y _ {i} ^ {\mathrm {t r}}\right) + \frac {1}{n _ {\mathrm {v}}} \sum_ {j = 1} ^ {n _ {\mathrm {v} 2}} \ell \left(\boldsymbol {f} \left(\boldsymbol {x} _ {j} ^ {\mathrm {v} 2}\right), y _ {j} ^ {\mathrm {v} 2}\right), \tag {8}
$$
where the two expectations in  $J_{\mathrm{G}}(\pmb {f})$  are approximated separately with  $\mathcal{D}_{\mathrm{tr}}$  and  $\mathcal{D}_{\mathrm{v2}}$  11
The practical implementation of GIW is presented in Algorithm 1. Here, we adopt the hidden-layer-output representation for O-SVM in VALDATA/SPLIT and the loss-value representation for DIW in
<sup>11</sup>In GIW, although  $J_{\mathrm{G}}(\pmb {f}) = R(\pmb {f})$ ,  $\widehat{J}_{\mathrm{G}}(\pmb {f})$  is not an unbiased estimator of  $J_{\mathrm{G}}(\pmb {f})$ , exactly the same as what happened in IW. Nevertheless,  $\widehat{J}_{\mathrm{G}}(\pmb {f})$  can still be statistically consistent with  $J_{\mathrm{G}}(\pmb {f})$  under mild conditions if  $\widehat{w} (\pmb {x},y)$  is consistent with  $w^{*}(\pmb {x},y)$ . Specifically, though the outer weighted classification is an optimization problem, the inner weight estimation is an estimation problem. The statistical consistency of  $\widehat{w} (\pmb {x},y)$  requires zero approximation error and thus non-parametric estimation is preferred. For kernel-based IW methods such as Huang et al. (2007) and Kanamori et al. (2009), it holds that as  $n_{\mathrm{tr}},n_{\mathrm{v}}\to \infty$ ,  $\widehat{w} (\pmb {x},y)\rightarrow w^{*}(\pmb {x},y)$  under mild conditions. If so, we can establish statistical consistency of  $\widehat{J}_{\mathrm{G}}(\pmb {f})$  with  $J_{\mathrm{G}}(\pmb {f})$ . If we further assume that the function class of  $\pmb{f}$  has a bounded complexity and  $\ell$  is bounded and Lipschitz continuous, we can establish statistical consistency of  $R(\widehat{\pmb{f}})$  with  $R(\pmb{f}^{*})$ , where  $\widehat{\pmb{f}} (\pmb {x})$  and  $f^{*}(\pmb {x})$  are the minimizers of  $\widehat{J}_{\mathrm{G}}(\pmb {f})$  and  $R(\pmb {f})$ .
Table 1: Specification of benchmark datasets, tasks, distribution shifts, and models.
Dataset	Task	Training data	Test data	Model
MNIST	odd and even digits	4 digits (0-3)	10 digits (0-9)*	LeNet-5
Color-MNIST	10 digits	digits in red	digits in red/blue/green	LeNet-5
CIFAR-20	20 superclasses	2 classes per superclass	5 classes per superclass	ResNet-18
See LeCun et al. (1998) for MNIST and Krizhevsky and Hinton (2009) for CIFAR-20. Color-MNIST is modified from MNIST. The model is a modified LeNet-5 (LeCun et al., 1998) or ResNet-18 (He et al., 2016). Please find in Appendix B.1 the details. *All setups in the table are for case (iii); for MNIST in case (iv), the test data consist of 8 digits (2-9).
MNIST, case (iii)
MNIST, case (iv)
Color-MNIST, case (iii)
CIFAR-20, case (iii)
Figure 3: Comparisons with IW-like and DA baselines under support shift (5 trails).
MODELTRAIN. This algorithm design is convenient for both O-SVM and DIW; the hidden-layer-output representation for DIW has been tried and can be found in Section 4.3.
In this section, we empirically evaluate GIW and compare it with baseline methods. To see how effective it is in cases (iii) and (iv), we designed two distribution shift (DS) patterns. In the first pattern, DS comes solely from the mismatch between the training and test supports, and we call it support shift (SS). Under SS, it holds that  $w^{*}(\boldsymbol{x},y)$  equals  $\alpha$  in case (iii) and 0 or another constant in case (iv), simply due to renormalization after imposing SS. Hence, the challenge is to accurately split  $D_{\mathrm{v}}$ . In the second pattern, there is some genuine DS (e.g., label noise or class-prior shift) on top of SS, and we call it support-distribution shift. Since  $w^{*}(\boldsymbol{x},y)$  is no longer a constant, we face the challenge to accurately estimate  $w^{*}(\boldsymbol{x},y)$ . Additionally, we conducted an ablation study to better understand the behavior of GIW. Detailed setups and more results are given in Appendices B & C.
The baseline methods involved in our experiments are as follows.
- Val-only: using only  $\mathcal{D}_{\mathrm{v}}$  to train the model from scratch.
- Pretrain-val: first pretraining on  $\mathcal{D}_{\mathrm{tr}}$  and then training on  $\mathcal{D}_{\mathrm{v}}$ .
- Reweight: learning to reweight examples (Ren et al., 2018).
- MW-Net: meta-weight-net (Shu et al., 2019), a parametric version of Reweight.
-  $DIW$ : dynamic importance weighting (Fang et al., 2020).
- R-DIW: DIW where IW is done with relative density-ratio estimation (Yamada et al., 2011).
- CCSA: classification and contrastive semantic alignment (Motiian et al., 2017).
- DANN: domain-adversarial neural network (Ganin et al., 2016).
4.1 Experiments under support shift
We first conducted experiments under support shift on benchmark datasets. The setups are summarized in Table 1. For MNIST, our task was to classify odd and even digits, where the training set has only 4 digits (0-3), while the test set has 10 digits (0-9) in case (iii) and 8 digits (2-9) in case (iv). For Color-MNIST, our task was to classify 10 digits; the dataset was modified from MNIST in such a way that the digits in the training set are colored in red while the digits in the test/validation set are colored
<sup>12</sup>Our implementation of GIW is available at https://github.com/TongtongFANG/GIW.
Figure 4: Visualizations of the learned convolution kernels on Color-MNIST under support shift.
Figure 5: Comparisons with IW-like baselines under support-distribution shift (5 trails).
in red/green/blue evenly. For CIFAR-100, our task was to classify the 20 predefined superclasses and thus we call it CIFAR-20; the training set contains data from 2 out of the 5 classes for each superclass while the test set contains all classes. For validation set, we sampled 2 data points per test digit for MNIST and Color-MNIST, and 10 data points per class for CIFAR-100.
Figure 3 shows the results on MNIST, Color-MNIST, and CIFAR-20 under support shift $^{13}$ , where GIW generally outperforms IW-like and domain adaptation (DA) baselines. We also confirmed that  $\alpha$  in (5) is accurately estimated in Appendix C.1. To further investigate how GIW works, we visualized the learned convolution kernels (i.e., weights) for Color-MNIST experiments in Figure 4, where the more observed color represents the larger weights learned on that color channel. Only GIW recovers the weights of all color channels while learning useful features, however, other methods fail to do so.
4.2 Experiments under support-distribution shift
We further imposed additional distribution shift, i.e., adding label noise or class-prior shift, on top of the support shift following the same setup in Table 1. Here we only show the results in case (iii) and defer the results in case (iv) to Appendix C.4 due to the space limitation.
Label-noise experiments In addition to the support shift, we imposed label noise by randomly flipping a label to other classes with an equal probability, i.e., the noise rate. The noise rates are set as  $\{0.2, 0.4\}$  and the corresponding experimental results are shown in Figure 5 and 6. We can see that compared with baselines, GIW performs better and tends to be robust to noisy labels.
Class-prior-shift experiments On top of the support shift, we imposed class-prior shift by reducing the number of training data in half of the classes to make them minority classes (as opposed to majority classes). The sample size ratio per class between the majority and minority classes is defined as  $\rho$ , chosen from  $\{10, 100\}$ . To fully use the data from the minority class, we did not split the validation data in class-prior-shift experiments and used all validation data in optimizing the two
<sup>13</sup>It is difficult to have the same number of mini-batches per epoch for the training and validation data. To this end, we adopt the definition of an epoch as a single loop over the training data, not the validation data.
Figure 6: Comparisons with DA baselines under support-distribution shift (5 trails).
(a) Validation split error
Figure 7: Impact of different validation data splits, sample sizes, and data representations.
(b) Training data size
(c) Validation data size
(d)-F/-L representation
terms in (8). In Figure 5 and 6, we can see that GIW consistently performs better than the baselines. Note that though domain adaptation (DA) baselines (i.e., CCSA & DANN) may achieve comparable performance to GIW under the support shift in Figure 3, their effectiveness declines significantly when confronting additional distribution shifts (e.g., label noise or class-prior shift) in Figure 6.
Finally, we performed an ablation study on MNIST under 0.2 label noise. Figure 7(a) and Table 2 in Appendix C.2 present the negative impact of validation data split errors by randomly flipping the IT/OOT data into the OOT/IT part. We can see that GIW is reasonably robust to split errors, and flipping OOT to IT is more problematic than the other direction since it makes the empirical risk estimator of GIW more similar to that of the standard IW. In Figure 7(b), the performance of all methods remains consistent across different values of  $n_{\mathrm{tr}} \in \{10000, 12500, 15000, 17500, 20000\}$ . In Figure 7(c), GIW performs better with more validation data, e.g., when  $n_{\mathrm{v}}$  increases from 10 to 20. Moreover, we compare the loss-value (-L) with the hidden-layer-output (-F) representation of data used in the DIW and GIW methods. Figure 7(d) shows GIW-L outperforms others.
We characterized distribution shift into four cases according to support shift to gain a deeper understanding of IW. Consequently, we found that IW is provably good in two cases but provably poor in the other two cases. Then, we proposed GIW which is a strict generalization of IW and is provably favorable in all the four cases. GIW is safer to be used in practice as it is difficult to know in which case the problem to be solved is. That said, there are still some potential limitations and thus future directions to work on: GIW requires exactly test-distributed validation data, which is restrictive; it requires not very small validation data, which is demanding; a small amount of OOT validation data joins training directly and is used for validation simultaneously, which might lead to imperceptible overfitting as well as overoptimism about the OOT part of the test support.
TF was supported by JSPS KAKENHI Grant Number 23KJ0438 and the Institute for AI and Beyond, UTokyo. NL was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy – EXC number 2064/1 – Project number 390727645. MS was supported by the Institute for AI and Beyond, UTokyo.