Title: Generalizing Importance Weighting to A Universal Solver for Distribution Shift Problems
Venue: NeurIPS 2023
Accepted: yes
Keywords: importance weighting

distribution shift

deep learning
Abstract: Distribution shift (DS) may have two levels: the distribution itself changes, and the support (i.e., the set where the probability density is non-zero) also changes. When considering the support change between the training and test distributions, there can be four cases: (i) they exactly match; (ii) the training support is wider (and thus covers the test support); (iii) the test support is wider; (iv) they partially overlap. Existing methods are good at cases (i) and (ii), while cases (iii) and (iv) are more common nowadays but still under-explored. In this paper, we generalize importance weighting (IW), a golden solver for cases (i) and (ii), to a universal solver for all cases. Specifically, we first investigate why IW might fail in cases (iii) and (iv); based on the findings, we propose generalized IW (GIW) that could handle cases (iii) and (iv) and would reduce to IW in cases (i) and (ii). In GIW, the test support is split into an in-training (IT) part and an out-of-training (OOT) part, and the expected risk is decomposed into a weighted classification term over the IT part and a standard classification term over the OOT part, which guarantees the risk consistency of GIW. Then, the implementation of GIW consists of three components: (a) the split of validation data is carried out by the one-class support vector machine, (b) the first term of the empirical risk can be handled by any IW algorithm given training data and IT validation data, and (c) the second term just involves OOT validation data. Experiments demonstrate that GIW is a universal solver for DS problems, outperforming IW methods in cases (iii) and (iv).

Generalizing Importance Weighting to A Universal Solver for Distribution Shift Problems

Abstract

Distribution shift (DS) may have two levels: the distribution itself changes, and the support (i.e., the set where the probability density is non-zero) also changes. When considering the support change between the training and test distributions, there can be four cases: (i) they exactly match; (ii) the training support is wider (and thus covers the test support); (iii) the test support is wider; (iv) they partially overlap. Existing methods are good at cases (i) and (ii), while cases (iii) and (iv) are more common nowadays but still under-explored. In this paper, we generalize importance weighting (IW), a golden solver for cases (i) and (ii), to a universal solver for all cases. Specifically, we first investigate why IW might fail in cases (iii) and (iv); based on the findings, we propose generalized IW (GIW) that could handle cases (iii) and (iv) and would reduce to IW in cases (i) and (ii). In GIW, the test support is split into an in-training (IT) part and an out-of-training (OOT) part, and the expected risk is decomposed into a weighted classification term over the IT part and a standard classification term over the OOT part, which guarantees the risk consistency of GIW. Then, the implementation of GIW consists of three components: (a) the split of validation data is carried out by the one-class support vector machine, (b) the first term of the empirical risk can be handled by any IW algorithm given training data and IT validation data, and (c) the second term just involves OOT validation data. Experiments demonstrate that GIW is a universal solver for DS problems, outperforming IW methods in cases (iii) and (iv).

1 Introduction

Deep supervised classification has been successful where the training and test data should come from the same distribution (Goodfellow et al., 2016). When this assumption does not hold in practice, we suffer from distribution shift (DS) problems and the learned classifier may often generalize poorly (Quionero-Candela et al., 2009; Pan and Yang, 2009; Sugiyama and Kawanabe, 2012). Let x and y be the instance (i.e., input) and class-label (i.e., output) random variables. Then, DS means that the underlying joint density of the training data p tr ( x , y ) differs from that of the test data p te ( x , y ) .

̸

There are two levels in the DS research. At the first level, only the change of the data distribution is considered. With additional assumptions, DS can be reduced into covariate shift p tr ( x ) = p te ( x ) , class-prior shift p tr ( y ) = p te ( y ) , class-posterior shift p tr ( y | x ) = p te ( y | x ) , and class-conditional shift p tr ( x | y ) = p te ( x | y ) (Quionero-Candela et al., 2009). We focus on joint shift p tr ( x , y ) = p te ( x , y ) , as it is the most general and difficult case of DS. At the second level, the change of the support of the data distribution is also considered, where given any joint density p ( x , y ) , its support is defined as the set { ( x , y ) : p ( x , y ) > 0 } . More specifically, denote by S tr and S te the support of p tr ( x , y ) and p te ( x , y ) , respectively. When considering the relationship between S tr and S te , there can be four cases:

̸

(i) S tr and S te exactly match, i.e., S tr = S te ;

̸

̸

RIKEN, Japan

̸

train

test

(i)

train/test

(i)

(ii)

(iv)

train

test

test

train

̸

(ii) S tr is wider and covers S te , i.e., S tr ⊃ S te and S tr \ S te = ∅ ;

̸

(iv) S tr and S te partially overlap, i.e., S tr ∩ S te = ∅ , S tr \ S te = ∅ , and S te \ S tr = ∅ . 1

(iii) S te is wider and covers S tr , i.e., S tr ⊂ S te and S te \ S tr = ∅ ;

̸

̸

̸

The four cases are illustrated in Figure 1. We focus on cases (iii) and (iv), as they are more general and more difficult than cases (i) and (ii).

Problem setting Denote by X and Y the input and output domains, where Y = { 1 , . . . , C } for C -class classification problems. Let f : X → R C be a classifier (to be trained) and ℓ : R C ×Y → (0 , + ∞ ) be a loss function (for training f ). 2 Then, the risk is defined as follows (Vapnik, 1998):

where E [ · ] denotes the expectation. In the joint-shift problems, we are given a training set D tr = { ( x tr i , y tr i ) } n tr i =1 i.i.d. ∼ p tr ( x , y ) and a validation set D v = { ( x v i , y v i ) } n v i =1 i.i.d. ∼ p te ( x , y ) , where D tr is much bigger than D v , i.e., n tr ≫ n v . The goal is to reliably estimate the risk from D tr and D v and train f by minimizing the empirical risk, which should outperform training f from only D v .

Motivation Importance weighting (IW) has been a golden solver for DS problems (Sugiyama and Kawanabe, 2012), and there are many great off-the-shelf IW methods (Huang et al., 2007; Sugiyama et al., 2007a,b; Kanamori et al., 2009). Recently, dynamic IW (DIW) was proposed to make IW compatible with stochastic optimizers and thus it can be used for deep learning (Fang et al., 2020). However, all IW methods including DIW have assumed cases (i) and (ii)-in cases (iii) and (iv), IW methods become problematic. Specifically, as the importance weights are only used on S tr , even though they become ill-defined on S te \ S tr , IW itself is still well-defined. Nevertheless, in such a situation, the IW identity will become an inequality (i.e., Theorem 2), which means that what we minimize for training is no longer an approximation of the original risk R ( f ) and thus IW may lead to poor trained classifiers (i.e., Proposition 3). Moreover, some IW-like methods based on bilevel optimization share a similar issue with IW (Jiang et al., 2018; Ren et al., 2018; Shu et al., 2019), since f is only trained from D tr where D v is used to determine the importance weights on D tr . In fact, cases (iii) and (iv) are more common nowadays due to data-collection biases , but they are still under-explored. For example, a class has several subclasses, but not all subclasses are presented in D tr (see Figure 2). Therefore, we want to generalize IW to a universal solver for all the four cases.

Contributions Our contributions can be summarized as follows.

Firstly, we theoretically and empirically analyze when and why IW methods can succeed/may fail. We reveal that the objective of IW is good in cases (i) and (ii) and bad in cases (iii) and (iv).

Secondly, we propose generalized IW (GIW). In GIW, S te is split into an in-training (IT) part S te ∩ S tr and an out-of-training (OOT) part S te \ S tr , and its objective consists of a weighted classification term over the IT part and a standard classification term over the OOT part. GIW is justified as its objective is good in all the four cases and reduces to IW in cases (i) and (ii). Thus, GIW is a strict generalization of IW from the objective point of view, and GIW is safer to be used when we are not sure whether the problem to be solved is a good case or a bad case for IW. 3

Thirdly, we provide a practical implementation of GIW: (a) following the split of S te , D v is split into an IT set and an OOT set using the one-class support vector machine (Schölkopf et al., 1999);

̸

The data

Val-only

IW (DIW)

GIW

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

This is binary classification to distinguish red and blue synthetic data that are uniformly distributed in a 2-by-2 grid (consisting of 4 squares in different positions). The training distribution p tr ( x , y ) includes only the left 2 squares, and the test distribution p te ( x , y ) includes all the 4 squares; there are only 4 validation data, 1 in each square. 'Val-only' means using only the validation data to train the model, 'DIW' refers to Fang et al. (2020), and 'GIW' is the proposed method. The learned decision boundaries are plotted to compare those methods. In the top panel, IW and GIW perform well, but in the bottom panel, IW completely fails and GIW still performs well. The performance of Val-only is not satisfactory since there are too few validation data. More details and discussions about this experiment can be found in the last part of Section 2.

(b) the IT set, instead of the whole D v , is used for IW; and (c) the OOT set directly joins training together with D tr since no data in D tr comes from the OOT part.

Finally, we design and conduct extensive experiments that demonstrate the effectiveness of GIW in cases (iii) and (iv). The experiment design is also a major contribution since no experimental setup is available for reference to simulate case (iii) or (iv) on benchmark datasets.

Organization The analyses of IW are in Section 2, the proposal of GIW is in Section 3, and the experiments are in Section 4. Related work and additional experiments are in the appendices.

2 A deeper understanding of IW

First, we review the traditional importance weighting (IW) and its modern implementation dynamic importance weighting (DIW). Then, we analyze when and why IW methods can succeed/may fail.

A review of IW Let w ∗ ( x , y ) = p te ( x , y ) /p tr ( x , y ) , which is the ratio of the test density p te ( x , y ) over the training density p tr ( x , y ) , known as the importance function . Then, the expected objective of IW can be expressed as

In order to empirically approximate J ( f ) in (2), we need to have an empirical version ̂ w ( x , y ) of w ∗ ( x , y ) , so that the empirical objective of IW is 4

The original IW method is implemented in two steps: (I) weight estimation (WE) where ̂ w ( x , y ) is obtained and (II) weighted classification (WC) where ̂ J ( f ) is minimized. The first step relies on the training data D tr and the validation data D v , and it can be either estimating the two density functions separately and taking their ratio or directly estimating the density ratio (Sugiyama et al., 2012).

A review of DIW The aforementioned two-step approach is very nice when the classifier f is a simple model, but it has a serious issue when f is a deep model (Fang et al., 2020). Since WE is not equipped with representation learning, in order to boost its expressive power, we need an external feature extractor such as an internal representation learned by WC. As a result, we are trapped by a circular dependency : originally we need w ∗ to train f ; now we need a trained f to estimate w ∗ .

DIW (Fang et al., 2020) has been proposed to resolve the critical circular dependency and to make IW usable for deep learning. Specifically, DIW uses a non-linear transformation π created from the current f (being trained) and replaces w ∗ ( x , y ) with w ∗ ( z ) = p te ( z ) /p tr ( z ) , where z = π ( x , y ) is the current loss-value or hidden-layer-output representation of ( x , y ) . DIW iterates between WE for estimating w ∗ ( z ) and WC for training f and thus updating π in a seamless mini-batch-wise manner. Given that WE enjoys representation learning inside WC, the importance-weight estimation quality of WE and the classifier training quality of WC can improve each other gradually but significantly.

Risk consistency/inconsistency of IW Now, consider how to qualify good or bad expected objectives under different conditions. To this end, we adopt the concepts of risk consistency and classifier consistency from the label-noise learning literature (Xia et al., 2019, 2020; Yao et al., 2020).

Definition 1. Given an (expected) objective J ( f ) , we say it is risk-consistent if J ( f ) = R ( f ) for any f , i.e., the objective is equal to the original risk for any classifier. On the other hand, we say J ( f ) is classifier-consistent if arg min f J ( f ) = arg min f R ( f ) where the minimization is taken over all measurable functions, i.e., the objective shares the optimal classifier with the original risk.

In the definition above, risk consistency is conceptually stronger than classifier consistency. If an objective is risk-consistent, it must also be classifier-consistent; if it is classifier-consistent, it may sometimes be risk-inconsistent. Note that a risk-inconsistent objective is not necessarily very bad, as it can still be classifier-consistent. 5 Hence, when considering expected objectives, risk consistency is a sufficient condition and classifier consistency is a necessary condition for good objectives.

In what follows, we analyze when and why the objective of IW, namely J ( f ) in (2), can be a good objective or may be a bad objective.

Theorem 1. In cases (i) and (ii), IW is risk-consistent. 6

Proof. Recall that S tr = { ( x , y ) : p tr ( x , y ) > 0 } and S te = { ( x , y ) : p te ( x , y ) > 0 } . Under case (i) or (ii), let us rewrite R ( f ) and J ( f ) with summations and integrals:

where w ∗ ( x , y ) = p te ( x , y ) /p tr ( x , y ) is always well-defined over S tr and we safely plugged this definition into the rewritten J ( f ) . Subsequently, in case (i), S tr = S te and thus J ( f ) = R ( f ) . In case (ii), S tr ⊃ S te and then we further have

Theorem 2. In cases (iii) and (iv), IW is risk-inconsistent, and it holds that J ( f ) < R ( f ) for any f . Proof. Since w ∗ ( x , y ) is well-defined over S tr but it becomes ill-defined over S te \ S tr , we cannot naively replace the integral domain in J ( f ) as in the proof of Theorem 1. In case (iii), S tr ⊂ S te , and consequently

According to Theorem 1 for case (i), the first term in the rewritten R ( f ) equals J ( f ) . Moreover, the second term is positive, since ℓ ( f ( x ) , y ) > 0 due to the positivity of ℓ , and p te ( x , y ) > 0 over S te including S te \ S tr . As a result, in case (iii), R ( f ) > J ( f ) .

Similarly, in case (iv), we can split S te into S te ∩ S tr and S te \ S tr and decompose R ( f ) as

Note that S te ∩ S tr ⊂ S tr , so that according to Theorem 1 for case (ii), the first term equals J ( f ) . Following case (iii), the second term is positive. Therefore, in case (iv), R ( f ) > J ( f ) .

Theorem 1 implies that the objective of IW can be a good objective in cases (i) and (ii). Theorem 2 implies that the objective of IW may be a bad objective in cases (iii) and (iv). As a consequence, the theorems collectively address when and why IW methods can succeed/may fail. 7

When the IW objective may be bad and IW methods may fail, whether an IW method fails or not depends on many factors, such as the underlying data distributions, the sampled data sets, the loss, the model, and the optimizer. To illustrate this phenomenon, here we give two concrete examples belonging to case (iii), where IW has no problem at all in one example and is as poor as random guessing in the other example.

Two concrete examples We have seen the examples in Figure 2. In both examples, there are two classes marked with red and blue colors and distributed in four squares. Each square has a unit area and is the support of a uniform distribution of x , i.e., p ( x , 1) = 1 and p ( x , 0) = 0 if its color is red, and p ( x , 0) = 1 and p ( x , 1) = 0 if its color is blue. There is a margin of 0.1 between two adjacent squares. The training distribution consists of the two squares on the left, and the test distribution consists of all the four squares. In the first example, on S te \ S tr , the label is red on the top and blue on the bottom, as same as the label on S tr . In the second example, on S te \ S tr , the label is blue on the top and red on the bottom, as opposite as the label on S tr .

We experimentally validated whether DIW works or not. The number of training data was 200. The number of validation data was only 4: we sampled one random point from each training square and added the center point of each test-only square. We can see that DIW performs very well in the first example, better than training from only the validation data; unfortunately, DIW performs very poorly in the second example, even worse than training from only the validation data.

The observed phenomenon should not be limited to DIW but be common to all IW methods. Here, we analyze why this phenomenon does not depend on the loss, the model, or the optimizer.

Proposition 3. In the first example, IW is classifier-consistent, while in the second example, IW is classifier-inconsistent. 8

Proof. Without loss of generality, assume that ℓ is classification-calibrated (Bartlett et al., 2006). 9 Let ( c (1) , c (2) ) be the center of S te , and then the four squares are located on the top-left, bottomleft, top-right, and bottom-right of ( c (1) , c (2) ) . For convenience, we abbreviate f ( x (1) , x (2) ) for x (1) > c (1) , x (2) > c (2) as f (+ , +) , f ( x (1) , x (2) ) for x (1) > c (1) , x (2) < c (2) as f (+ , -) , and so on.

Consider the first example. The minimizer of R ( f ) can be any Bayes-optimal classifier , i.e., any f such that f ( · , +) > 0 and f ( · , -) < 0 . Next, on the top-left square, we have p te ( x , 1) = 1 / 4 , p te ( x , 0) = 0 , p tr ( x , 1) = 1 / 2 , and p tr ( x , 0) = 0 , and thus w ∗ ( x , y ) = 1 / 2 . Likewise, on the bottom-left square, we have w ∗ ( x , y ) = 1 / 2 . As a result, J ( f ) = 1 2 E p tr ( x ,y ) [ ℓ ( f ( x ) , y )] , meaning that the minimizer of J ( f ) can be any Bayes-optimal classifier on S tr , i.e., any f such that f ( -, +) > 0 and f ( -, -) < 0 . The simplest manner for f to transfer its knowledge from p tr to p te is to have a linear decision boundary and extend it to S te \ S tr , so that f ( · , +) > 0 and f ( · , -) < 0 on S te . We can see that the set of minimizers is shared and thus IW is classifier-consistent.

Consider the second example. The minimizer of J ( f ) is still the same while the minimizer of R ( f ) significantly changes to any f such that f ( -, +) > 0 , f ( -, -) < 0 , f (+ , +) < 0 , and f (+ , -) > 0 . This non-linear decision boundary is a checkerboard where any two adjacent squares have opposite predictions. It is easy to see that IW is classifier-inconsistent and its test accuracy is 0.5. For binary classification with balanced classes, this accuracy is as poor as random guessing.

3 Generalized importance weighting (GIW)

We have seen two examples where IW is as good/bad as possible in case (iii). In practice, we cannot rely on the luck and hope that IW would work. In this section, we propose generalized importance weighting (GIW), which is still IW in cases (i) and (ii) and is better than IW in cases (iii) and (iv).

3.1 Expected objective of GIW

The key idea of GIW is to split the test support S te into the in-training (IT) part S te ∩ S tr and the out-of-training (OOT) part S te \ S tr . More specifically, we introduce a third random variable, the support-splitting variable s ∈ { 0 , 1 } , such that s takes 1 on S tr and 0 on S te \ S tr . As a result, the underlying joint density p ( x , y, s ) can be defined by p te ( x , y ) as 10

Let α = p ( s = 1) . Then, the expected objective of GIW is defined as

The corresponding empirical version ̂ J G ( f ) will be derived in the next subsection. Before proceeding to the empirical objective of GIW, we establish risk consistency of GIW.

Theorem 4. GIW is always risk-consistent for distribution shift problems.

Proof. Let us work on the first term of J G ( f ) in (5). When ( x , y ) ∈ S tr ,

where p te ( x , y ) = p ( x , y | s = 1) given ( x , y ) ∈ S tr according to (4). Since p ( x , y, s = 1) = 0 on S te \ S tr , we have

Next, for the second term of J G ( f ) , since (1 -α ) p ( x , y | s = 0) = p ( x , y, s = 0) , we have

Note that p ( x , y, s = 1) + p ( x , y, s = 0) = p te ( x , y ) according to (4). By adding (6) and (7), we can obtain that J G ( f ) = R ( f ) . This conclusion holds in all the four cases.

Theorem 4 is the main theorem of this paper. It implies that the objective of GIW can always be a good objective. Recall that IW is also risk-consistent in cases (i) and (ii), and it is interesting to see how IW and GIW are connected. By definition, given fixed p tr ( x , y ) and p te ( x , y ) , if there exists a risk-consistent objective, it is unique. Indeed, in cases (i) and (ii), GIW is reduced to IW, simply due to that α = 1 and J G ( f ) = J ( f ) for any f .

3.2 Empirical objective and practical implementation of GIW

Approximating J G ( f ) in (5) is more involved than approximating J ( f ) in (2). Following (3), we need an empirical version ̂ w ( x , y ) , and we need further to split the validation data D v into two sets and estimate α . Obviously, how to accurately split the validation data is the most challenging part. After splitting D v and obtaining an estimate ̂ α , the empirical objective will have two terms, where

the first term can be handled by any IW algorithm given training data and IT validation data, and the second term just involves OOT validation data.

To split D v and estimate α , we employ the one-class support vector machine (O-SVM) (Schölkopf et al., 1999). Firstly, we pretrain a deep network for classification on the training data D tr a little bit and obtain a feature extractor from the pretrained deep network. Secondly, we apply the feature extractor on the instances in D tr and train an O-SVM based on the latent representation of these instances, giving us a score function g ( z ) that could predict whether p tr ( x ) > 0 or not, where z is the latent representation of x . Thirdly, we apply the feature extractor on the instances in D v and then employ g ( z ) to obtain the IT validation data D v1 = { ( x v1 i , y v1 i ) } n v1 i =1 and the OOT validation data D v2 = { ( x v2 i , y v2 i ) } n v2 i =1 . Finally, α can be naturally estimated as ̂ α = n v1 /n v .

We have two comments on the split of D v . The O-SVM g ( z ) predicts whether p tr ( x ) > 0 or not rather than whether p tr ( x , y ) > 0 or not. This is because the x -support change is often sufficiently informative in practice: when the x -support changes, O-SVM can detect it; when the ( x , y ) -support changes without changing the x -support, it will be very difficult to train an O-SVM based on the loss-value representation of ( x , y ) to detect it, but such changes are very rare. The other comment is about the choice of the O-SVM. While there are more advanced one-class classification methods (Hido et al., 2011; Zaheer et al., 2020; Hu et al., 2020; Goldwasser et al., 2020) (see Perera et al. (2021) for a survey), the O-SVM is already good enough for the purpose (see Appendix C.1).

Subsequently, D v1 can be viewed as being drawn from p ( x , y | s = 1) , and D v2 can be viewed as being drawn from p ( x , y | s = 0) . Based on D tr and D v1 , we can obtain either ̂ w ( x , y ) or ̂ w i for each ( x tr i , y tr i ) by IW. IW has no problem here since the split of D v can reduce case (iii) to case (i) and case (iv) to case (ii). In the implementation, we employ DIW (Fang et al., 2020) because it is friendly to deep learning and it is a state-of-the-art IW method. Finally, the empirical objective of GIW can be expressed as

where the two expectations in J G ( f ) are approximated separately with D tr and D v2 . 11

The practical implementation of GIW is presented in Algorithm 1. Here, we adopt the hidden-layeroutput representation for O-SVM in VALDATASPLIT and the loss-value representation for DIW in

See LeCun et al. (1998) for MNIST and Krizhevsky and Hinton (2009) for CIFAR-20. Color-MNIST is modified from MNIST. The model is a modified LeNet-5 (LeCun et al., 1998) or ResNet-18 (He et al., 2016). Please find in Appendix B.1 the details. *All setups in the table are for case (iii); for MNIST in case (iv), the test data consist of 8 digits (2-9).

MNIST, case (iii)

MNIST, case (iv)

Color-MNIST, case (iii)

CIFAR-20, case (iii)

IW-like

0

50

100 150 200 250 300 350 400

Epoch

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Test Accuracy

Val-only

Pretrain-val

Reweight

MW-Net

R-DIW

DIW

GIW

0

50

100 150 200 250 300 350 400

Epoch

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.2

0.3

0.4

0.5

0.6

Test Accuracy

DA

0

50

100 150 200 250 300 350 400

Epoch

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

Val-only

Pretrain-val

CCSA

DANN

GIW

0

50

100 150 200 250 300 350 400

Epoch

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.2

0.3

0.4

0.5

0.6

Test Accuracy

MODELTRAIN. This algorithm design is convenient for both O-SVM and DIW; the hidden-layeroutput representation for DIW has been tried and can be found in Section 4.3.

4 Experiments

In this section, we empirically evaluate GIW and compare it with baseline methods. 12 To see how effective it is in cases (iii) and (iv), we designed two distribution shift (DS) patterns. In the first pattern, DS comes solely from the mismatch between the training and test supports, and we call it support shift (SS). Under SS, it holds that w ∗ ( x , y ) equals α in case (iii) and 0 or another constant in case (iv), simply due to renormalization after imposing SS. Hence, the challenge is to accurately split D v . In the second pattern, there is some genuine DS (e.g., label noise or class-prior shift) on top of SS, and we call it support-distribution shift. Since w ∗ ( x , y ) is no longer a constant, we face the challenge to accurately estimate w ∗ ( x , y ) . Additionally, we conducted an ablation study to better understand the behavior of GIW. Detailed setups and more results are given in Appendices B & C.

The baseline methods involved in our experiments are as follows.

Val-only : using only D v to train the model from scratch.

Pretrain-val : first pretraining on D tr and then training on D v .

Reweight : learning to reweight examples (Ren et al., 2018).

MW-Net : meta-weight-net (Shu et al., 2019), a parametric version of Reweight.

DIW : dynamic importance weighting (Fang et al., 2020).

R-DIW : DIW where IW is done with relative density-ratio estimation (Yamada et al., 2011).

CCSA : classification and contrastive semantic alignment (Motiian et al., 2017).

DANN : domain-adversarial neural network (Ganin et al., 2016).

4.1 Experiments under support shift

Wefirst conducted experiments under support shift on benchmark datasets. The setups are summarized in Table 1. For MNIST, our task was to classify odd and even digits, where the training set has only 4 digits (0-3), while the test set has 10 digits (0-9) in case (iii) and 8 digits (2-9) in case (iv). For Color-MNIST, our task was to classify 10 digits; the dataset was modified from MNIST in such a way that the digits in the training set are colored in red while the digits in the test/validation set are colored

Val-only

DIW

Pretrain-val

GIW

十

Label noise 0.2

Label noise 0.4

Class-prior shift 10

Class-prior shift 100

MNIST

0

50

100 150 200 250 300 350 400

Epoch

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Test Accuracy

Val-only

Pretrain-val

Reweight

MW-Net

R-DIW

DIW

GIW

0

50

100 150 200 250 300 350 400

Epoch

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Test Accuracy

Color-MNIST

0

50

100 150 200 250 300 350 400

Epoch

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

CIFAR-20

0

50

100 150 200 250 300 350 400

Epoch

0.1

0.2

0.3

0.4

0.5

0.6

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.1

0.2

0.3

0.4

0.5

0.6

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.1

0.2

0.3

0.4

0.5

0.6

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.1

0.2

0.3

0.4

0.5

0.6

Test Accuracy

in red/green/blue evenly. For CIFAR-100, our task was to classify the 20 predefined superclasses and thus we call it CIFAR-20; the training set contains data from 2 out of the 5 classes for each superclass while the test set contains all classes. For validation set, we sampled 2 data points per test digit for MNIST and Color-MNIST, and 10 data points per class for CIFAR-100.

Figure 3 shows the results on MNIST, Color-MNIST, and CIFAR-20 under support shift 13 , where GIW generally outperforms IW-like and domain adaptation (DA) baselines. We also confirmed that α in (5) is accurately estimated in Appendix C.1. To further investigate how GIW works, we visualized the learned convolution kernels (i.e., weights) for Color-MNIST experiments in Figure 4, where the more observed color represents the larger weights learned on that color channel. Only GIW recovers the weights of all color channels while learning useful features, however, other methods fail to do so.

4.2 Experiments under support-distribution shift

We further imposed additional distribution shift, i.e., adding label noise or class-prior shift, on top of the support shift following the same setup in Table 1. Here we only show the results in case (iii) and defer the results in case (iv) to Appendix C.4 due to the space limitation.

Label-noise experiments In addition to the support shift, we imposed label noise by randomly flipping a label to other classes with an equal probability, i.e., the noise rate. The noise rates are set as { 0 . 2 , 0 . 4 } and the corresponding experimental results are shown in Figure 5 and 6. We can see that compared with baselines, GIW performs better and tends to be robust to noisy labels.

Class-prior-shift experiments On top of the support shift, we imposed class-prior shift by reducing the number of training data in half of the classes to make them minority classes (as opposed to majority classes). The sample size ratio per class between the majority and minority classes is defined as ρ , chosen from { 10 , 100 } . To fully use the data from the minority class, we did not split the validation data in class-prior-shift experiments and used all validation data in optimizing the two

Label noise 0.2

Label noise 0.4

Class-prior shift 10

Class-prior shift 100

MNIST

0

50

100 150 200 250 300 350 400

Epoch

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

Val-only

Pretrain-val

CCSA

DANN

GIW

0

50

100 150 200 250 300 350 400

Epoch

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

Color-MNIST

0

50

100 150 200 250 300 350 400

Epoch

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

CIFAR-20

0

50

100 150 200 250 300 350 400

Epoch

0.1

0.2

0.3

0.4

0.5

0.6

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.1

0.2

0.3

0.4

0.5

0.6

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.1

0.2

0.3

0.4

0.5

0.6

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.1

0.2

0.3

0.4

0.5

0.6

Test Accuracy

0

0.1

0.3

0.5

0.7

Flip rate

0.80

0.85

0.90

0.95

1.00

Test Accuracy

IT to OOT

OOT to IT

(a) Validation split error

10k

12.5k

15k

17.5k

20k

Number of training data

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Test Accuracy

Val-only

Pretrain-val

Reweight

MW-Net

R-DIW

DIW

GIW

(b) Training data size

10

20

30

40

50

Number of validation data

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Test Accuracy

Val-only

Pretrain-val

Reweight

MW-Net

R-DIW

DIW

GIW

(c) Validation data size

0

50

100 150 200 250 300 350 400

Epoch

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Test Accuracy

DIW-F

DIW-L

GIW-F

GIW-L

(d) -F/-L representation

Figure 7: Impact of different validation data splits, sample sizes, and data representations.

4.3 Ablation study

Finally, we performed an ablation study on MNIST under 0.2 label noise. Figure 7(a) and Table 2 in Appendix C.2 present the negative impact of validation data split errors by randomly flipping the IT/OOT data into the OOT/IT part. We can see that GIW is reasonably robust to split errors, and flipping OOT to IT is more problematic than the other direction since it makes the empirical risk estimator of GIW more similar to that of the standard IW. In Figure 7(b), the performance of all methods remains consistent across different values of n tr ∈ { 10000 , 12500 , 15000 , 17500 , 20000 } . In Figure 7(c), GIW performs better with more validation data, e.g., when n v increases from 10 to 20. Moreover, we compare the loss-value (-L) with the hidden-layer-output (-F) representation of data used in the DIW and GIW methods. Figure 7(d) shows GIW-L outperforms others.

5 Conclusions

We characterized distribution shift into four cases according to support shift to gain a deeper understanding of IW. Consequently, we found that IW is provably good in two cases but provably poor in the other two cases. Then, we proposed GIW which is a strict generalization of IW and is provably favorable in all the four cases. GIW is safer to be used in practice as it is difficult to know in which case the problem to be solved is. That said, there are still some potential limitations and thus future directions to work on: GIW requires exactly test-distributed validation data, which is restrictive; it requires not very small validation data, which is demanding; a small amount of OOT validation data joins training directly and is used for validation simultaneously, which might lead to imperceptible overfitting as well as overoptimism about the OOT part of the test support.

Acknowledgments and Disclosure of Funding

TF was supported by JSPS KAKENHI Grant Number 23KJ0438 and the Institute for AI and Beyond, UTokyo. NL was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC number 2064/1 - Project number 390727645. MS was supported by the Institute for AI and Beyond, UTokyo.

References

L. Bartlett, M. I. Jordan, and J. D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association , 101(473):138-156, 2006.

Ben-David, J. Blitzer, K. Crammer, and F. Pereira. Analysis of representations for domain adaptation. In NeurIPS , 2006.

Duchi, P. Glynn, and H. Namkoong. Statistics of robust optimization: A generalized empirical likelihood approach. arXiv preprint arXiv:1610.03425 , 2016.

Fang, N. Lu, G. Niu, and M. Sugiyama. Rethinking importance weighting for deep learning under distribution shift. In NeurIPS , 2020.

Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research , 17(59):1-35, 2016.

Goldwasser, A. T. Kalai, Y. Kalai, and O. Montasser. Beyond perturbations: Learning guarantees with arbitrary adversarial test examples. In NeurIPS , 2020.

Gong, K. Zhang, T. Liu, D. Tao, C. Glymour, and B. Schölkopf. Domain adaptation with conditional transferable components. In ICML , 2016.

Goodfellow, Y. Bengio, and A. Courville. Deep learning . The MIT Press, 2016.

He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR , 2016.

Hido, Y. Tsuboi, H. Kashima, M. Sugiyama, and T. Kanamori. Statistical outlier detection using direct density ratio estimation. Knowledge and information systems , 26:309-336, 2011.

Hu, M. Wang, Q. Qin, J. Ma, and B. Liu. Hrn: A holistic approach to one class learning. In NeurIPS , 2020.

Huang, A. Gretton, K. Borgwardt, B. Schölkopf, and A. Smola. Correcting sample selection bias by unlabeled data. In NeurIPS , 2007.

Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML , 2015.

Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML , 2018.

Kanamori, S. Hido, and M. Sugiyama. A least-squares approach to direct importance estimation. Journal of Machine Learning Research , 10(7):1391-1445, 2009.

P. Kingma and J. L. Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.

LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Motiian, M. Piccirilli, D. A. Adjeroh, and G. Doretto. Unified deep supervised domain adaptation and generalization. In ICCV , 2017.

Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering , 22 (10):1345-1359, 2009.

Perera, P. Oza, and V. M. Patel. One-class classification: A survey. arXiv preprint arXiv:2101.03064 , 2021.

Quionero-Candela, M. Sugiyama, A. Schwaighofer, and N. Lawrence. Dataset shift in machine learning . The MIT Press, 2009.

Ren, W. Zeng, B. Yang, and R. Urtasun. Learning to reweight examples for robust deep learning. In ICML , 2018.

Saito, Y. Ushiku, and T. Harada. Asymmetric tri-training for unsupervised domain adaptation. In ICML , 2017.

Schölkopf, R. C. Williamson, A. Smola, J. Shawe-Taylor, and J. Platt. Support vector method for novelty detection. In NeurIPS , 1999.

Shu, Q. Xie, L. Yi, Q. Zhao, S. Zhou, Z. Xu, and D. Meng. Meta-weight-net: Learning an explicit mapping for sample weighting. In NeurIPS , 2019.

Sugiyama and M. Kawanabe. Machine learning in non-stationary environments: Introduction to covariate shift adaptation . The MIT Press, 2012.

Sugiyama, M. Krauledat, and K. Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research , 8(5):985-1005, 2007a.

Sugiyama, S. Nakajima, H. Kashima, P. Buenau, and M. Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. In NeurIPS , 2007b.

Sugiyama, T. Suzuki, and T. Kanamori. Density ratio estimation in machine learning . Cambridge University Press, 2012.

Sugiyama, H. Bao, T. Ishida, N. Lu, T. Sakai, and G. Niu. Machine learning from weak supervision: An empirical risk minimization approach . The MIT Press, 2022.

N. Vapnik. Statistical learning theory . John Wiley & Sons, 1998.

Xia, T. Liu, N. Wang, B. Han, C. Gong, G. Niu, and M. Sugiyama. Are anchor points really indispensable in label-noise learning? In NeurIPS , 2019.

Xia, T. Liu, B. Han, N. Wang, M. Gong, H. Liu, G. Niu, D. Tao, and M. Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In NeurIPS , 2020.

Yamada, T. Suzuki, T. Kanamori, H. Hachiya, and M. Sugiyama. Relative density-ratio estimation for robust distribution comparison. In NeurIPS , 2011.

Yao, T. Liu, B. Han, M. Gong, J. Deng, G. Niu, and M. Sugiyama. Dual T: Reducing estimation error for transition matrix in label-noise learning. In NeurIPS , 2020.

Z. Zaheer, J.-h. Lee, M. Astrid, and S.-I. Lee. Old is gold: Redefining the adversarially learned one-class classifier training paradigm. In CVPR , 2020.

A Related work

In this section, we discuss relevant prior studies for addressing distribution shift problems, including importance weighting (IW), IW-like methods, and domain adaptation (DA).

Importance weighting (IW) IWhas been a powerful tool for mitigating the influence of distribution shifts. The general idea is first to estimate the importance, which is the test over training density ratio, and then train a classifier by weighting the training losses according to the importance. Numerous IW methods have been developed in this manner, utilizing different techniques for importance estimation.

The kernel mean matching (KMM) approach (Huang et al., 2007) learns the importance function by matching the distributions of training and test data in terms of the maximum mean discrepancy in a reproducing kernel Hilbert space, while the Kullback-Leibler importance estimation procedure (KLIEP) (Sugiyama et al., 2007b) employs the KL divergence for density fitting and the least-squares importance fitting (LSIF) (Kanamori et al., 2009) employs squared loss for importance fitting. The unconstrained LSIF (uLSIF) (Kanamori et al., 2009) is an approximation version of LSIF that removes the non-negativity constraint in optimization, allowing for more efficient computation. To boost the performance of such traditional IW methods, dynamic importance weighting (DIW) (Fang et al., 2020) is recently proposed to make them compatible with stochastic optimizers, thereby facilitating their effective integration with deep learning frameworks.

However, in order to establish a well-defined notion of importance, all IW methods including DIW assume cases (i) and (ii), while they become problematic in cases (iii) and (iv).

IW-like methods A relevant IW invariant is the relative unconstrained least-squares importance fitting (RuLSIF) (Yamada et al., 2011), which considers a smoothed and bounded extension of the importance. Instead of estimating the importance w ∗ ( x , y ) = p te ( x , y ) /p tr ( x , y ) , they estimate the η -relative importance w ∗ η ( x , y ) = p te ( x , y ) / ( ηp te ( x , y ) + (1 -η ) p tr ( x , y )) where 0 ≤ η ≤ 1 . While the relative importance is well-defined in cases (iii) and (iv), experiments have demonstrated that it is inferior to GIW, since its training does not incorporate any out-of-training (OOT) data.

Moreover, some reweighting approaches based on bilevel optimization look like DIW in the sense of iterative training between weighted classification on the training data for learning the classifier and weight estimation with the help of a small set of validation data for learning the weights (Jiang et al., 2018; Ren et al., 2018; Shu et al., 2019). However, they encounter a similar issue as IW and RuLSIF, where validation data is solely used for learning the weights, while the training data (without any OOT data) is used for training the classifier. This makes them hard to handle the cases (iii) and (iv).

Domain adaptation (DA) DA relates to DS problems where the p te ( x , y ) and the p tr ( x , y ) are usually named as target and source domain distributions (Ben-David et al., 2006), or in-domain and out-of-domain distributions (Duchi et al., 2016). It can be categorized into supervised DA (SDA) or unsupervised DA (UDA): the former has labeled test data while the latter has unlabeled test data. The setting of SDA is similar as that of GIW. One representative SDA work is classification and contrastive semantic alignment (CCSA) (Motiian et al., 2017) method. In CCSA, a contrastive semantic alignment loss is added to the classification loss, for minimizing the distances between the samples that come from the same class and maximizing the distances between samples from different classes. In DA research, UDA is more popular than SDA. Based on different assumptions, UDA involves learning domain-invariant (Ganin et al., 2016) or conditional domain-invariant features (Gong et al., 2016), or giving pseudo labels to the target domain data (Saito et al., 2017).

Note that DA can refer to either certain problem settings or the corresponding learning methods (or both). When regarding it as problem settings, SDA is exactly the same as joint shift and UDA is fairly similar to covariate shift, which assumes p ( y | x ) doesn't change too much between the training and test domain. When regarding it as learning methods, the philosophy of both SDA and UDA is to find good representations to link the source and target domains and transfer knowledge from the source domain to the target domain, which is totally different from the philosophy of IW.

Supplementary Material

Label: 0

Label: 0

Label: 1

Label: 1

Label: 2

Label: 2

Label: 3

Label: 3

Label: 4

Label: 4

Label: 5

Label: 5

Label: 6

Label: 6

Label: 7

Label: 7

Label: 8

Label: 8

Label: 9

Label: 9

Label: 0

Label: 0

Label: 1

Label: 1

Label: 2

Label: 2

Label: 3

Label: 3

Label: 4

Label: 4

Label: 5

Label: 5

Label: 6

Label: 6

Label: 7

Label: 7

Label: 8

Label: 8

Label: 9

Label: 9

4

3

3

4

7

7

8

9

Training data

Validation data

B Supplementary information on experimental setup

In this section, we present supplementary information on the experimental setup. All experiments were implemented using PyTorch 1.13.1 14 and carried out on NVIDIA Tesla V100 GPUs 15 .

B.1 Datasets and base models

MNIST MNIST (LeCun et al., 1998) is a 28*28 grayscale image dataset for 10 hand-written digits (0-9). The original dataset includes 60,000 training data and 10,000 test data. See http: //yann.lecun.com/exdb/mnist/ for more details.

In the experiments, we converted it for binary classification to classify even/odd digits as follows:

Class 0: digits '0', '2', '4', '6', and '8';

Class 1: digits '1', '3', '5', '7', and '9'.

In our setup, the training data only included 4 digits (0-4). The test data could access all digits (0-9) in case (iii) and 8 digits (2-9) in case (iv). Since the number of training data was reduced, we added two data augmentations to the training and validation data: random rotation with degree 10 and random affine transformation with degree 10, translate of (0.1, 0.1) and scale of (0.9, 1.1). Note that the data augmentations were only added during procedure 2 MODELTRAIN in Algorithm 1.

Accordingly, we modified LeNet-5 (LeCun et al., 1998) as the base model for MNIST:

0th (input) layer: (32*32)-

1st layer:

C(5*5,6)-S(2*2)-

2nd layer: C(5*5,16)-S(2*2)-

3rd layer: FC(120)-

4th to 5th layer:

FC(84)-2,

where C(5*5,6) represents a 5*5 convolutional layer with 6 output channels followed by ReLU, S(2*2) represents a max-pooling layer with a filter of size 2*2, and FC(120) represents a fully connected layer with 120 outputs followed by ReLU, etc. The hidden-layer-output representation of data used in the implementation was the normalized output extracted from the 3rd layer.

Color-MNIST Color-MNIST was modified from MNIST for 10 hand-written digit classification, where the digits in training data were colored in red and the digits in test/validation data were colored in either red, green or blue evenly. See Figure 8 for a plot of the training data and validation data. We did not add any data augmentation for experiments on Color-MNIST.

To process RGB input data, we modified LeNet-5 as the base model for Color-MNIST:

where the abbreviations and the way of extracting the hidden-layer-output representation of data were the same as that in MNIST.

CIFAR-20 CIFAR-100 (Krizhevsky and Hinton, 2009) is a 32*32 colored image dataset in 100 classes, grouped in 20 superclasses. It contains 50,000 training data and 10,000 test data. We call this dataset CIFAR-20 since we use it for 20-superclass classification-the predefined superclasses and classes as shown below, where each superclass includes five distinct classes. See https: //www.cs.toronto.edu/~kriz/cifar.html for more details.

Superclass

Class

In our setup, the training data only included the data in 2 out of the 5 classes per superclass, i.e., the classes in ( ) were seen by the training data as shown above. The test data included the data in all classes. Since the number of training data was reduced, we added several data augmentations to the training and validation data: random horizontal flip, random vertical flip, random rotation of degree 10 and random crop of size 32 with padding 4. Same as that in MNIST experiments, the data augmentations were only added during procedure 2 MODELTRAIN in Algorithm 1.

where [ · , · ] denotes a building block (He et al., 2016) and [ · ]*2 means 2 such blocks, etc. Batch normalization (Ioffe and Szegedy, 2015) was applied after convolutional layers. The hidden-layeroutput representation of data was the normalized output after pooling operation in the 18th layer.

B.2 Experiments under support shift

For all compared methods except Val-only, we pre-trained the model for 10 epochs as the initialization. For the one-class support vector machine (O-SVM) (Schölkopf et al., 1999), we adopted

the implementation from scikit-learn 16 , where the radial basis function (RBF) kernel was used: k ( x i , x j ) = e -γ ∥ x i -x j ∥ 2 with γ = 10000 . All other hyperparameters about O-SVM were set as the default. For the distribution matching by dynamic importance weighting (DIW) (Fang et al., 2020), we again used the RBF kernel where γ was the median distance between the training data. And we used K + ωI as the kernel matrix K , where I was an identity matrix and ω was set to be 1e-05. The upper bound of weights was set as 50.

In all experiments under support shift, Adam (Kingma and Ba, 2015) was used as the optimizer, the learning rate was 0.0005, decaying every 100 epochs by multiplying a factor of 0.1, and the batch size was set as 256. For MNIST, Color-MNIST, and CIFAR-20 experiments, the weight decay was set as 0.005, 0.002, and 0.0001, respectively.

B.3 Experiments under support-distribution shift

For experiments under support-distribution shift, all the setups and hyperparameters about the initialization, O-SVM, and distribution matching by DIW were the same as that in Section B.2. Moreover, the same that Adam was used as the optimizer, the learning rate was 0.0005, decaying every 100 epochs by multiplying a factor of 0.1, and the batch size was set as 256. Next we show the setups and hyperparameters specific to the support-distribution shift.

Label-noise experiments On top of the support shift in Section B.2, we added a symmetric label noise to the training data, where a label may flip to all other classes with an equal probability (this probability was defined as the noise rate, set as { 0 . 2 , 0 . 4 } ). The type of label noise and the noise rate were unknown to the model. For MNIST, Color-MNIST, and CIFAR-20 experiments, the weight decay was set as 0.005, 0.002, and 0.008, respectively.

Class-prior shift experiments We induced class-prior shift in the training data by randomly sampling half of the classes as minority classes (other classes were the majority classes) and reducing the number of samples in minority classes. The sample size ratio per class between the majority and minority classes was ρ , chosen from { 10 , 100 } . The randomly selected minority classes in class-prior shift experiments were shown as follows:

MNIST: class of odd digits;

Color-MNIST: digits '1', '2', '6', '7', and '8';

CIFAR-20: superclasses of 'fish', 'fruit and vegetables', 'household electrical device', 'household furniture', 'large carnivores', 'large omnivores and herbivores', 'medium-sized mammals', 'people', 'small mammals', and 'vehicles 1'.

For MNIST, Color-MNIST, and CIFAR-20 experiments, the weight decay was set as 0.005, 1e-05, and 1e-07, respectively. Since we did not split the validation data in class-prior shift experiments, we set the α in (5) as 0.5 for class-prior shift experiments on all datasets.

C Supplementary experimental results

In this section, we present supplementary experimental results, including the histogram plots of the learned O-SVM score, more ablation study results on the validation data split error, visualizations of convolution kernels for all methods under label noise, additional experimental results for case (iv), and the summary of classification accuracy.

C.1 On the learned O-SVM score

Figure 9 shows the histogram plots of the learned O-SVM score on MNIST, Color-MNIST, and CIFAR-20 under support shift. From the histogram plots, we observe that the score distribution consists of two peaks without overlapping; therefore, any value between the two peaks (e.g., 0.4) could be made as a threshold to split the validation data into two parts. If the score of validation data is higher than the threshold, then the data is identified as an (in-training) IT validation data; otherwise, it is an (out-of-training) OOT validation data.

0.0

0.1

0.2

0.3

0.4

0.5

Score

0

2

4

6

8

10

12

Count

(a) MNIST

0.0

0.1

0.2

0.3

0.4

0.5

Score

0

2

4

6

8

10

12

Count

(b) Color-MNIST

0.0

0.1

0.2

0.3

0.4

0.5

Score

0

100

200

300

400

500

600

Count

(c) CIFAR-20

After splitting the validation data, α in (5) is estimated as the ratio of the sample size between the IT validation data and the whole validation data, i.e., ̂ α = n v1 /n v . For example, in Figure 9(a), ̂ α = 8 8+12 = 0 . 4 , which is equal to the true value in MNIST experiments. Similarly, it can be verified that the α in Figure 9(b) and 9(c) are also accurately estimated.

C.2 Additional results on the validation data split error

As mentioned in the main paper in Figure 7(a), generally GIW is quite robust to split errors and OOT → IT is more problematic than the other direction. Here we present additional results on MNIST under 0.4 label noise in Table 2. We can further observe that as the label noise increases from 0.2 to 0.4, IT → OOT is more susceptible to such negative impacts than the other direction.

C.3 Visualizations of convolution kernels under label noise

Then, we visualized the learned convolution kernels for all methods on Color-MNIST under 0.2 label noise. From Figure 10, we can see that the results aligned with the discussions about Figure 4. Previous IW or IW-like methods (i.e., DIW, R-DIW, Reweight and MW-Net), DA methods (i.e., CCSA and DANN), and Pretrain-val learned most weights on the red channel, which may cause the failure on the test data with green/blue color. Although Val-only had weights on all color channels, it may fail to learn useful data representation due to its limited training data. Only GIW could successfully recover the weights on all color channels while capturing the data representation.

C.4 Additional experimental results for case (iv)

Here we present the results on MNIST in case (iv) under distribution-support shift, comparing with IW-like and domain adaptation (DA) baselines. From Figure 11, we can see that GIW outperforms other methods by a large margin in case (iv) under both label-noise and class-prior-shift settings.

C.5 Summary of classification accuracy

Table 3 and 4 present the mean accuracy (standard deviation) in percentage on MNIST, Color-MNIST, and CIFAR-20 over the last ten epochs under support shift, comparing with IW-like methods and domain adaptation (DA) methods respectively, corresponding to Figure 3. Table 5 and 6 present such results under support-distribution shift, corresponding to Figure 5 and 6. Table 7 and 8 shows the summary of results in case (iv) on MNIST, corresponding to Figure 11.

Reweight

MW-Net

R-DIW

DIW

Val-only

Pretrain-val

CCSA

DANN

GIW

X

Label noise 0.2

Label noise 0.4

Class-prior shift 10

Class-prior shift 100

IW-like

0

50

100 150 200 250 300 350 400

Epoch

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Test Accuracy

Val-only

Pretrain-val

Reweight

MW-Net

R-DIW

DIW

GIW

0

50

100 150 200 250 300 350 400

Epoch

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Test Accuracy

0

50

100

150

200

250

300

350

400

Epoch

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Test Accuracy

0

50

100

150

200

250

300

350

400

Epoch

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Test Accuracy

DA

0

50

100 150 200 250 300 350 400

Epoch

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

Val-only

Pretrain-val

CCSA

DANN

GIW

0

50

100 150 200 250 300 350 400

Epoch

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

0

50

100 150 200 250 300 350 400

Epoch

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studied importance weighting that is a fundamental technique in machine learning and data mining. It pointed out a serious issue of existing importance weighting methods that has not been realized or considered carefully --- the support of the training distribution may not fully enclose the support of the test distribution. Then, it proposed a generalized version of importance weighting that can solve the serious issue and has some nice theoretical properties.\n\nSTRENGTHS:\nThe motivation is clear. People employ importance weighting where the importance weights are defined as pte(x,y)/ptr(x,y), and people assume pte(x,y)=0 whenever ptr(x,y)=0, to avoid unbounded importance weights. Consider the case that the support of the training distribution may not fully enclose the support of the test distribution. Since the importance weights are only used for the training distribution, even though they become ill-defined when pte(x,y)>0 and ptr(x,y)=0, importance weighting itself is still well-defined and it seems to be fine... However, this paper pointed out that in such a case the importance-weighting \"identity\" becomes an \"inequality\", which means that what we minimize for training will not be an approximation of the original risk and thus training may be biased. Indeed, this paper constructed a toy example (Figure 2 and Proposition 3) where training is still unbiased or totally biased. I like Proposition 3 very much that emphasizes what people have overlooked so far.\n\nThe idea is novel. It splits the validation data coming from the test distribution into two parts based on one-class classification and pretrained representation (note that pretraining is on the distribution-shifted training data and the proposed method is still end-to-end). Then, the original importance-weighting term is based on one part of the validation data and there is a second term based on the other part of the validation data. In such a way, what we minimize for training is an approximation of the original risk and thus training is still unbiased, as long as the split of validation data is not wrong. As far as I know, this idea or a similar idea has not been explored in importance-weighting research.\n\nThe experiments look very interesting. The Color-MNIST experiments (both the design and the results) are thought-provoking.\n\nWEAKNESSES:\nThe experiments are all done on benchmark datasets at the MNIST/CIFAR scale and there is no big data experiment like ImageNet.\n\nThe biggest model in the experiments is ResNet-18 and there is no big model experiment involving WideResNet, DenseNet, or ViT.\n\nQUESTIONS:\nA point in the motivation should be clarified. As I mentioned above in \"the motivation is clear\", when the importance weights are ill-defined outside the training distribution, importance weighting itself is still well-defined. So have people not noticed the serious issue of importance weighting, or have they noticed it but thought it not a big deal and then ignored it? BTW, you only argued that \"IW methods become ill-defined and problematic\", which is not enough as an explanation. The importance weights may be ill-defined, but those ill-defined importance weights will not be used for training. As a result, IW methods are not ill-defined, but they are not good for training and may lead to poor trained classifiers. However, if we are lucky, they can lead to good trained classifiers, as shown in Figure 2. Am I right?\n\nBefore Proposition 3, you have assumed that \"the classifier would transfer its knowledge about ptr(x,y) to pte(x,y) in the simplest manner\". Why do you need this assumption, or can you explain the implicit meaning of this assumption? If I am not wrong, importance weighting can only train linear classifiers given such training and validation data.\n\nIn this paper, the definition of importance weighting is to use pte(x,y)/ptr(x,y) as the importance weights. There is a more general definition of importance weighting: weighting the loss function on training data to emphasize the loss on some or de-emphasize it on others with the objective of correcting the mismatch between the training and test distributions (see Corinna Cortes, Yishay Mansour, Mehryar Mohri, \"Learning Bounds for Importance Weighting\", NIPS 2010). This more general definition can include your \"IW-like methods\" as IW methods. You theoretically and empirically analyzed when and why IW methods can succeed/may fail. Can your results still hold for this more general definition?\n\nI think importance weighting is not good at the case where the support of the training distribution may not fully enclose the support of the test distribution. This is the motivation of generalizing importance weighting in this paper. However, transfer learning and domain adaptation should also be good at the case which importance weighting is not good at and generalized importance weighting is good at. I found transfer learning or domain adaptation was not mentioned or discussed in this paper. So can you comment on their potential and compare their ability with generalized importance weighting?\n\nLIMITATIONS:\nLabeled validation data coming from the test distribution are required for estimating the importance weights and training. Such validation data may be hard to obtain in practice.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on an important technique in the field of learning under distribution shift. Compared to previous importance weighting techniques, this paper consider more general scenarios: support shift. This paper first investigates how the previous techniques fail on general support-shift scenarios, then proposes a theoretical-justified objective to consider such scenarios. To this end, the whole pipeline is proposed based the objective. In the experiments, necessary empirical justification is included, verifying the effectiveness of the proposed method.\n\nSTRENGTHS:\n1.  The problem setting is interesting and important to the field. In the real world, cases (iii) and (iv) are very common. However, previous techniques do not consider such scenarios. \n\n2. This paper contributes a principle to the field: what kind of objective is classifier-consistency when solving the general distribution-shift problem (i.e., the support-shift problem). \n\n3. The experiments justify the proposed component. The selected baselines cover a lot of existing methods, which is very appreciated.\n\nWEAKNESSES:\n1. The four cases are clearly defined. However, potential practical scenarios can be justified better. In my opinion, test distribution’s support set containing training distribution’s support set might less happen in the current era (as researchers tend to train networks with a lot data). Case (iv) is the most common one, instead of case (iii).\n\n2. In Figure 2, the word “unseen” will cause confusion. We indeed have validation data that come from the test distribution. Thus, it is better to use training data instead of seen/unseen data in Figure 2, or clearly show the decision boundary to avoid potential confusion.\n\n3. It is not well-motivated why we need to introduce risk consistency/inconsistency from label-noise literature. It is a plain definition and should be motivated from its own.\n\n4. Theorem 4 is important to justify the designed objective. However, the most important part, how estimation/observation of s influences the generalization ability is not explored. Necessary discussions should be considered to strengthen the theoretical contribution of this paper.\n\n5. In the experiment, the considered dataset is relatively small. Tasks from partial domain adaptation problem can be used for validating the performance on case (iii). Some large training sets should be considered, e.g., ImageNet.\n\n6. It is unclear how the performance of one-class classifier influences the performance of the proposed method. More experimental analysis should be considered towards this direction.\n\nQUESTIONS:\n1. It is not well-motivated why we need to introduce risk consistency/inconsistency from label-noise literature. It is a plain definition and should be motivated from its own.\n\n2. Theorem 4 is important to justify the designed objective. However, the most important part, how estimation/observation of s influences the generalization ability is not explored. Necessary discussions should be considered to strengthen the theoretical contribution of this paper.\n\n3. In the experiment, the considered dataset is relatively small. Tasks from partial domain adaptation problem can be used for validating the performance on case (iii). Some large training sets should be considered, e.g., ImageNet.\n\n4. It is unclear how the performance of one-class classifier influences the performance of the proposed method. More experimental analysis should be considered towards this direction.\n\nLIMITATIONS:\nThis paper does not have potential negative societal impact to be addressed. Paper’s technical limitations can be found in the above comments.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper introduces a Generalized Importance Weighting (GIW) method designed to address distribution shift problems, where the support of the probability density function varies between training and test data. The authors demonstrate that GIW outperforms existing methods in scenarios where the test support is wider or partially overlaps with the training support.

STRENGTHS:
**Originality**: The paper introduces a novel method called Generalized Importance Weighting (GIW) that extends the traditional Importance Weighting (IW) method.

**Quality**: The paper is well-structured and provides a thorough analysis of why IW may fail in certain cases and how GIW addresses these issues.

**Clarity**: This paper is well-written and easy to follow.

WEAKNESSES:
- This paper evaluates GIW on a few small datasets. However, it would be beneficial to evaluate the method on a large and complex dataset such as ImageNet.   This would provide a more comprehensive understanding of the performance and applicability of GIW.

-  All experiments in this paper are based on small models LetNet-5 and resnet-18. It would be beneficial to evaluate the method on other modern architectures such as vision transformers and VGG.  This would provide a more comprehensive understanding of whether the proposed method is sensitive to model architectures.

QUESTIONS:
- Can the proposed GIM perform well on other model architectures such as VGG/Vision Transfomer and large real-world datasets such as ImageNet?

- The authors utilize O-SVM to split D_v to IT and OOT validation data. How to make sure the O-SVM correctly split the D_v set and how does the quality of O-SVM splitting affect the performance of the  proposed GIW?

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a distribution shift problem where the test distribution $D_\\text{te}$ is different from the training distribution $D_\\text{tr}$.\nSpecifically, the learner is given training data sampled according to $D_\\text{tr}$ and (typically a much smaller size of) validation data sampled according to $D_\\text{te}$, and is required to output a classifier with small generalization error (measured by $D_\\text{te}$).\nThe paper first exemplifies a particular pair of $D_\\text{tr}$ and $D_\\text{te}$ such that the existing method called the importance weighting fails, and then\nimproves the method to better handle such hard cases.\n\nSTRENGTHS:\nGive a hard example where the existing method has significantly poor performance but the proposed method has reasonable performance.\n\nRevise the objective function so as to be risk-consistent, and then propose its empirical version to be minimized, which gives a learning algorithm that outperforms the existing method in experiments.\n\nWEAKNESSES:\nFirst of all, Theorem 4, claimed as the main theorem, is just a simple rewrite of the generalization risk and thus straightforward.\nMoreover, the theorem does not give any justification of the proposed method. Instead, the consistency of the estimator (the empirical objective) should be investigated. (Generally, the notion of consistency is defined for esitimators.)\nI do not think that the expectation of $\\hat J_G(f)$ coincides $R(f)$.\n---\nMy concern is addressed thru the discussion with the authors.\n\nQUESTIONS:\nI wonder why the authors divide the problems into four cases according to the supports of $D_\\text{tr}$ and $D_\\text{te}$. \nApparently, any distribution $D$ with support $S$ is empirically indistinguishable from a distribution $D'$ with support $S'$ if $D$ (and $D'$, resp.) has very small but non-zero probability mass for all $x \\notin S'$ ($x \\notin S$, resp.).\n\nLIMITATIONS:\nIt seems to be impossible to establish a perfectly risk-consistent estimator in general. So we may need to put some assumptions about the difference between $D_\\text{tr}$ and $D_\\text{te}$ and investigate to what extent an estimator is risk-consistent.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}