Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: In this work, the authors proposed a method to address the covariate shift and label shift problems simultaneously. In detail, the prototype-based conditional alignment and self-training based label distribution estimation is utilized. Empirically evaluation is conducted on three datasets to show the superiority of the proposed method. However, the work suffers the following weaknesses:
 
1). The main concern of this work is its shift assumption. In the language of dataset shift, the joint distribution p(x, y) can decompose into two different manners, which are p(y|x)p(x) and p(x|y)p(y). Covariate shift is defined as p(x) not equals to q(x), while the conditional output distribution is invariant p(x|y) = q(x|y), where p(.) and q(.) are distribution for source and target domains. Label shift is defined as p(y) not equals to q(y), while the conditional input distribution p(y|x) = q(y|x). The work assumes that p(x|y) not equals to q(x|y) meanwhile p(x) not equals to q(y). It means to minimize the joint distribution p(x, y), which is well motivated. However, it does not solve the two abovementioned shifts simultaneously here. Instead, it aims to minimize the marginal distribution and conditional distribution in the anticausal direction. See more definitions in the papers “When training and test sets are different: characterizing learning transfer” and “On causal and anticausal learning.”
 
2). The novelty of the paper is limited. While the authors claim that it is the first time to approach it in the proposed manner, the problem of both p(y) and p(x|y) change is not new. For instance, in the paper “Domain adaptation under the target and conditional shift,” the case of distribution shift correction also does not assume the same conditional distribution and marginal distribution for the source domain and the target domain. Also, the fulfill of conditional alignment is used the formulation and architecture of the work “Semi-supervised Domain Adaptation via Minimax Entropy,” except that there is no labeled target data in the target domain (see Eq.(1)). Besides, the notation f in Fig. 2 is missing the description of Section 3.           
 
3).  Although the prototype-based method does help in minimizing the problem of p(x|y) not equals to q(x|y), using the minimax entropy domain adaptation in an unsupervised setting is problematic. Without a few labeled target data points, it is challenging to learn the discriminative features for the target domain. If positives and negatives (suppose it is a binary classification) are severely overlapped in the target domain,  the learned prototypes could be not consistent with those in the source domain. In other words, the prototypes might not indicate the same classes for source and target. Another issue is that the proposed model cannot solve the problem given in the assumption. In detail, the assumption is p(x, y) not equals to q(x, y), using the shared feature function F(.) and classifier C(.) for the source and the target cannot obtain domain-invariant feature and adaptive target predictor at the same time.
 
4). There is an issue in the label distribution by self-training. As the authors claimed, balanced sampling could diminish the effect of the label shift. However, there is no substantial theoretical evidence on this. Intuitively, the balanced sampling only ignores the original marginal distribution of the target domain. The authors should provide more explanation on it. Meanwhile, the sampling couldn’t work when there is a large number of categories. For self-training, it seems no mechanism to alleviate the label shift. Besides, the iterative learning manner heavily depends on the initialization of self-training, i.e., the top-k samples might not represent the marginal distribution.
 
5). For the theoretical insights, first of all, Eq.(5) is given in “A theory of learning from different domains,” Shai Ben-David et al., Machine Learning, 2010. Second, there is a mistake in Eq.(6). The second term on the right of the inequality is not JS divergence of distributions over x. Instead, it is JS divergence of those after transformation of x, z (see the subsection An information-theoretic lower bound, Zhao et al. 2019b).  Also, the descriptions of the insights are not correct.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: 


This paper proposes domain adverarial approaches modified to address covariate shift 
even in the case of shifting label distributions. The paper is interesting and the authors
are looking at an important problem, but the paper suffers several major misconceptions
and the exposition is full of errors.  

The paper proposes a method called “self training” to “estimate and align” the  target label distribution
and a “prototype-based method” for conditional alignment.

The paper purports to introduce a new problem: “generalized domain adaptation”
but actually this is just covariate shift. 
They confuse the term “label shift” and the colloquial “shifting label distribution”.
Moreover when they describe the method formally they fail 
to even state the covariate shift assumption (that p(y|x) = q(y|x) ).
Absent this statement, the problem is unfdamentally underspecified.
They have stated only that p(y) \neq q(y) and that p(x|y) \neq q(x|y).
Thus the claimed contribution is to solve an impossible (more accurately, underspecified problem).

The proposed method leverages a “similarity classifier”
The thing that the authors call a similarity-based classifier
isn’t well explained. It looks like an ad-hoc variation on a standard
softmax prediction layer. 
Here, not that similarity means similarity between x and the label weights w,
not similarity between examples and each other.
Moreover this classifier is trained as a classifier using a 
standard cross-entropy objective on the target domain.
The paper lacks any formal justification for what this is offering
that we do not get from a standard classifier.

The next component of the model is "Prototype-based Conditional Alignment by Minimax Entropy”
and it is also confusingly explained.

The paper attempts to make some reference to the theory of Shai Ben David
which has been badly misapplied in the deep domain papers
(see discussion by Johannson et al 2019 and Wu et al 2019).
Strangely, the authors misattribute the theory to Zhao 2019.


There are some nice ideas in the paper and the experimental results 
(flaws in domain adaptation benchmarks notwithstanding)
appear to be promising.

However the paper is written too confusingly, is outright wrong in many places
and runs the risk of badly misleading readers over even the most basic of definitions.
I encourage the authors to give the paper a gut rewrite
and do not believe that it can be published while resembling its current form.

“only aligns the covariate shift”
>>> 	Be more formal, not clear what it means to “align the shift”
	Moreover, note that owing to lack of shared support , it’s not clear 
	what precisely the current methods (domain-adversarial) do
	or according to what principles they work.


“the covariate shift needs to be minimized”
>>> 	This is not a coherent way of describing the problem.
	The covariate shift is a property of the data.
	You cannot “minimize the shift”

“label shift (p(y) ̸= q(y))”
>>> 	Actually this is a “shfit in label distribution”. 
	Note that you can have a shift in label distribution 
	even under the covariate shift assumption.
	Label shift is the reverse assumption that p(x|y) = q(x|y).


“minimizing the label shift”
>>>	Again, this doesn’t make sense. The practitioner doesn’t get to choose the data
	that they will face at test time. The “shift” refers to the data. 


"Specifically, we assume p(x|y) ̸= q(x|y) and p(y) ̸= q(y)”
>>>	The problem described in this paper is called Generalized Domain Adaptation”
	but actually it is just “covariate shift” this is not a new problem. 
	The use of new terminology for old problems and misapplication of old terminology,
	e.g. “label shift” make this paper a potential danger to readers 
	who then will be confused in their subsequent interactions with 
	the wider literature on distribution shift.

“These methods have achieved state-of-the- art performance on several domain adaptation benchmarks“
>>>	It’s worth pointing out that benchmark SOTA is a dubious way to assess performance out of sample.
	The point is that in supervised learning you get to know that your target is the same as your source,
	so it’s ok to have a whole community smash the validation set and then see if we push the leaderboard on test data
	With domain adaption, the relevant sample size is the number of shifts, no the number of images.
	Having the community pound on 2-3 shifts tells us virtually nothing.


"our approach diminishes covariate and label shift”
>>>	Again this is not the accurate way to describe what you do.
	You attempt to salvage classifier performance under these shifts,
	you cannot “diminish the shift”. 

“Recently, Azizzadenesheli et al. (2019b) propose a regularized algorithm to correct shifts in the label distribution by estimating the importance weights using labeled source data and unlabeled target data. Lipton et al. (2018b) introduce a test distribution estimator to detect and correct for label shift.”
>>>	This is not exactly the right characterization of the related work.
	The method proposed by Lipton 2018 is precisely what 
	Azzizadeneshelli 2019 build upon (by adding a regularizer) 

“Conventional domain adaptation approaches … only align marginal feature distribution”
>>>	Again, the authors speak about “aligning the marginal feature distribution”,
	but this is confusing, Representations are aligned, not features.
	

“This motivates us to align the conditional feature distribution, i.e. p(x|y) and q(x|y)”
>>>	Again these are not things that can be “aligned”. They are properties of the data.
	The entire paper needs to be re-written to be semantically coherent.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper deals with covariate and label shift in common
evaluated on some standard benchmark data.
The paper is scientific sound and appears to be in good shape,
just a few comments:
- how is your label shift different to concept drift in supervised learning?
  --> if this is basically the same I would expect that you can use,mention methods from there
- the plots with t-SNE are obviously colorful but do not provide a lot of information - they should
  be removed - I do not see a benefit
- it is very common that all these methods (like yours) are provided on some kind of image
  data ... is there a particular reason / limitation?
- I would like to see additional experiments on similar text data -> reuters
- self-training is a concept from semi-supervised learning and not particular well supported
  in the community - how do you make sure that the result remain valid
- how do you make sure that the adaptation of the labels, the covariate shift and the classifier training
  are not in facting cheating the result to an optimum within the optimized cost function?

review_title: Official Blind Review #2

Meta-review: This paper proposes a method to address the covariate shift and label shift problems simultaneously. 

The paper is an interesting attempt towards an important problem. However, Reviewers and AC commonly believe that the current version is not acceptable due to several major misconceptions and misleading presentations. In particular:
- The novelty of the paper is not very significant.
- The main concern of this work is that its shift assumption is not well justified.
- The proposed method may be problematic by using the minimax entropy and self-training with resampling.
- The presentation has many errors that require a full rewrite.

Hence I recommend rejection.
Decision: Reject