Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors consider the problem of a mismatch between the distribution of the training observations and the test distribution (a transfer learning setup). The paper seems technically sound but it is not easy to read. Even Section 2 it is difficult to read.
 
-  Main drawback: Please define the weights w_i of Eq. (5) in Section 2. 

- I have a question: is it possible to extend your work considering Multiple Importance Sampling and Generalized  Multiple Importance Sampling  schemes? please discuss.



review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper targets the transfer learning problem. It wants to construct the unbiased estimator of the true risk for the target domain based on data from the source domain. To have the unbiased estimator, samples in the source domain are weighted based on some auxiliary information of both the source domain data distribution and the target domain data distribution. Especially, similar to previous works, the paper first assumes P(Y=1) is known for the target domain, and give a generalization bound for learning on the target domain. Then they consider two more concrete problems, one is learning with stratified information when the conditional probability given the stratified information of the source domain is equal to that of the target domain. Then the paper considers PU learning. Generalization bounds are also given for these two problems. Finally, the paper shows some empirical results showing the reweighting effect of its proposal. 

The paper is a theoretical study of transfer learning, and a generalization of other learning problems including transfer learning, learning from stratified data and PU learning. It assumes that when some auxiliary information is known, generalization bound can be given by only minimizing a reweighted loss of the biased source domain data. However, the auxiliary information proposed in this paper is difficult to be got. Thus, the practical use of this paper may be limited. The paper also lacks discussion with related theoretical work (such as generalization bound of PU learning). Due to these reasons, I rate a weak reject for the paper.

In Sec. 2, to have an unbiased risk estimator as well as a generalization bound, the prior probability P(Y=1) should be known. However, the paper fails to provide any practical way to estimate this value. Although in the auxiliary part, some results when such a value cannot be accurately estimated are given, estimation methods are also required for the method to be practical. Moreover, such as result is already studied in Sugiyama et al. (2008). Thus, the novelty of this part is limited. 

In Sec. 3.1, the paper focuses on the learning from stratified data problem, when some stratified information s for the data are given. The paper further assumes P(x|S=k) = P(x’|S’=k). First, in a general learning problem, no matter transfer learning or not, only information of x and y is available. To justify that the information s is available, some real applications should be given as motivations. Moreover, the assumption on the stratified data, i.e. P(x|S=k) = P(x’|S’=k) and P(S=k) \neq P(S’=k) should also be justified. 

In Sec. 3.2, the generalization bound of PU learning is also studied before in for example [Niu et al., Theoretical Comparisons of Positive-Unlabeled Learning against Positive-Negative Learning. NIPS 2016]. Discussion on the relationship between these theoretical results should be given. Also, in the experimental part, there is no empirical results comparing the proposed method with the existing PU learning methods. Since one of the main contributions of this paper is on PU learning, empirical studies should also be provided to show the superior of the proposed method. 

----------------------------
The rebuttal is subjective (without enough support but expressions such as "we believe", "there is no point") and fails to address my concern. I will not raise my score.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary: This paper aims to show that we can estimate a density ratio for using the importance weighted ERM from the given sample and some auxiliary information on the population. Several learning bounds were proven to promote the use of importance weighted ERM.

========================================================
Clarity:
This paper is mathematically concise and understandable overall. Here, I list some comments on the clarity.

1. I found that the word auxiliary information has been used extensively from the very beginning when referring to the estimation of the density ratio. However, there is no explanation what kind of auxiliary information we need to achieve this goal until page 5, where the authors discussed about strata random variable as the additional information (if I didn't make a mistake). I believe there is a better way to introduce the intuition about what kind of auxiliary information is sufficient to make learning possible.

2. For the proposed PU-learning setting (case-control) by du Plessis et al. 2014, the assumption is the marginal density unlabeled data is identical to the test marginal density and the positive data is drawn from the class-conditional probability p(x|y=1). I am not sure if it's appropriate to discuss about it in page 2, where authors want to discuss about the situation where the train stage and test stage have different class probabilities.

3. In page 3, authors suggested that 
"it is very common that the fraction of positive instances in the training dataset is significantly lower than the test set (p' < p), supposed to be known here". I have two questions about this.
3.1 Does this mean we suppose to know p', p, or both? I am aware that the appendix discussed about when p is misspecified.
3.2  I am not convinced that it is common that p' < p. It maybe nice to cite some findings or provide more explanation why it is the case. 

4. In page 4, authors mentioned few shot learning problem, then describe that it is a scenario where almost no training data with positive labels is available. Is this the same problem setting as the well-known few-shot learning one? In my recognition, few-shot learning is the scenario where we want to learn from small data, e.g., p can be 0.5 but we have a very small number of data but balanced (n_pos=n_neg). Instead of few-shot, I feel it might be better to use the word like "extreme class prior or extreme class probability scenario".

5. In page 3, I'm not sure why authors suddenly focused on binary classification with varying class probabilities. A bit of introduction or motivation would be helpful. As far as I understand this is learning from class-prior shift scenario (or class-prior change), which also has been considered in the literature. Authors may consider citing some work in this line and discuss the difference in the findings of the proposed results and the existing work.

========================================================
Comments:
My impression is the novelty of this paper is modest. It is known that importance weighted ERM is unbiased and consistent to the true risk. I believe there exists theoretical analysis of learning under using WERM, especially in the situation where the weight is importance weight function is known. For Lemma 2 and Corollary 1, it is suggest that p' should not be too small but also the author suggested that p' < p. I would like to know more about the setting the authors described here, e.g., what is the example of the practical p' and p. 

1. Eq. (11) is identical to the proposed unbiased risk estimator of PU-learning in du Plessis et al. (NeurIPS2014). It would be better to clarify that they are equivalent (Eq. (3) of PU-learning in du Plessis et al. (NeurIPS2014)). They also provided a generalization error bound and the analysis when p is misspecified. More theoretical analysis of this empirical risk estimator for case-control PU learning (e.g., estimation error bound) can also be found in the following paper:

Niu et al. Theoretical Comparisons of Positive-Unlabeled Learning against Positive-Negative Learning, NeurIPS2016.

2. How many trials were run in the experiments? It would be nice to see the standard error not only the mean result. It is known that importance weighting method can have high variance and it might be expected that WERM may have high variance yet have better performance. It would be helpful to explain how to read the table, e.g., what is No Bias, top-5 error. Why half of the table are in gray?

Although the paper is well-written overall. I found that it is difficult to quantify a novelty of this paper. I believe the goal, as suggested by page 2, is to "set theoretical grounds for the application of ideas behind weighted ERM". As the author suggested, this approach has been studied quite extensively both theoretically and experimentally. It would be helpful to explain what is new and the relationship of the proposed methods or bounds with the existing work to highlight the novelty of this paper.

For these reasons, I vote a weak reject for this paper.

========================================================
Potential typos:
1. There are "du Plessis et al." and "Du Plessis et al." in this paper. This indicates the same person and it should be better to use only one convention (I think du Plessis is preferable).

=========================================================
After Rebuttal:
Thanks for the reader for clarifying my several questions. I have read the rebuttal.
However, I feel that in the current form, I would like to stay with the same evaluation. The clarification of the difference between theoretical results is definitely crucial to highlight the novelty of the paper. I would like to add more comments on the PU learning part. I hope the authors find the comments useful.

1. du Plessis et al. (ICML 2015) "Convex formulation for learning from positive and unlabeled data", which was already cited in the paper, suggested that if we replace the 0-1 loss to a loss that does not have the symmetric property (e.g., logistic, squared), the form of the unbiased estimator can be different from Eq. (11) in this submitted paper (please see the paper for more details).
 
2. Although we obtain an unbiased risk estimator by the WERM-like method, in deep learning, minimizing such a risk may lead to overfitting, as we can see from Eq. (11) that although it is a cost-sensitive risk, it still treats all unlabeled data as negative. If we have a complex enough model, to minimize the risk, a classifier may classify all unlabeled data to negative, which undoubtedly leads to overfitting. This is discussed in Kiryo et al. (NeurIPS 2017), which also has been already cited in the submitted paper too.

Next, I would like to add more comments on the experiment.

For the experiments, I appreciate the authors' effort to do experiments on such a big dataset. In that case, it may be nice to also include an experiment on a smaller dataset, e.g., MNIST, which I believe this has already been conducted but it was in the appendix, to the main body of the paper as well to strengthen the experimental results in the paper. 

I think the writing in the experiment section can be improved. For example, I don't see the first paragraph, which has lots of texts, contains much information. Also, instead of suggesting a reader to see Figure 1 for comparison, we may use more space to interpret the result. If I didn't miss it, Figure 2 and Figure 3 were never explained or referred to in the main body. In that case, we may consider removing these figures and adding the result on MNIST dataset. 


review_title: Official Blind Review #3

Meta-review: This paper aims to address transfer learning by importance weighted ERM that estimates a density ratio from the given sample and some auxiliary information on the population. Several learning bounds were proven to promote the use of importance weighted ERM.

Reviewers and AC feel that the novelty of this paper is modest given the rich relevant literature and the practical use of this paper may be limited. The discussion with related theoretical work such as generalization bound of PU learning can be expanded significantly. The presentation can be largely improved, especially in the experiment part. The rebuttal is somewhat subjective and unconvincing to address the concerns.

Hence I recommend rejection.
Decision: Reject