Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposed an interesting idea of using two adversarial classifiers for PU learning. The first classifier tries to introduce samples from unlabeled data that are similar to the existing labeled positive data, and the second one tries to detect if a sample has a ground truth label or drawn from unlabeled data by the first classifier (fake). The idea of the paper is interesting; it is well-motivated and well supported with a range of experiments from NLP and computer vision tasks. The paper is a good read but required a pass of proofreading; some rearrangement of the concepts (for example, in second paragraph C(.) and D(.) is used, but they are introduced properly in section 4. Also, the paper could use some clarifications.

* How the proposed method handles unlabeled positive samples that have a different distribution from P or has a similar distribution with some of the negative samples that might exist in unlabeled samples. 

* The experiment section could have enjoyed from an ablation study in which a system that only implements terms I and II from Eq(3). The authors mentioned that such an objective function is asymmetric but didn't explore the implications of such an objective function in the empirical experiments.

* PGAN's results are not compared in the fair condition since the PU version of CIFAR 10 is different from PGAN's version.

* Using MLP classifier for the text classification (e.g., for YELP) makes a very weak baseline for the system. Also, training the word embedding by the system itself is unrealistic. Therefore, the sentence might need to be rewritten.

* Some readability issues: 
(i) C(.) and D(.) needs an introduction in the section "I Introduction" before their usage.
(ii) The idea could be illustrated easily. Such a figure significantly improves the readability of the system.
(iii) be careful with the use of \cite{} and \citep{} interchangeably ("{Liu et al., 2003; Shi et al., 2018" -> Liu et al. (2003) and Shi et al. (2018) ..., 
(iv) The first paragraph of section 2 should be split into two from this phrase "None of these works..."
(v) Please rewrite the latter half of paragraph 2 in section 2. Also, please rewrite the beginning sentences of section 4.1 and the final paragraph of section 3.
(vi) right after equation (2), please change x_s to \mathbf(x)^s for the consistency of your formulation.
(vii) favorible -> favorable, radio (in section 5.1) -> ratio,
(viii) please add a reference for this statement. "This is one of the best architectures for CIFAR10."
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper considers the problem of learning a binary classifier from only positive and unlabeled data (PU learning), where they develop a Predictive Adversarial Networks (PAN) method by using the GAN-like network architecture with a KL-divergence based objective function. Experiments and comparisons with SOTA are provided.

Pros:
Their idea of making an adaption to GAN architecture by replacing the generator by a classifier to select P from U and using the discriminator to distinguish whether the selected data is from P or U for PU learning is interesting, and benefits from not relying on the class prior estimation.

Question:
The authors claim that Eq. (2) cannot be trained in an end-to-end fashion directly, this statement may need some modification since there are some existing works replacing c(x) by a score function or some other continuous function and then this direct adaptation can be trained, for example, see Eq. (5) in “Discriminative adversarial networks for positive-unlabeled learning. arXiv:1906.00642, 2019”. Can any explanation be given on this?

Remarks:
The clarity of the paper could be improved in multiple places. For example, the data generation processes can be mathematically defined in the problem setting part, now it is quite confusing to me. And more details on experimental protocol may be needed: e.g. what kind of hyperparameter tuning was done?
 
In general, the paper proposed an interesting GAN-like network architecture to learn from PU data, but some unclear parts need to be improved. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: <Paper summary>
The authors proposed a novel method for positive-unlabeled learning. In the proposed method, adversarial training is adopted to extract positive samples from unlabeled data. In the experiments, the proposed method achieves better performance compared with state-of-the-art methods. 

<Review summary>
Although the idea to utilize adversarial training for PU learning is interesting, the proposed method is not sufficiently validated in theory. In addition, the manuscript is hard to follow due to confusing notations and lack of figures. I vote for rejection.

<Details>
* Strength
 + The main idea is simple and interesting.
 + The proposed method performs well in the experiments.

* Weakness and concerns
 - Confusing notations and lack of figures.
  -- Lack of mathematical definition of C and D.
  -- The argument of P^p and that of P^u are different (x^p and x^u), which implies that those distributions are defined at different space (but actually same).
  -- Shared index ``i" for positive and unlabeled data in Eq. (3).
  -- The notation with ``hat" often imply the empirically estimated (or approximated) value in the field of ML. 
  -- No figures about the proposed method. Specifically, it is hard to understand the relationship between C and D. 

 - Since Eq. (3) looks totally different from Eq. (2), why Eq. (3) is reasonable remains unclear. 
  -- About I: first, P^{pu} cannot be calculated, because it requires unavailable labels of x^u. If you treat unlabeled data as negative, it should not be called ``ground-truth," and the term I cannot help D correctly recognize positive samples. Second, the positive samples are almost ignored in this term, because the number of positive data should be substantially small in a common setting of PU learning. 
  -- About II: the authors explain the role of this term by min-max game between C and D during optimization, but the most important point here is what will happen when we obtain the optimal C and D after the optimization. What property or behavior do the optimal C and D have? 

 - What do the authors want to claim with Proposition 1? The right-hand side of Eq. (5) cannot be easily calculated due to the density ratio between P^p and P^u. There is no explanation about what f and eps mean. What ``optimal" means is also ambiguous. 


* Minor concerns that do not have an impact on the score
 - Although the problem setting is quite different, the idea of this paper is partially similar to the importance weighting technique adopted in some recent domain adaptation methods [R1, R2]. Do you have any comment on that?

[R1] ``Reweighted adversarial adaptation network for unsupervised domain adaptation," CVPR2018 
[R2] ``Importance weighted adversarial nets for partial domain adaptation," CVPR2018


review_title: Official Blind Review #3

Meta-review: Thanks for your feedback to the reviewers, which helped us a lot to better understand your paper.
Through the discussion, the overall evaluation of this paper was significantly improved.
However, given the very high competition at ICLR2020, this submission is still below the bar unfortunately.
We hope that the discussion with the reviewers will help you improve your paper for potential future publication.
Decision: Reject