Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work proposes to optimize PU classifiers and conditional generative models jointly. A CNI-CGAN framework is proposed, in which the generated examples and pseudo labels are applied to PU classifications and the conditional generation, respectively.

The leverage of noisy labels in the joint optimization across classification and generation seems interesting. Especially, both PU classification and conditional generation can benefit from such a joint optimization.

I have some concerns about the details of this work.
1. This work extends binary PU learning to a multi-class version so that deep neural networks can be applied. In the proposed CNI-CGAN framework, to my understanding, any PU classifier can be applied here, not necessarily a differentiable one. The multi-PU classifier proposed in [1] can also be applied. It might also work on MNIST and FMNIST.

2. Another question is about PU(X_r) and y_tilde in Figure 1. They are one-hot coding in this work. A trivial but promising choice is to use soft labels. Any special reason for choosing one-hot encoding?

3. Towards the estimation of C_tilde: it is updated with Exponential Moving Average (EMA). Why use the moment-based update instead of the instantaneous update? The noise label corresponding to the newest PU classifier can be obtained with the instantaneous update. Or it just stabilizes the training process?

4. It is expensive to estimate k + 1 by k + 1 confusion matrix C_tilde. The estimation is conducted (L-L_0) times in each update. Its efficiency should be discussed, especially when compared to the cost of the whole training process.


[1] Yixing Xu, Chang Xu, Chao Xu, and Dacheng Tao. Multi-positive and unlabeled learning. In IJCAI, pp. 3182–3188, 2017. 
review_title: A work on jointly optimizing PU and conditional generation.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
This paper proposed the combination of two techniques for improved learning with unlabelled data: 1)  Positive-Unlabelled (PU) classifier, and 2) class-conditional GAN (cGAN). The idea is that the PU classifier can help produce more accurate pseudo labels for training of a cGAN, and with the improved cGAN, the generated images can be used in turn to further improve the PU classifier. The idea looks interesting and the empirical results verified its effectiveness. 

The major weakness of this paper is the presentation. 
1. The paper is hard to read. The problems are not well defined and connected. The exact learning setting is vague. What is the main problem the authors try to solve here? The classification problem or the generation problem? 
2. The technical contribution to PU classification is very limited. The proposed learning pipeline is basically: 1) training PU classifier, 2) using the classifier do something else, then 3) retraining the classifier with more data. This does not seem to be a solid contribution.
3.  On the other hand, what is the contribution to generative modelling with extra unlabelled data? Using a more accurate predictor (PU classifier) to obtain high-quality pseudo labels is also trivial.
4. Why the proposed approach helps learning with out-of-distribution data? How does OOD data help GAN learning? In other words, what is the goal of OOD GAN?
5. Why PU data is a practical way for using web data?  Why not simply use a pre-trained models to do pseudo labelling, along with open-set or OOD learning strategies? The experiments were only run on small datasets MNIST, Fashion MNIST and CIFAR-10. I am not convinced what the authors proposed in this paper is useful for dealing with real-world web data like WebVision. 
6. Is PU Acc a fair performance metric for baseline methods? 
7. Many typos needed to be fixed.

Comments after rebuttal:
-------------
Thank the authors for the clarifications. I will raise my score to 5. Theoretical analysis of the proposed method is nice. But I still think the proposed approach was not well justified or motivated. Why is it the best option? And how are other simple baselines for improving both (not standalone) settings? 
review_title: Lack of justification and fair comparison

final_score: rating: 6: Marginally above acceptance threshold
review_content: The topic of the work is interesting. It uses a joint training system to improve two divergent techniques in a novel way which jointly bootstraps better performance on the problems that they demonstrate. They extend their comparisons in ways to increase the data imbalance problem and show robustness to even that. The tests they do show are convincing that there is definitely something of value here, however, in the next paragraph, I will discuss some problems I find with the work and comparisons themselves.

Did every technique get the same number of iterations to converge, or did the authors use early stopping? The authors state that "[they] take the Inception Score into consideration," what does that mean? How did they take it into consideration? The table of their results lists a percentage, and inception score is not a percentage. The authors state they use an "almost oracle" for MNIST and Fashion-MNIST, but they do not state how they got percentages for CIFAR-10, nor, if they used an oracle, what the accuracy metrics are for the CIFAR-10 oracle. How does the model hold up on more than 10 classes? There is no discussion on that topic (many real world problems deal with more than 10 classes). What about data imbalance worse than 2:1 (also shows up all the time in real world problems)? How long did training take in clock time vs these other techniques? What about number of parameters used for the different models? It would also be nice to see how well it performs on slightly harder datasets with larger samples.

There were some clarity issues as well: variables and notation are often used before they are explained, and there are gaps in explaining how exactly one would implement these results: what hyperparameters did the authors use for $L_0$, $\lambda$, $\beta$, and $\kappa$ for their results? Poor grammar throughout the paper also causes some confusion when reading. A summary of the proof of theorem 1 would be nice, as appendix A is not released so is not verifiable to this reviewer.

One small issue is that the paper seems to cite a lot of survey papers and not the original work itself, however, there is nothing majorly concerning on that front and it is well supported.


Overall, the paper seems original and decently significant, however, it would be a much stronger submission if some of the issues here were addressed. As it stands, it is a 6/10.
review_title: Interesting idea with obviously interesting results is hampered by poor communication of ideas, poor organization, poor grammar, and incomplete testing. Despite the flaws, the idea is interesting enough to warrant publishing. With updates to grammar and more care and attention paid to communicating core ideas, it would be a 7.

Meta-review: The paper studies the problem of leveraging Positive-Unlabeled~(PU) classification and conditional generation with extra unlabeled data simultaneously in one learning framework. Some major review concerns on the weaknesses include limited novel technical contributions, poor presentation and weak experimental results (e.g., experiments were mostly conducted on small toy datasets). Overall, the paper has some interesting idea, but the work is clearly below the ICLR acceptance bar. 
Decision: Reject