Classify and Generate Reciprocally: Simultaneous Positive-Unlabelled Learning and Conditional Generation with Extra Data

Abstract: The scarcity of class-labeled data is a ubiquitous bottleneck in a wide range of machine learning problems. While abundant unlabeled data normally exist and provide a potential solution, it is extremely challenging to exploit them. In this paper, we address this problem by leveraging Positive-Unlabeled~(PU) classification and conditional generation with extra unlabeled data \emph{simultaneously}, both of which aim to make full use of agnostic unlabeled data to improve classification and generation performances. In particular, we present a novel training framework to jointly target both PU classification and conditional generation when exposing to extra data, especially out-of-distribution unlabeled data, by exploring the interplay between them: 1) enhancing the performance of PU classifiers with the assistance of a novel Conditional Generative Adversarial Network~(CGAN) that is robust to noisy labels, 2) leveraging extra data with predicted labels from a PU classifier to help the generation. Our key contribution is a Classifier-Noise-Invariant Conditional GAN~(CNI-CGAN) that can learn the clean data distribution from noisy labels predicted by a PU classifier. Theoretically, we proved the optimal condition of CNI-CGAN and experimentally, we conducted extensive evaluations on diverse datasets, verifying the simultaneous improvements on both classification and generation.

Reviews and author responses:

Summary:
This paper proposed the combination of two techniques for improved learning with unlabelled data: 1)  Positive-Unlabelled (PU) classifier, and 2) class-conditional GAN (cGAN). The idea is that the PU classifier can help produce more accurate pseudo labels for training of a cGAN, and with the improved cGAN, the generated images can be used in turn to further improve the PU classifier. The idea looks interesting and the empirical results verified its effectiveness. 

The major weakness of this paper is the presentation. 
1. The paper is hard to read. The problems are not well defined and connected. The exact learning setting is vague. What is the main problem the authors try to solve here? The classification problem or the generation problem? 
2. The technical contribution to PU classification is very limited. The proposed learning pipeline is basically: 1) training PU classifier, 2) using the classifier do something else, then 3) retraining the classifier with more data. This does not seem to be a solid contribution.
3.  On the other hand, what is the contribution to generative modelling with extra unlabelled data? Using a more accurate predictor (PU classifier) to obtain high-quality pseudo labels is also trivial.
4. Why the proposed approach helps learning with out-of-distribution data? How does OOD data help GAN learning? In other words, what is the goal of OOD GAN?
5. Why PU data is a practical way for using web data?  Why not simply use a pre-trained models to do pseudo labelling, along with open-set or OOD learning strategies? The experiments were only run on small datasets MNIST, Fashion MNIST and CIFAR-10. I am not convinced what the authors proposed in this paper is useful for dealing with real-world web data like WebVision. 
6. Is PU Acc a fair performance metric for baseline methods? 
7. Many typos needed to be fixed.

Comments after rebuttal:
-------------
Thank the authors for the clarifications. I will raise my score to 5. Theoretical analysis of the proposed method is nice. But I still think the proposed approach was not well justified or motivated. Why is it the best option? And how are other simple baselines for improving both (not standalone) settings? 

We thank you for your recommendation for acceptance and thoughtful suggestions. Here is our clarification.

$\textbf{1.Number of Iterations.}$ We used the same and sufficient epoch for all settings (180 epochs for joint optimization) to guarantee the convergence as well as for fair comparisons.

$\textbf{2.Inception Score. }$ We meant that we use the Inception Score as an evaluation metric. Please refer to our results on CIFAR-10 in Figure 2, where the Inception Scores of our method outperform other baselines across almost all positive rates. Please also refer to our next answer for how we use Inception Score on CIFAR-10.

$\textbf{3.Oracle Classifier. }$ Note that in Section 3 we state that the Generator Label Accuracy based on oracle classifiers are only applied on MNIST and Fashion-MNIST as the oracle classifiers are more accessible on these small datasets. For CIFAR-10, we utilize the Inception Score instead as the metric to measure the generation quality as shown in Figure 2, as well as the final PU accuracy exhibited in Figure 4. This means that we did not leverage oracle classifiers as the metric on complex datasets, such as CIFAR-10.

$\textbf{4.Imbalanced data. }$ We discussed this in the Discussion part of our paper. Admittedly, we could incorporate more ablation tests on the variations of the data imbalance ratio. But we believe the current experiments are representative and conclusive. Also, it will be promising to incorporate other learning strategies on imbalanced data, e.g., cost-sensitive learning, into our approach to cater to real world scenarios where only highly unbalanced data are available. But they are out of the scope of the paper and we therefore leave them in future work.

$\textbf{5.Other experimental details.}$ (1) We have already released our appendix in the supplementary materials in submission. As for the number of parameters w.r.t. our models, please refer to Appendix D for more details. (2) The computation about the estimation of $\tilde{C}$ and auxiliary loss is negligible compared with the training of the PU classifier and GAN themselves. (3) Please refer to Appendix D for the choice of hyper-parameters. (4) We promise to improve our grammar and better organize our paper in the revised version. Thank you for these suggestions.


Thank you for your effort on reviewing our paper. Here is our response.

$\textbf{1. Definition of our problem. }$ Our problem definition strictly follows the setting of a standard PU classification problem, where Positive and (extra) Unlabeled data, including both positive and out-of-distributional data, are given. Note that under this standard PU classification setting, all the comparisons are scientific and fair. More generally, as stated in the abstract and introduction, we focus on leveraging unlabeled data by exploring the interplay between classification and generation. Specifically, we attempt to improve both generation and classification, aiming for cGAN to learn a clean data distribution out of data with pseudo labels predicted by the PU classifier, and to further boost the accuracy of the PU classifier via data augmentation by CNI-CGAN. PU learning [1] is a classic and important machine learning problem and the data generation with few labels [2] is also a trendy topic. We simultaneously explore the two settings by proposing a unified joint optimization framework.

$\textbf{2. Technical contribution.}$ 
The novelty of our work is a joint training system to improve two divergent techniques in a novel way, which jointly bootstraps better performance. This has also been recognized by Reviewers 1 and 5. Concretely, we clarify our contributions as follows.
(1) Our framework is agnostic to the choice of PU classifiers, demonstrating the generality of our framework. 
(2) We leverage the PU classifier to construct pseudo labels for a better generation. A similar philosophy has been employed in research such as [3] to achieve the state-of-the-art performance, but in very different settings. Our setting is new and important. (3) We argue that it is nontrivial to design a GAN that can learn a clean data distribution based on noisy labels from an imperfect classifier. Our design CNI-CGAN is based on a solid theoretical convergence guarantee shown in Theorem 1, which is the most important contribution in our paper. 

$\textbf{3. Contribution of CNI-CGAN.}$
Firstly, learning a good generative model based on limited labeled data is extremely challenging, which is a common issue, e.g., [2]. The data scarcity can only be addressed by either obtaining more labels or through data augmentation from extra unlabeled data. Considering the laborious nature of labeling, a natural choice is to “pick” images with high-confidence pseudo labels to expand the labeled dataset for a better generation. However, this will inevitably bring the problem of unreliable labels due to the imperfect classifier which is limited by the scarcity of the labels. The most important contribution to generative modeling is that our CNI-CGAN model (in Figure 1) can learn a clean data distribution out of noisy labels given by an imperfect classifier. Previous works like [4] can achieve this optimal condition only when the label confusion matrix is given as a priori. We have discussed this contribution in ‘Comparison with RCGAN’ on Page 4. We also provide a solid theoretical guarantee for our CNI-CGAN, which is by no means a trivial contribution to the machine learning community.

$\textbf{4. OOD data. }$
In the PU classification setting, the PU classifier is trained on both K-class positive data and 1-class negative data, where out-of-distribution (negative) data can help “depict” the decision boundary via optimizing the PU risk estimator in Eq.4. Next, its benefit on the GAN learning lies in the fact the K-class pseudo labels predicted by the PU classifier augment the labeled dataset, significantly improving the situation when the scarcity of data can restrict the performance of GAN. Please refer to “CGAN on Few Labeled Data” in Appendix B for more details.

$\textbf{5. The Choice about PU learning.}$ 
The PU classification is an approach to cope with both labeled and unlabeled data, including both OOD data and in-distributional data but with unknown labels. It is a well-defined and important machine learning problem. Moreover, as GAN-based methods are normally conducted on MNIST, Fashion-MNIST and CIFAR, we focus on the empirical study on these datasets in our paper as well. The consistent performance of our proposal on these representative datasets has demonstrated the huge potential of our model on larger real datasets. 

$\textbf{6. PU Accuracy. }$
The PU accuracy is the most direct and obvious metric to demonstrate the efficacy of generated images in order to boost the performance of the original PU classifier.

$\textbf{7. Typos. }$
We will fix the typos.

[1] Ryuichi Kiryo, et al. Positive-unlabeled learning with non-negative risk estimator. NIPS 2017.   
[2] Mario Lucic, et al. High-fidelity image generation with fewer labels. ICML 2019.  
[3] Berthelot, et al. "Mixmatch: A holistic approach to semi-supervised learning." NeurIPS. 2019.  
[4] Thekumparampil, et al. "Robustness of conditional gans to noisy labels." NeurIPS 2018.


We thank you for your recommendation towards acceptance and constructive feedback. We would like to clarify your questions below: 

1. Speficially, it allows a non-differentiable PU classifier in Step 1 (Line 1 in Algorithm) in order to pre-train a PU classifier, but the differentiability is needed when updating the PU classifier via data augmentation in Step 3 (Line 14 in Algorithm) as the backpropagation is the default choice in the supervised learning. Besides, as our approach is implemented on a popular differentiable programming framework, i.e., Tensorflow, employing a differentiable classifier enables the whole framework to be easily implemented, which will facilitate code sharing.

2. Thanks for your suggestion. Soft label learning [1] can be used to further enhance the performance based on one-hot labels. We will further exploit this in future work.

3. Owing to the stochastic optimization, e.g., mini-batch SGD, of training deep neural networks, the estimation of C_tilde is expected to consider all training samples. Thus, Exponential Moving Average is a natural choice that can consider both the information from the newest PU classifier and training samples in the previous batches.

4. Based on our experiments, a (k+1)x(k+1) matrix is small compared with the number of parameters in deep neural networks. Estimating it takes a fraction of the training time of the whole network. We will add the performance statistics if accepted.

[1] Müller, Rafael, Simon Kornblith, and Geoffrey E. Hinton. "When does label smoothing help?." Advances in Neural Information Processing Systems. 2019.


The topic of the work is interesting. It uses a joint training system to improve two divergent techniques in a novel way which jointly bootstraps better performance on the problems that they demonstrate. They extend their comparisons in ways to increase the data imbalance problem and show robustness to even that. The tests they do show are convincing that there is definitely something of value here, however, in the next paragraph, I will discuss some problems I find with the work and comparisons themselves.

Did every technique get the same number of iterations to converge, or did the authors use early stopping? The authors state that "[they] take the Inception Score into consideration," what does that mean? How did they take it into consideration? The table of their results lists a percentage, and inception score is not a percentage. The authors state they use an "almost oracle" for MNIST and Fashion-MNIST, but they do not state how they got percentages for CIFAR-10, nor, if they used an oracle, what the accuracy metrics are for the CIFAR-10 oracle. How does the model hold up on more than 10 classes? There is no discussion on that topic (many real world problems deal with more than 10 classes). What about data imbalance worse than 2:1 (also shows up all the time in real world problems)? How long did training take in clock time vs these other techniques? What about number of parameters used for the different models? It would also be nice to see how well it performs on slightly harder datasets with larger samples.

There were some clarity issues as well: variables and notation are often used before they are explained, and there are gaps in explaining how exactly one would implement these results: what hyperparameters did the authors use for $L_0$, $\lambda$, $\beta$, and $\kappa$ for their results? Poor grammar throughout the paper also causes some confusion when reading. A summary of the proof of theorem 1 would be nice, as appendix A is not released so is not verifiable to this reviewer.

One small issue is that the paper seems to cite a lot of survey papers and not the original work itself, however, there is nothing majorly concerning on that front and it is well supported.


Overall, the paper seems original and decently significant, however, it would be a much stronger submission if some of the issues here were addressed. As it stands, it is a 6/10.

This work proposes to optimize PU classifiers and conditional generative models jointly. A CNI-CGAN framework is proposed, in which the generated examples and pseudo labels are applied to PU classifications and the conditional generation, respectively.

The leverage of noisy labels in the joint optimization across classification and generation seems interesting. Especially, both PU classification and conditional generation can benefit from such a joint optimization.

I have some concerns about the details of this work.
1. This work extends binary PU learning to a multi-class version so that deep neural networks can be applied. In the proposed CNI-CGAN framework, to my understanding, any PU classifier can be applied here, not necessarily a differentiable one. The multi-PU classifier proposed in [1] can also be applied. It might also work on MNIST and FMNIST.

2. Another question is about PU(X_r) and y_tilde in Figure 1. They are one-hot coding in this work. A trivial but promising choice is to use soft labels. Any special reason for choosing one-hot encoding?

3. Towards the estimation of C_tilde: it is updated with Exponential Moving Average (EMA). Why use the moment-based update instead of the instantaneous update? The noise label corresponding to the newest PU classifier can be obtained with the instantaneous update. Or it just stabilizes the training process?

4. It is expensive to estimate k + 1 by k + 1 confusion matrix C_tilde. The estimation is conducted (L-L_0) times in each update. Its efficiency should be discussed, especially when compared to the cost of the whole training process.


[1] Yixing Xu, Chang Xu, Chao Xu, and Dacheng Tao. Multi-positive and unlabeled learning. In IJCAI, pp. 3182–3188, 2017. 

Meta-review: The paper studies the problem of leveraging Positive-Unlabeled~(PU) classification and conditional generation with extra unlabeled data simultaneously in one learning framework. Some major review concerns on the weaknesses include limited novel technical contributions, poor presentation and weak experimental results (e.g., experiments were mostly conducted on small toy datasets). Overall, the paper has some interesting idea, but the work is clearly below the ICLR acceptance bar. 

Decision: withdrawn-rejected-submissions