Year: 2021

Abstract:
The scarcity of class-labeled data is a ubiquitous bottleneck in a wide range of machine learning problems. While abundant unlabeled data normally exist and provide a potential solution, it is extremely challenging to exploit them. In this paper, we address this problem by leveraging Positive-Unlabeled (PU) classification and the conditional generation with extra unlabeled data simultaneously, both of which aim to make full use of agnostic unlabeled data to improve classification and generation performance. In particular, we present a novel training framework to jointly target both PU classification and conditional generation when exposing to extra data, especially out-of-distribution unlabeled data, by exploring the interplay between them: 1) enhancing the performance of PU classifiers with the assistance of a novel Conditional Generative Adversarial Network (CGAN) that is robust to noisy labels, 2) leveraging extra data with predicted labels from a PU classifier to help the generation. Our key contribution is a Classifier-Noise-Invariant Conditional GAN (CNI-CGAN) that can learn the clean data distribution from noisy labels predicted by a PU classifier. Theoretically, we proved the optimal condition of CNI-CGAN and experimentally, we conducted extensive evaluations on diverse datasets, verifying the simultaneous improvements on both classification and generation.

heading: 1 INTRODUCTION
text: Existing machine learning methods, particularly deep learning models, typically require big data to pursue remarkable performance. For instance, conditional deep generative models are able to generate high-fidelity and diverse images, but they have to rely on vast amounts of labeled data (Lucic et al., 2019). Nevertheless, it is often laborious or impractical to collect large-scale accurate class-labeled data in real-world scenarios, and thus the label scarcity is ubiquitous. Under such circumstances, the performance of classification and conditional generation (Mirza & Osindero, 2014) drops significantly (Lucic et al., 2019). At the same time, diverse unlabeled data are available in enormous quantities, and therefore a key issue is how to take advantage of the extra data to enhance the conditional generation or classification.
Within the unlabeled data, both in-distribution and out-of-distribution data exist, where indistribution data conform to the distribution of the labeled data while out-of-distribution data do not. Our key insight is to harness the out-of-distribution data. In the generation with extra data, most related works focused on the in-distribution data (Lucic et al., 2019; Gui et al., 2020; Donahue & Simonyan, 2019). When it comes to the out-of-distribution data, the majority of existing methods (Noguchi & Harada, 2019; Yamaguchi et al., 2019; Zhao et al., 2020) attempted to forcibly train generative models on a large amount of unlabeled data, and then transferred the learned knowledge of the pre-trained generator to the in-distribution data. In classification, a common setting to utilize unlabeled data is semi-supervised learning (Miyato et al., 2018; Sun et al., 2019; Berthelot et al., 2019), which usually assumes that the unlabeled and labeled data come from the same distribution, ignoring their distributional mismatch. In contrast, Positive and Unlabeled (PU) Learning (Bekker & Davis, 2020; Kiryo et al., 2017) is an elegant way of handling this under-studied problem, where a model has the only access to positive samples and unlabeled data. Therefore, it is possible to utilize pseudo labels predicted by a PU classifier on unlabeled data to guide the conditional gen-
eration. However, the predicted signals from the classifier tend to be noisy. Although there are a flurry of papers about learning from noisy labels for classification (Tsung Wei Tsai, 2019; Ge et al., 2020; Guo et al., 2019), to our best knowledge, no work has considered to leverage the noisy labels seamlessly in the joint classification and generation. Additionally, another work (Hou et al., 2018) leveraged GANs to recover both positive and negative data distribution to step away from overfitting, but they never considered the noise-invariant generation or their mutual improvement. The generative-discriminative complementary learning (Xu et al., 2019) was investigated in weakly supervised learning, but we are the first attempt to tackle the (Multi-) Positive and Unlabeled learning setting while developing the method of noise-invariant generation from noisy labels. Please refer to Section 5 for the discussion about more related works.
In this paper, we focus on the mutual benefits of conditional generation and PU classification, when we are only accessible to little class-labeled data, but extra unlabeled data, including outof-distribution data, can be available. Firstly, a parallel non-negative multi-class PU estimator is derived to classify both the positive data of all classes and the negative data. Then we design a Classifier-Noise-Invariant Conditional Generative Adversarial Network (CNI-CGAN) that is able to learn the clean data distribution on all unlabeled data with noisy labels provided by the PU classifier. Simultaneously, we also leverage our CNI-CGAN to enhance the performance of the PU classification through data augmentation, demonstrating a reciprocal benefit for both generation and classification. We provide the theoretical analysis on the optimal condition of our CNI-CGAN and conduct extensive experiments to verify the superiority of our approach.

heading: 2 OUR METHOD

heading: 2.1 POSITIVE-UNLABELED LEARNING
text: Traditional Binary Positive-Unlabeled Problem Setting Let X ∈ Rd and Y ∈ {±1} be the input and output variables and p(x, y) is the joint distribution with marginal distribution pp(x) = p(x|Y = +1) and pn(x) = p(x|Y = −1). In particular, we denote p(x) as the distribution of unlabeled data. np, nn and nu are the amount of positive, negative and unlabeled data, respectively.
Parallel Non-Negative PU Estimator Vanilla PU learning (Bekker & Davis, 2020; Kiryo et al., 2017; Du Plessis et al., 2014; 2015) employs unbiased and consistent estimator. Denote gθ : Rd → R as the score function parameterized by θ, and ` : R×{±1} → R as the loss function. The risk of gθ can be approximated by its empirical version denoted as R̂pn(gθ):
R̂pn(gθ) = πpR̂ + p (gθ) + πnR̂ − n (gθ), (1)
where πp represents the class prior probability, i.e. πp = P (Y = +1) with πp+πn = 1. In addition, R̂+p (gθ) = 1 np ∑np i=1 ` (gθ (x p i ) ,+1) and R̂ − n (gθ) = 1 nn ∑nn i=1 ` (gθ (x n i ) ,−1) .
As negative data xn are unavailable, a common strategy is to offset R−n (gθ). We also know that πnpn(x) = p(x) − πppp(x), and hence πnR̂−n (gθ) = R̂−u (gθ) − πpR̂−p (gθ). Then the resulting unbiased risk estimator R̂pu(gθ) can be formulated as:
R̂pu(gθ) = πpR̂ + p (gθ)− πpR̂−p (gθ) + R̂−u (gθ), (2)
where R̂−p (gθ) = 1 np ∑np i=1 ` (gθ (x p i ) ,−1) and R̂−u (gθ) = 1nu ∑nu i=1 ` (gθ (x u i ) ,−1). The advantage of this unbiased risk minimizer is that the optimal solution can be easily obtained if g is linear in θ. However, in real scenarios we tend to leverage more flexible models gθ, e.g., deep neural networks. This strategy will push the estimator to a point where it starts to suffer from overfitting. Hence, we decide to utilize non-negative risk (Kiryo et al., 2017) for our PU learning, which has been verified in (Kiryo et al., 2017) to allow deep neural network to mitigate overfitting. The non-negative PU estimator is formulated as:
R̂pu(gθ) = πpR̂ + p (gθ) + max { 0, R̂−u (gθ)− πpR̂−p (gθ) } . (3)
In pursue of the parallel implementation of R̂pu(gθ), we replace max { 0, R̂−u (gθ)− πpR̂−p (gθ) }
with its lower bound 1N ∑N i=1 max { 0, R̂−u (gθ;X iu)− πpR̂−p (gθ;X ip) } where X iu and X ip denote as the unlabeled and positive data in the i-th mini-batch, and N is the number of batches.
From Binary PU to Multi-PU Learning Previous PU learning focuses on learning a classifier from positive and unlabeled data, and cannot easily be adapted to K + 1 multi-classification tasks where K represents the number of classes in the positive data. Multi-Positive and Unlabeled learning (Xu et al., 2017) was ever developed, but the proposed algorithm may not allow deep neural networks. Instead, we extend binary PU learning to multi-class version in a straightforward way by additionally incorporating cross entropy loss on all the positive data with labels for different classes. More precisely, we consider theK+1-class classifier fθ as a score function fθ = ( f1θ (x), . . . , f K+1 θ (x) ) . After the softmax function, we select the first K positive data to construct cross-entropy loss `CE, i.e., `CE(fθ(x), y) = log ∑K+1 j=1 exp ( f jθ (x) ) − fyθ (x) where y ∈ [K]. For the PU loss, we
consider the composite function h(fθ(x)) : Rd → R where h(·) conducts a logit transformation on the accumulative probability for the first K classes, i.e., h(fθ(x)) = ln( p1−p ) in which
p = ∑K j=1 exp ( f jθ (x) ) / ∑K+1 j=1 exp ( f jθ (x) ) . The final mini-batch risk of our PU learning can be presented as:
R̃pu(fθ;X i) = πpR̂+p (h(fθ);X ip) + max { 0, R̂−u (h(fθ);X iu)− πpR̂−p (h(fθ);X ip) }
+ R̂CEp (fθ;X ip), (4)
where R̂CEp (fθ;X ip) = 1np ∑np i=1 ` CE (fθ (x p i ) , y).

heading: 2.2 CLASSIFIER-NOISE-INVARIANT CONDITIONAL GENERATIVE ADVERSARIAL
text: NETWORK (CNI-CGAN)
To leverage extra data, i.e., all unlabeled data, to benefit the generation, we deploy our conditional generative model on all data with pseudo labels predicted by our PU classifier. However, these predicted labels tend to be noisy, reducing the reliability of the supervision signals and thus worsening the performance of the conditional generative model. Besides, the noise depends on the accuracy of the given PU classifier. To address this issue, we focus on developing a novel noise-invariant conditional GAN that is robust to noisy labels provided by a specified classifier, e.g. a PU classifier. We call our method Classifier-Noise-Invariant Conditional Generative Adversarial Network (CNI-CGAN) and the architecture is depicted in Figure 1. In the following, we elaborate on each part of it.
Principle of the Design of CNI-CGAN
Albeit being noisy, the pseudo labels given by the PU classifier still provide rich information that we can exploit. The key is to take the noise generation mechanism into consideration during the generation. We denote the real data as xr and the predicted hard label through the PU classifier as PUθ(xr), i.e., PUθ(xr) = arg maxi f i θ(xr), as displayed in Figure 1. We let the generator “imitate” the noise generation
mechanism to generate pseudo labels for the labeled data. With both pseudo and real labels, we can leverage the PU classifier fθ to estimate a confusion matrix C̃ to model the label noise from the classifier. During the generation, a real label y, while being fed into the generator G, will also be polluted by C̃ to compute a noisy label ỹ, which then will be combined with the generated fake sample xg for the following discrimination. Finally, the discriminator D will distinguish the real samples [xr, PUθ(xr)] out of fake samples [xg, ỹ]. Overall, the noise “generation” mechanism from both sides can be balanced.
Estimation of C̃ The key in the design of C̃ is to estimate the label noise of the pre-trained PU classifier by considering all the samples of each class. More specifically, the confusion matrix C̃ is k+ 1 by k+ 1 and each entry C̃ij represents the probability of a generated sample xg , given a label i, being classified as class j by the PU classifier. Mathematically, we denote C̃ij as:
C̃ij = P (PUθ(xg) = j|y = i) = Ez[I{PUθ(xg)=j|y=i}], (5)
where xg = G(z, y = i) and I is the indicator function. Owing to the stochastic optimization nature when training deep neural networks, we incorporate the estimation of C̃ in the processing of training by Exponential Moving Average (EMA) method. This choice can balance the utilization of information from previous training samples and the updated PU classifier to estimate C̃. We formulate the update of C̃(l+1) in the l-th mini-batch as follows:
C̃(l+1) = λC̃(l) + (1− λ)∆C̃Xl , (6)
where ∆C̃Xl denotes the incremental change of C̃ on the current l-th mini-batch data Xl via Eq. 5. λ is the averaging coefficient in EMA.
Theoretical Guarantee of Clean Data Distribution Firstly, we denote O(x) as the oracle class of sample x from an oracle classifier O(·). Let πi, i = 1, ...,K+1, be the class-prior probability of the class i in the multi-positive unlabeled setting. Theorem 1 proves the optimal condition of CNI-CGAN to guarantee the convergence to the clean data distribution. The proof is provided in Appendix A. Theorem 1. (Optimal Condition of CNI-CGAN) Let P g be a probabilistic transition matrix where P gij = P (O(xg) = j|y = i) indicates the probability of sample xg with the oracle label j generated by G with the initial label i. We assume that the conditional sample space of each class is disjoint with each other, then
(1) P g is a permutation matrix if the generator G in CNI-CGAN is optimal, with the permutation, compared with an identity matrix, only happens on rows r where corresponding πr, r ∈ r are equal. (2) If P g is an identity matrix and the generator G in CNI-CGAN is optimal, then pr(x, y) = pg(x, y) where pr(x, y) and pg(x, y) are the real and the generating joint distribution, respectively.
Briefly speaking, CNI-CGAN can learn the clean data distribution if P g is an identity matrix. More importantly, the method we elaborate till now has already guaranteed Pg as a permutation matrix, which is very close to an identity one. We need an additional constraint, although the permutation happens only when same class-prior probabilities exist.
The Auxiliary Loss The optimal G in CNI-CGAN can only guarantee that pg(x, y) is close to pr(x, y) as the optimal permutation matrix P g is close to the identity matrix. Hence in practice, to ensure that we can exactly learn an identity matrix for P g and thus achieve the clean data distribution, we introduce an auxiliary loss to encourage a larger trace of P g , i.e., ∑K+1 i=1 P (O(xg) = i)|y = i). As O(·) is intractable, we approximate it by the current PU classifier PUθ(xg). Then we obtain the auxiliary loss `aux:
`aux(z, y) = max{κ− 1
K + 1 K+1∑ i=1 Ez(I{PUθ(xg)=i|y=i}), 0}, (7)
where κ ∈ (0, 1) is a hyper-parameter. With the support of auxiliary loss, P g has the tendency to converge to the identity matrix where CNI-CGAN can learn the clean data distribution even in the presence of noisy labels.
Comparison with RCGAN (Thekumparampil et al., 2018; Kaneko et al., 2019) The theoretical property of CNI-CGAN has a major advantage over existing Robust CGAN (RCGAN) (Thekumparampil et al., 2018; Kaneko et al., 2019), for which the optimal condition can only be achieved when the label confusion matrix is known a priori. Although heuristics can be employed, such as RCGAN-U (Thekumparampil et al., 2018), to handle the unknown label noise setting, these approaches still lack the theoretical guarantee to converge to the clean data distribution.
To guarantee the efficacy of our approach, one implicit and mild assumption is that our PU classifier will not overfit on the training data, while our non-negative estimator helps to ensure that it as
explained in Section 2.1. To further clarify the optimization process of CNI-CGAN, we elaborate the training steps of D and G, respectively.
D-Step: We train D on an adversarial loss from both the real data and the generated (xg, ỹ), where ỹ is corrupted by C̃. C̃y denotes the y-th row of C̃. We formulate the loss of D as:
max D∈F E x∼p(x) [φ(D(x, PUθ(x)))] + E z∼PZ,y∼PY ỹ|y∼C̃y [φ(1−D(G(z, y), ỹ))], (8)
where F is a family of discriminators and PZ is the distribution of latent space vector z, e.g., a Normal distribution. PY is a discrete uniform distribution on [K + 1] and φ is the measuring function.
G-Step: We train G additionally on the auxiliary loss `aux(z, y) as follows:
min G∈G E z∼PZ,y∼PY ỹ|y∼C̃y [φ(1−D(G(z, y), ỹ)) + β`aux(z, y)] , (9)
where β controls the strength of auxiliary loss and G is a family of generators. In summary, our CNI-CGAN conducts K+ 1 classes generation, which can be further leveraged to benefit the K+ 1 PU classification via data augmentation.
Algorithm 1 Alternating Minimization for PU Learning and Classifier-Noise-Invariant Generation. Input: Training data (Xp, Xu). Batch size M and hyper-parameter β > 0, λ, κ ∈ (0, 1). L0 and L ∈ N+. Initializing C̃(1) as identity matrix. Number of batches N during the training. Output: Model parameter for generator G, and θ for the PU classifier fθ.
1: / * Pre-train PU classifier fθ * / 2: for i = 1 to N do 3: Update fθ by descending its stochastic gradient of R̃pu ( fθ;X i ) via Eq. 4. 4: end for 5: repeat 6: / * Update CNI-CGAN * / 7: for l = 1 to L do 8: Sample {z1, ..., zM}, {y1, ...,yM} and {x1, ...,xM} from PZ , PY and all training data,
respectively, and then sample {ỹ1, ..., ỹM} through the current C̃(l). Then, update the discriminator D by ascending its stochastic gradient of
1
M M∑ i=1 [φ(D(xi, PUθ(xi)))] + φ(1−D(G(zi,yi), ỹi))].
9: Sample {z1, ..., zM} and {y1, ...,yM} from PZ and PY , and then sample {ỹ1, ..., ỹM} through the current C̃(l). Update the generator G by descending its stochastic gradient of
1
M M∑ i=1 [φ(1−D(G(zi,yi), ỹi)) + β`aux(yi, zi)].
10: if l ≥ L0 then 11: Compute ∆C̃Xl = 1 M ∑M i=1 I{PUθ(G(zi,yi))|yi} via Eq. 5, and then estimate C̃ by
C̃(l+1) = λC̃(l) + (1− λ)∆C̃Xl . 12: end if 13: end for 14: / * Update PU classifier via Data Augmentation * / 15: Sample {z1, ..., zM} and {y1, ...,yM} from PZ and PY , respectively, and then update the
PU classifier fθ by descending its stochastic gradient of
1
M M∑ i=1 `CE (fθ (G(zi,yi)) ,yi) .
16: until convergence

heading: 3 ALGORITHM
text: Firstly, we obtain a PU classifier fθ trained on multi-positive and unlabeled dataset with the parallel non-negative estimator derived in Section 2.1. Then we train our CNI-CGAN, described in Section 2.2, on all data with pseudo labels predicted by the pre-trained PU classifier. As our CNICGAN is robust to noisy labels, we leverage the data generated by CNI-CGAN to conduct data augmentation to improve the PU classifier. Finally, we implement the joint optimization for the training of CNI-CGAN and the data augmentation of the PU classifier. We summarize the procedure in Algorithm 1 and provide more details in Appendix C.
Computational Cost Analysis In the implementation of our CNI-CGAN, we need to additionally estimate C̃, a (K + 1)× (K + 1) matrix. The computational cost of this small matrix is negligible compared with the updating of discriminator and generator networks, although the estimation of C̃ is crucial.
Simultaneous Improvement on PU Learning and Generation with Extra Data From the perspective of PU classification, due to the theoretical guarantee from Theorem 1, CNI-CGAN is capable of learning a clean data distribution out of noisy pseudo labels predicted by the pre-trained PU classifier. Hence, the following data augmentation has the potential to improve the generalization of PU classification regardless of the specific form of the PU estimator. From the perspective of generation with extra data, the predicted labels on unlabeled data from the PU classifier can provide CNI-CGAN with more supervised signals, thus further improving the quality of generation. Due to the joint optimization, both the PU classification and conditional generative models are able to improve each other reciprocally, as demonstrated in the following experiments.

heading: 4 EXPERIMENT
text: Experimental Setup We perform our approaches and several baselines on MNIST, Fashion-MNIST and CIFAR-10. We select the first 5 classes on MNIST and 5 non-clothes classes on FashionMNIST, respectively, for K + 1 classification (K = 5). To verify the consistent effectiveness of our method in the standard binary PU setting, we pick the 4 categories of transportation tools in CIFAR10 as the one-class positive dataset. As for the baselines, the first is CGAN-P, where a Vanilla CGAN (Mirza & Osindero, 2014) is trained only on limited positive data. Another natural baseline is CGAN-A where a Vanilla CGAN is trained on all data with labels given by the PU classifier.
The last baseline is RCGAN-U (Thekumparampil et al., 2018) where the confusion matrix is totally learnable while training. For fair comparisons, we choose the same GAN architecture. Through a line search of hyper-parameters, we choose κ as 0.75, β as 5.0 and λ = 0.99 across all the datasets. We set L0 as 5 in Algorithm 1. More details about hyper-parameters can be found in Appendix D.
Evaluation Metrics For MNIST and Fashion-MNIST, we mainly use Generator Label Accuracy (Thekumparampil et al., 2018) and PU Accuracy to evaluate the quality of generated images. Generator Label Accuracy compares specified y from CGANs to the true class of the generated examples through a pre-trained (almost) oracle classifier f . In experiments, we pre-trained two K+1 classifiers with 99.28% and 98.23% accuracy on the two datasets, respectively. Additionally, the increased PU Accuracy measures the closeness between generated data distribution and test (almost real) data distribution for the PU classification, serving as a key indicator to reflect the quality of generated images. For CIFAR 10, we use both Inception Score (Salimans et al., 2016) to evaluate the quality of the generated samples, and the increased PU Accuracy to quantify the improvement of generated samples on the PU classification.

heading: 4.1 GENERATION AND CLASSIFICATION PERFORMANCE
text: We set the whole training dataset as the unlabeled data and select certain amount of positive data with the ratio of Positive Rate. Figure 2 presents the trend of Generator Label Accuracy, Inception Score and PU Accuracy as the Positive Rate increases. It turns out that CNI-CGAN outperforms CGAN-P and CGAN-A consistently especially when the Positive Rate is small, i.e. little positive data. Remarkably, our approach enhances the PU accuracy greatly when exposed to low positive rates, while CGAN-A even worsens the original PU classifier sometimes in this scenario due to the existence of too much label noise given by a less accurate PU classifier. Meanwhile, when more supervised positive data are given, the PU classifier generalizes better and then provides more accurate labels, conversely leading to more consistent and better performance for all methods. Besides, note that even though the CGAN-P achieves comparable generator label accuracy on MNIST, it results in a lower Inception Score. We demonstrate this in Appendix D.
To verify the advantage of theoretical property for our CNI-CGAN, we further compare it with RCGCN-U (Thekumparampil et al., 2018; Kaneko et al., 2019), the heuristic version of robust generation against unknown noisy labels setting without the theoretical guarantee of optimal condition. As observed in Table 1, our method outperforms RCGAN-U especially when the positive rate is low. When the amount of positive labeled data is relatively large, e.g., 10.0%, both our approach and RCGAN-U can obtain comparable performance.
Visualization To further demonstrate the superiority of CNI-CGAN compared with the other baselines, we present some generated images withinK+1 classes from CGAN-A, RCGAN-U and CNICGAN on MNIST, and high-quality images from CNI-CGAN on Fashion-MNIST and CIFAR-10, in Figure 3. In particular, we choose the positive rate as 0.2% on MNIST, yielding the initial PU classifier with 69.14% accuracy. Given the noisy labels on all data, our CNI-CGAN can generate more accurate images of each class visually compared with CGAN-A and RCGAN-U. Results of Fashion-MNIST and comparison with CGAN-P on CIFAR-10 can refer to Appendix E.

heading: 4.2 ROBUSTNESS OF OUR APPROACH
text: Robustness against the Initial PU accuracy The auxiliary loss can help the CNI-CGAN to learn the clean data distribution regardless of the initial accuracy of PU classifiers. To verify that, we select distinct positive rates, yielding the pre-trained PU classifiers with different initial accuracies. Then we perform our method based on these PU classifiers. Figure 4 suggests that our approach can still attain the similar generation quality under different initial PU accuracies after sufficient training, although better initial PU accuracy can be beneficial to the generation performance in the early phase.
Robustness against the Unlabeled data In real scenarios, we are more likely to have little knowledge about the extra data we have. To further verify the robustness of CNI-CGAN against the unknown distribution of extra data, we test different approaches across different amounts and distributions of the unlabeled data. Particularly, we consider two different types of distributions for unlabeled data. Type 1 is [ 1K+1 , ..., 1 K+1 , 1 K+1 ] where the number of data in each class, including the negative data, is even, while type 2 is [ 12K , ... 1 2K , 1 2 ] where the negative data makes up half of all unlabeled data. In experiments, we focus on the PU Accuracy to evaluate both the generation quality and the improvement of PU learning. For MNIST, we choose 1% and 0.5% for two settings while we opt for 0.5% and 0.2% on both Fashion-MNIST and CIFAR-10.
Figure 5 manifests that the accuracy of PU classifier exhibits a slight ascending tendency with the increasing of the number of unlabeled data. More importantly, our CNI-CGAN almost consistently outperforms other baselines across different amount of unlabeled data as well as distinct distributions of unlabeled data. This verifies that the robustness of our proposal to the distribution of extra data can be maintained potentially. We leave the investigation on the robustness against more imbalanced situations as future works.

heading: 5 RELATED WORKS
text: Positive-Unlabeled (PU) Learning. Positive and Unlabeled (PU) Learning is the setting where a learner has only access to positive examples and unlabeled data (Bekker & Davis, 2020; Kiryo et al., 2017). One related work (Hou et al., 2018) employed GANs (Goodfellow et al., 2014) to recover both positive and negative data distribution to step away from overfitting. Kato et al. (Kato et al., 2018) focused on remedying the selection bias in the PU learning. Besides, Multi-Positive and Unlabeled Learning (Xu et al., 2017) extended the binary PU setting to the multi-class version, therefore adapting to more practical applications. By contrast, our multi-positive unlabeled method absorbs the advantages of previous approaches, and in the meanwhile intuitively extends them to fit the differential deep neural networks optimization.
Conditional GANs on Few Labels Data. To attain high-quality images with both fidelity and diversity, the training of generative models requires a large dataset. To reduce the need of huge amount of data, the vast majority of methods (Noguchi & Harada, 2019; Yamaguchi et al., 2019; Zhao et al., 2020) attempted to transfer prior knowledge of the pre-trained generator. Another branch (Lucic et al., 2019) is to leverage self- and supervised learning to add pseudo labels on the in-distribution unlabeled data in order to expand labeled dataset. Compared with this approach, our strategy can be viewed to automatically “pick” useful in-distribution data from total unknown unlabeled data via PU learning framework, and then constructs robust conditional GANs to generate clean data distribution out of predicted label noise. Please refer to more related works in Appendix B.

heading: 6 DISCUSSION AND CONCLUSION
text: In this paper, we proposed a new method, CNI-CGAN, to jointly exploit PU classification and conditional generation. It is, to our best knowledge, the first method of such kind to break the ceiling of class-label scarcity, by combining two promising yet separate methodologies to gain massive mutual improvements. CNI-CGAN can learn the clean data distribution from noisy labels given by a PU classifier, and then enhance the performance of PU classification through data augmentation in various settings. We have demonstrated, both theoretically and experimentally, the superiority of our proposal on diverse benchmark datasets in an exhaustive and comprehensive manner. In the future, it will be promising to investigate learning strategies on imbalanced data, e.g., cost-sensitive learning (Elkan, 2001), to extend our approach to broader settings, which will further cater to realworld scenarios where highly unbalanced data are commonly available. In addition, the leverage of soft labels in the design of CNI-CGAN is also promising.
Ethics Statement. Our designed CNI-CGAN framework can interplay with the PU classification and robust generation, which can mitigate the scarcity of class-labeled data. Leveraging extra data may correlate with the privacy issue as the privacy issue still exists in generative models. Thus, a privacy-guaranteed version of our algorithm can be further proposed in the future to handle the potential privacy issue.
Reproducibility Statement. For the theoretical part, we clearly state the related assumption and detailed proof process in Appendix A. In terms of the algorithm, our implementation is directly adapted from the public one of generative models and PU learning.

heading: A APPENDIX: PROOF OF THEOREM 1
text: Firstly, we recall some definitions. Denote xr , xg as the real training and generated samples, respectively. x are the population of all data, and xr are sampled from p(x). yg represents the initial labels for the generator G, while ỹ indicates the labels perturbed by C̃ from yg . The class-prior πi meets πi = P (yg = i) = P (O(xr) = i). For a rigorous proof of Theorem 1, we elaborate it again in the appendix.
Theorem 1 We assume that the following three mild assumptions can be met: (a) PU classifier is not overfitting on the training data, (b) P (PUθ(xg)|O(xg), yg) = P (PUθ(xg)|O(xg)), (c) the conditional sample space is disjoint from each other class. Then,
(1) P g is a permutation matrix if the generator G in CNI-CGAN is optimal, with the permutation, compared with an identity matrix, only happens on rows r where corresponding πr, r ∈ r are equal.
(2) If P g is an identity matrix and the generator G in CNI-CGAN is optimal, then pr(x, y) = pg(x, y) where pr(x, y) and pg(x, y) are the real and generating joint distribution, respectively.
A.1 PROOF OF (1)
Proof. For a general setting, the oracle class of xg given by label yg is not necessarily equal to PUθ(xg). Thus, we consider the oracle class of xg , i.e., O(xg) in the proof.
Optimal G. In CNI-CGAN, G is optimal if and only if
pr(xr, PUθ(xr)) = p g(xg, ỹ). (10)
The equivalence of joint probability distribution can further derive the equivalence of marginal distribution, i.e., pr(xr) = p
g(xg). We define a probability matrix C where Cij = P (PUθ(x) = j|O(x) = i) where x are the population data. According to (c), we can apply O(·) on both xr and xg in Eq. 10. Then we have:
P (O(xr) = i, PUθ(xr) = j) (c) = P (O(xg) = i, ỹ = j)
P (O(xr) = i)P (PUθ(xr) = j|O(xr) = i) = K+1∑ k=1 P (yg = k,O(xg) = i)P (ỹ = j|yg = k,O(xg) = i)
πiCij (a) = K+1∑ k=1 P (O(xg) = i|yg = k)P (yg = k)P (ỹ = j|yg = k)
πiCij = K+1∑ k=1 P g>ik πkC̃kj ,
(11)
where assumption (a) indicates that PUθ(xr) is close to PUθ(x) so that P (PUθ(xr) = j|O(xr) = i) = P (PUθ(x) = j|O(x) = i). Then the corresponding matrix form follows as
ΠC = P g>ΠC̃ (12)
Definition. According to the definition of C̃ and Law of Total Probability, we have:
P (yg = i)P (PUθ(xg) = j|yg = i) =
πi K+1∑ k=1 P (O(xg) = k|yg = i)P (PUθ(xg) = j|O(xg) = k, yg = i)
πiC̃ij (b) = πi K+1∑ k=1 P gikP (PUθ(xg) = j|O(xg) = k)
πiC̃ij = πi K+1∑ k=1 P gikCkj ,
(13)
where the last equation is met as p(xg) is close to p(x) whenG is optimal, and thus P (PUθ(xg) = j|O(xg) = k) = P (PUθ(x) = j|O(x) = k). Then we consider the corresponding matrix form as follows
ΠC̃ = ΠP gC (14)
where Π is the diagonal matrix of prior vector π. Combining Eq. 14 and 12, we have P g>ΠP g = Π, which indicates P g is a general orthogonal matrix. In addition, the element of P g is non-negative and the sum of each row is 1. Therefore, we have P g is a permutation matrix with permutation compared with the identity matrix only happens on rows r where corresponding πr, r ∈ r are equal. Particularly, if all πi are different from each other, then permutation operation will not happen, indicating the optimal conditional of P g is the identity matrix.
A.2 PROOF OF (2)
We additionally denote yr as the real label of real sample xr , i.e., yr = O(xr). According to the optimal condition ofG in Eq. 10, we have pr(xr) = pg(xg). Since we have P g is an identity matrix, thenO(xg) = yg a.e. Thus, we have pg(xg|yg = i) = pg(xg|O(xg) = i),∀i = 1, ..,K + 1. According the assumption (c) and Eq. 10, we have pr(xr|O(xr) = i) = pg(xg|O(xg) = i). In addition, we know that pr(xr|O(xr) = i) = pr(xr|yr = i), thus we have pr(xr|yr = i) = pg(xg|yg = i). Further, we consider the identical class-prior πi. Finally, we have
pr(xr|yr = i)πi = pg(xg|yg = i)πi pr(xr|yr = i)p(O(xr) = i) = pg(xg|yg = i)p(yg = i)
pr(xr|yr = i)p(yr = i) = pg(xg|yg = i)p(yg = i) pr(xr, yr) = p g(xg, yg).
(15)

heading: B APPENDIX: MORE RELATED WORKS
text: Positive-Unlabeled (PU) Learning. Positive and Unlabeled (PU) Learning is the setting where a learner has only access to positive examples and unlabeled data (Bekker & Davis, 2020; Kiryo et al., 2017). One related work (Hou et al., 2018) employed GANs (Goodfellow et al., 2014) to recover both positive and negative data distribution to step away from overfitting. Kato et al. (Kato et al., 2018) focused on remedying the selection bias in the PU learning. Besides, Multi-Positive and Unlabeled Learning (Xu et al., 2017) extended the binary PU setting to the multi-class version, therefore adapting to more practical applications. By contrast, our multipositive unlabeled method absorbs the advantages of previous approaches, and in the meanwhile intuitively extends them to fit the differential deep neural networks optimization.
Conditional GANs on Few Labels Data. To attain high-quality images with both fidelity and diversity, the training of generative models requires a large dataset. To reduce the need of huge amount of data, the vast majority of methods (Noguchi & Harada, 2019; Yamaguchi et al., 2019; Zhao et al., 2020) attempted to transfer prior knowledge of the pre-trained generator. Another branch (Lucic et al., 2019) is to leverage selfand supervised learning to add pseudo labels on the in-distribution unlabeled data in order to expand labeled dataset. Compared with this approach, our strategy can be viewed to automatically “pick” useful in-distribution data from total unknown unlabeled data via PU learning framework, and then constructs robust conditional GANs to generate clean data distribution out of predicted label noise.
Robust GANs. Robust Conditional GANs (Thekumparampil et al., 2018; Kaneko et al., 2019) were proposed to defend against class-dependent noisy labels. The main idea of these methods is to corrupt labels of generated samples before feeding to the adversarial discriminator, forcing the generator to produce sample with clean labels. Another supplementary investigation (Koshy Thekumparampil et al., 2019) explored the scenario when CGANs get exposed to missing or ambiguous labels, while another work (Chrysos et al., 2018) leveraged the structure of the model in the target space to address this issue. In contrast, the noises in our model stem from the prediction error of a given classifier. We employ the imperfect classifier to estimate the label confusion noise, yielding a new branch of Robust CGANs against “classifier” label noises.
Semi-Supervised Learning (SSL). One crucial issue in SSL (Miyato et al., 2018; Yu et al., 2019; Sun et al., 2019) is how to tackle with the mismatch of unlabeled and labeled data. Augmented Distribution Alignment (Wang et al., 2019) was proposed to leverage adversarial training to alleviate the bias, but they focused on the empirical distribution mismatch owing to the limited number of labeled data. Further, Uncertainty Aware Self-Distillation (Yanbei Chen, 2019) was proposed to concentrate on this under-studied problem, which can guarantee the effectiveness of learning. In contrast, our approach leverages the PU learning to construct the “open world” classification.
Out-Of-Distribution (OOD) Detection OOD Detection is one classical but always vibrant machine learning problem. PU learning can be used for the detection of outliers in an unlabeled dataset with knowledge only from a collection of inlier data (Hido et al., 2008; Smola et al., 2009). Another interesting and related
work is Outlier Exposure (Hendrycks et al., 2018), an approach that leveraged an auxiliary dataset to enhance the anomaly detector based on existing limited data. This problem is similar to our generation task, the goal of which is to take better advantage of extra dataset, especially out-of-distribution data, to boost the generation.
Learning from Noisy Labels Rotational-Decoupling Consistency Regularization (RDCR) (Tsung Wei Tsai, 2019) was designed to integrate the consistency-based methods with the self-supervised rotation task to learn noise-tolerant representations. Mutual Mean-Teaching (Ge et al., 2020) was proposed to refine the soft labels on person re-identification task by averaging the parameters of two neural networks . In addition, the data with noisy labels can also be viewed as bad data. Another work (Guo et al., 2019) provided a worstcase learning formulation from bad data, and designed a data-generation scheme in an adversarial manner, augmenting data to improve the current classifier.

heading: C APPENDIX: DETAILS ABOUT ALGORITHM 1
text: Similar in (Kiryo et al., 2017), we utilize the sigmoid loss `sig(t, y) = 1/(1 + exp(ty)) in the implementation of the PU learning. Besides, we denote ri = R̂−u ( g;X iu ) − πpR̂−p ( g;X ip ) in the i-th mini-batch. Instructed by the algorithm in (Kiryo et al., 2017), if ri < 0 we turn to optimize −∇θri in order to make this mini-batch less overfitting, which is slightly different from Eq. 4.

heading: D APPENDIX: DETAILS ABOUT EXPERIMENTS
text: PU classifier and GAN architecture For the PU classifier, we employ 6 convolutional layers with different number of filters on MNIST, Fashion-MNIST and CIFAR 10, respectively. For the GAN architecture, we leverage the architecture of generator and discriminator in the tradition conditional GANs (Mirza & Osindero, 2014). To guarantee the convergence of RCGAN-U, we replace Batch Normalization with Instance Batch Normalization. The latent space dimensions of generator are 128, 128, 256 for the three datasets, respectively. As for the optimization of GAN, we deploy the avenue same as WGAN-GP (Gulrajani et al., 2017) to pursue desirable generation quality. Specifically, we set update step of discriminator as 1.
Choice of Hyper-parameters We choose κ as 0.75, β as 5.0 and λ = 0.99 across all the approaches. The learning rates of PU classifier and CGAN are 0.001 and 0.0001, respectively. In the alternate minimization process, we set the update step as 1 for PU classifier after updating the CGAN, and L0 as 5 in Algorithm 1. We used the same and sufficient epoch for all settings (180 epochs for joint optimization) to guarantee the convergence as well as for fair comparisons.
Further Evaluation of CGAN-P and Ours from the Aspect of Inception Score To better verify our approach can generate more pleasant images than CGAN-P, we additionally compare the Inception Score these two methods attain. Specifically, we trained a (almost) perfect classifier with 99.21 % and 91.33% accuracy for MNIST and Fashion-MNIST respectively. Then we generate 50,000 samples from the two approaches to compute Inception Score, the results of which are exhibited in Table 2. It turns out that our method attain the consistent superiority against CGAN-P on the Inception Score for MNIST, even though the generator label accuracy of these two approaches are comparable. Note that the two method obtains the similar Inception Score on Fashion-MNIST, but our strategy outperforms CGAN-P significantly from the perspective of generator label accuracy. Overall, we can claim that our method is better than CGAN-P.

heading: E APPENDIX: MORE IMAGES
text: We additionally show some generated images on other datasets generated by baselines and CNI-CGAN, shown in Figure 6. Note that we highlight the erroneously generated images with red boxes. Specifically, on FashionMNIST our approach can generated images with more accurate labels compared with CGAN-A and RCGANU. Additionally, the quality of generated images from our approach are much better than those from CGAN-P that only leverages limited supervised data, as shown in Figure 7 on CIFAR-10.

Review summaries:
This paper proposed the combination of two techniques for improved learning with unlabelled data: 1)  Positive-Unlabelled (PU) classifier, and 2) class-conditional GAN (cGAN). The idea is that the PU classifier can help produce more accurate pseudo labels for training of a cGAN, and with the improved cGAN, the generated images can be used in turn to further improve the PU classifier. The idea looks interesting and the empirical results verified its effectiveness. 

This paper targets at relieving the massive labeled data consumption of deep learning through the framework of semi-supervised learning. In particular, it finds out that two training approaches, Positive-Unlabeled classification and the conditional generation, can benefit each other. Jointly conducting these two approaches can push better performance on both tasks, thus eventually achieving better performance with a limited amount of labeled data. The authors combined the two tasks with a new type of GAN network. They further gave the corresponding theoretical proof for this new GAN model and verified its performance on the benchmark datasets.