Classify and Generate Reciprocally: Simultaneous Positive-Unlabelled Learning and Conditional Generation with Extra Data
Authors: Bing Yu
Ke Sun
He Wang
Zhanxing Zhu
Zhouchen Lin
Venue: ICLR 2022 Submitted (2022)
Keywords: PU learning
Robust Generative Models
Lable noises

Abstract: The scarcity of class-labeled data is a ubiquitous bottleneck in a wide range of machine learning problems. While abundant unlabeled data normally exist and provide a potential solution, it is extremely challenging to exploit them. In this paper, we address this problem by leveraging Positive-Unlabeled~(PU) classification and the conditional generation with extra unlabeled data \emph{simultaneously}, both of which aim to make full use of agnostic unlabeled data to improve classification and generation performance. In particular, we present a novel training framework to jointly target both PU classification and conditional generation when exposing to extra data, especially out-of-distribution unlabeled data, by exploring the interplay between them: 1) enhancing the performance of PU classifiers with the assistance of a novel Conditional Generative Adversarial Network~(CGAN) that is robust to noisy labels, 2) leveraging extra data with predicted labels from a PU classifier to help the generation. Our key contribution is a Classifier-Noise-Invariant Conditional GAN~(CNI-CGAN) that can learn the clean data distribution from noisy labels predicted by a PU classifier. Theoretically, we proved the optimal condition of CNI-CGAN and experimentally, we conducted extensive evaluations on diverse datasets, verifying the simultaneous improvements on both classification and generation. 

Under review as a conference paper at ICLR 2022
CLASSIFY AND GENERATE RECIPROCALLY: SIMULTANEOUS POSITVE-UNNABELLED LEARNING AND CONDITIONAL GENERATION WITH EXTRA DATA
Anonymous authors Paper under double-biee blind review
ABSTRACT
The scarcity of class-labeled data is ubiquitous bottleneck in wide range of machine learning problems. While abundant unlabeled data normally exist and provide potential solution, it is extremely challenging to exploit them. In this pa- per, we address this problem by leveraging Positive-Unlabeled (PU) classification and the conditional generation with extra unlabeled data sinultaneously, both of which aim to make full use of agnostic unlabeled data to improve classification and generation performance. In particular, present novel training framework jointly target both PU classification and conditional generation when exposing to extra data, especially catoo--distribution unlabeled data, by exploring the interplay between them: 1) enhancing the performance of PU classifiers with the assistance of novel Conditional Generative Adversarial Network (CGAN) that robust noisy labels, 2) leveraging extra data with predicted labels from PU classifier to help the generation. Our key contribution is Classifier-Noofs Moise-Invariant Con- ditional GAN (CNI-CGAN) that can learn the clean data distribution from noisy labels predicted by a PU classifier. Theoretically, we proved the optimal condi- tion of CNI-CGAN and experimentally, conducted extensive evaluations diverse datasets, verifying the simultaneous improvements on both classification and generation.
INTRODUCTION
Existing machine learning methods, particularly deep learning models, typically require big data to pursue remarkable performance. For instance, conditional deep generative models are able to generate high-fidelity and diverse images, but they have to rely on vast amounts labeled data (Lu-) cic al. 2019). Nevertheless, it is often laborious or impractical to collect large-scale accurate class-labeled data real-world scenarios, and thus the label scarcity is ubiquitous. Under such cir- cumstances, the performance of classification and conditional generation (Mirza & Osindero 2014) drops significantly (Lucic et al.) 2019). At the same time, diverse unlabeled data are available in enormous quantities, and therefore key issue is how to take advantage of the extra data enhance the conditional generation or classification.
Within the unlabeled data, both in-distribution and out-of-distribute data exist, where in- distribution data conform to the distribution of the labeled data while out-of-distrrbution data do not. Our key insight is harness the ditor--distributionn data. In the generation with extra data, most related works focused on the in-distribution data (Lucic et al. 2019; Gui er al., 2020: Donahue & Simonyan| 2019). When it comes the -utoo--distriuutin data, the majority existing meth- ods Noguchi & Harada 2019: Yamaguchi et al., 2019; Zhao et al., (020) attempted to forcibly train generative models on large amount of unlabeled data, and then transferred the learned knowledge of the pre-trained generator to the in-distribution data. In classification, a common setting to utilize unlabeled data semi-supervised learning (Miyato et al.) 2018 Sun et al. 2019 Berthelot et al.) 2019), which usually assumes that the unlabeled and labeled data come from the same distribution, ignoring their distributional mismatch. In contrast, Positive and Unlabeled (PU) Learning (Bekker & Davis 2020 Kiryo al. 2017) is an elegant way of handling this under-studied problem, where a model has the only access to positive samples and unlabeled data. Therefore, it is possible to utilize pseudo labels predicted by PU classifier unlabeled data guide the conditional gen-

---

Under review as a conference paper at ICLR 2022
eration. However, the predicted signals from the classifier tend to be noisy. Although there are flurry of papers about learning from noisy labels for classification Tsunng Wei Tsai 2019 Ge et al, Guo et al. 2019), to our best knowledge. work has considered to leverage the noisy labels seamlessly in the joint classification and generation. Additionally, another work (Hou et al. 2018) leveraged GANs recover both positive and negative data distribution step away from overfit- ting, but they never considered the noise-invariant generation or their mutual improvement. The generatie-discrimnattve complementary learning (Xu al., 2019) was investigated in weakly su- pervised learning, but we are the first attempt to tackle the (Multi-) Positive and Unlabeled learning setting while developing the method of moise-invariant generation from noisy labels. Please refer to Section 5 for the discussion about more related works.
2020
In this paper, we focus on the mutual benefits of conditional generation and PU classification, when we are only accessible to little class-labeled data, but extra unlabeled data, including out- of-distribution data, can be available. Firstly, a parallel non-negative multi-class PU estimator is derived to classify both the positive data of all classes and the negative data. Then we design Conditional Generative Adversarial Network (CNI-CGAN) that is able to learn the clean data distribution on all unlabeled data with noisy labels provided by the PU clas- sifier. Simultaneously, we also leverage our CNI-CGAN to enhance the performance of the PU classification through data augmentation, demonstrating a reciprocal benefit for both generation and classification. We provide the theoretical analysis on the optimal condition of our CNI-CGAN and conduct extensive experiments to verify the superiority of our approach.
2
OUR METHOD
2.1
POSITIVE-UNLABELA LEARNING
Traditional Binary Positive- Unllabbledd Problem Setting Let X R and Y E {+1} be the input and output variables and p(x, y) is the joint distribution with marginal distribution (a(ar) = p(=:YY= +1) and p.) (a) = p(x = -1). In particular, we denote p(x) as the distribution of unlabeled data. np. The and n,, are the amount of positive, negative and unlabeled data, respectively.
Parallel Non-Negative PU Estimator Vanilla PU learning (Bekker & Davis, 2020; Kiryo et al. 2017 Du Plessis et al. 2014 2015) employs unbiased and consistent estimator. Denote ga Ra R as the score function parameterized by 0, and f R x [=1] - R as the loss function. The risk of go can be approximated by its empirical version denoted as Rpn (ge):
(9o R (go) TT R, (90);
where nr represents the class prior probability, i.e. Tp P(Y +1) with TTP + T. = 1. In addition, 1 Rt (go) Ee l (go (x.), +1) and R. (go) = n. E (go (x.²), -1).
As negative data x" are unavailable, common strategy is to offset RT (go). We also know that (x) p(x)- - (x),(x), and hence RR (go) R2 (ga) - Rp (go). Then the resulting unbiased risk estimator (go can be formulated as: (go) = T (ge) Tp (go) + (ga);
(2)
where (9)) =
ny E2 l (go (x)( (1)),-1 and R. (ge) 1 En l (ge -1). The advan-
tage of this unbiased risk minimizer is that the optimal solution can be easily obtained if is linear in 6. However, in real scenarios we tend to leverage more flexible models go. e.g., deep neural networks. This strategy will push the estimator to point where it starts suffer from overfit- ting. Hence, we decide to utilize non-negative risk (Kiryo et al. 2017) for our PU learning, which has been verified in (Kiryo et al.) 2017) to allow deep neural network to mitigate overfitting. The non-negative PU estimator is formulated as:
Rp = xp R (go) + max {0, R, (go) u (90)
(3)
In pursue of the parallel implementation of Rpu(go), we replace max f0, Ri (go) - T N. (mw)
with its lower bound 1 Σ max (0, R. X2) - Tp R (go; x;)} where Xi and X denote as the unlabeled and positive data in the i-th mini-batch, and N is the number of batches.
N
2

---

Under review as a conference paper at ICLR 2022
From Binary PU to Multi-PU Learning Previous PU learning focuses learning classifier from positive and unlabeled data, and cannot easily be adapted to K + 1 and Unlabeled learning (Xu et al. 2017) was ever developed, but the proposed algorithm may not allow deep neural networks. Instead, extend binary PU learning multi-class version in straightforward way by addition- ally incorporating cross entropy loss on all the positive data with labels for different classes. More precisely, we consider the K + 1-class classifier fo as a score function fo (z)....... So +X+11 (x)). After the softmax function, we select the first K positive data to construct cross-entropy loss eCv, i..,, (fo())( = log E exp (f(())) - (z) where y E [K]. For the PU loss, we consider the composite function h([o(z))) Rdd - where h(.)) conducts a logit transforma- tion on the accumulative probability for the first K classes, i.c., h(fo(z)) In( in which
tasks where
K represents the number of classes in the positive data. Multi-Positte
p= 2j=1 f=1 exp (fd(x)) Eز exp (fz(m)) The final mini-batch risk of our PU learning can be presented as:
Roo = R Rt (h(fo): + max {o. (M) + RCE (fo: Xz),
where TC X²) E. pcE (So (x))
2.2 MANSSIFIER-NOISEE INVARIANT CONDITIONAL GENERATIVE ADVERSARIAL NETWORK (CNI-CGAN)
To leverage extra data, i.e., all unlabeled data, deploy our condi-
to benefit the generation, tional generative model on all data with pseudo labels predicted by our PU classifier. However, these predicted labels tend to be noisy, reduc- ing the reliability of the supervision signals and thus worsening the performance of the condi- tional generative model. Besides, the noise de- pends on the accuracy of the given PU classi- fier. To address this issue, we focus on devel- oping novel noise-invariant conditional GAN that is robust to noisy labels provided by a spec- ified classifier, e.g. PU classifier. We call our method Classifier Invariant Conditional Generative Adversarial Network (CNI-CGAN) and the architecture is depicted in Figure [ In the following, we elaborate on each part of
Xy
PU
PU(x))
D
Xg
G
y
PU
PU
y
Principle of the Design of CNI-CGAN
Albeit being noisy, the pseudo labels given by the PU classifier still provide rich information that we can exploit. The key take the noise generation mechanism into consideration dur- ing the generation. We denote the real data as Fr and the predicted hard label through the PU classifier as (x,(x,), i.e., (U. arg max; f/ (x,,, as displayed in Figure [ We let the generator "imitate" the noise generation mechanism to generate pseudo labels for the labeled data. With both pseudo and real labels, can leverage the PU classifier fe to estimate confusion matrix Č' model the label noise from the classifier. During the generation, a real label y, while being fed into the generator G. will also be polluted by Ö to compute noisy label ÿ. which then will be combined with the generated fake sample 2g for the following discrimination. Finally, the discriminator D will distinguish the real samples [x,, (x, out of fake samples [zg. 7. Overall, the noise "generation" mechanism from both sides can be balanced.
Figure 1: Model architecture of our Classifier- Noise-Invariant Conditional GAN (CNI-CGAN). The output Hg of the conditional generator G is paired with a noisy label ÿ corrupted by the PU- dependent confusion matrix Ò. The discriminator D distinguishes between whether given labeled sample comes from the real data (I,, PUg (x,)) or generated data (xg, 7).
3

---

Under review as a conference paper at ICLR 2022
Estimation of Ć The key in the design of Č is to estimate the label noise of the pre-trained PU classifier by considering all the samples of each class. More specifically, the confusion matrix C is k + 1 by k ++1 and each entry Čs represents the probability of generated sample zg. given label i, being classified class j by the PU classifier. Mathematically, we denote Ciz as:
C. (() jly i) E, (ts.))))))))))
(5)
where Tgg = G( = 1) and I is the indicator function. Owing to the stochastic optimization
nature when training deep neural networks, incorporate the estimation of Č in the processing of training by Exponential Moving Average (EMA) method. This choice can balance the utilization
of information from previous training samples and the updated PU classifier to estimate C. We formulate the update of ((+1) in the (-th mini- as follows:
(1(++) = AC + (1 - X)
(6)
where A denotes the incremental change of C on the current I-th mini-batch data X via Eq. is the averaging coefficient in EMA.
Theoretical Guarantee of Clean Data Distribution Firstly, we denote (zz) as the oracle class of sample x from an oracle classifier (-). Let Tे, = I,..., K +++, 1, be the class-prior probability of the class in the multi-positive unlabeled setting. Theorem I proves the optimal condition of CNI-CGAN to guarantee the convergence to the clean data distribution. The proof is provided in Appendix A
Theorem 1. (Optimal Condition of CNI-CGAN Let P9 be a probabilistic transition matrix where Pi = (((((1g) = -ily 1) indicates the probability of sample Iy with the oracle label j generated by G with the initial label i. We assume that the conditional sample space of each class is disjoint with each other, then
(1) 69 is a permutation matrix if the generator G in CNI-CGAN is optimal, with the permutation, compared with an identity matrix, only happens on rows r where corresponding Tr, E r are equal.
(2) If Ps is an identity matrix and the generator G in CNI-CGAN is optimal, then p" y) pa y) where p" (z, y) and (2, y) are the real and the generating joint distribution, respectively.
Briefly speaking. CNI-CGAN can learn the clean data distribution if p9 is an identity matrix. More importantly, the method we elaborate till now has already guaranteed Pg as a permutation matrix, which is very close to an identity one. We need an additional constraint, although the permutation happens only when same class-prior probabilities exist.
The Auxiliary Loss The optimal G in CNI-CGAN can only guarantee that p (T y) is close to p" (I. y) as the optimal permutation matrix Pa is close to the identity matrix. Hence in practice, to ensure that we can exactly learn an identity matrix for Pa and thus achieve the clean data distri- bution, we introduce an auxiliary loss to encourage a larger trace of Pa, i.c.. >K+1 P(O(x,) = i)ly i). As O-)) is intractable, we approximate it by the current PU classifier PUg(zg). Then obtain the auxiliary loss laux:
K+1 Σ
Laux(2, v) max(s- K
E. (((
where к (0, 1) is hyper-parameter With the support of auxiliary loss, Pa has the tendency to converge to the identity matrix where CNI-CGAN can learn the clean data distribution even in the presence of noisy labels.
Comparison with RCGAN (Thekumparampil et al., 2018; Kaneko et al., 2019) The the- oretical property of CNI-CGAN has major advantage over existing Robust CGAN (RC- GAN) TTekumparampii et al. 2018 Kaneko et al.) 2019), for which the optimal condition can only be achieved when the label confusion matrix known a priori. Although heuristics can be employed, such as RCGAN-U (Thekumparampil et al., 2018), to handle the unknown label noise setting, these approaches still lack the theoretical guarantee converge to the clean data distribution.
To guarantee the efficacy of our approach, one implicit and mild assumption is that our PU classifier will not overfit the training data, while our non-negative estimator helps ensure that it as
4

---

Under review as a conference paper at ICLR 2022
explained in Section 2.1 To further clarify the optimization process of CNI-CGAN, elaborate the training steps of D and G, respectively.
D-Step: We train D on adversarial loss from both the real data and the generated (Zg. W), where y is corrupted by Č. Cy denotes the y-th row of Ć. We formulate the loss of D
max (x( (U,()) + DEFamp(z)
E
[6(1 (( v).
where FF family of discriminators and Pz the distribution of latent space vector 2, e.g., Normal distribution. Py is a discrete uniform distribution on [K' + I] and @ is the measuring function.
G-Step: We train G additionally on the auxiliary loss Laux (z, v) as follows:
min GEG
I
[o(1 - D(G(z v). 5)) + Blaux u)],
Pu Po.yooP gla
where 8 controls the strength of auxiliary loss and G is family of generators. In summary, our CNI-CGAN conducts + classes generation, which can be further leveraged to benefit the K + PU classification via data augmentation.
Algorithm Alternating Minimization for Learning and Classifier-Noise-Invvria Generation.
Input: Training data (Ap, Xx). Batch size M and hyper-parameter > 0, A, (0, 1). Lo and L N Initializing (1)) identity matrix. Number of batches N during the training.
Output: Model parameter for generator G, and 0 for the PU classifier S'o
1: Pre-train PU classifier fo"
2: for i 1 to N do
3: Update fe by descending its stochastic gradient of Rpa X() via Eq.
4: end for
5: repeat
6: ** Update CNI-CGAN #/
7:
f to L do Sample {Zi,..., ZM}, }. YM and {x1,..., XM} from Pz. Py and all training data,
8:
respectively, and then sample (f1..... Far} through the current (()) Then, update the discriminator D by ascending its stochastic gradient of
Σ 18(D(x., PU,()) 6(1 Di).
9:
Sample (z,,.., AM3 and (y,,...ya from Pz and Py, and then sample (f1,.., through the current O0)) Update the generator G by descending its stochastic gradient of
M
1
o0 3()) + 3lanx (y,, 2;)].
M
if Lo then Compute A
EM II via Eq. 5 and then estimate Ö' by ((+1) = x0 + A)
12:
end if end for
13:
14:
Update PU classifier via Data Augmentation *
Sample (21, ZM and (y,,..,, YMM
from Pz and Py, respectively, and then update the PU classifier fe by descending its stochastic gradient of
CB (fo (G(z,
M
16:
until convergence
5

---

Under review as a conference paper at ICLR 2022
3 ALGORITHM
Firstly, we obtain a PU classifier fo trained on multi-positive and unlabeled dataset with the par- allel non-negative estimator derived in Section 2.1] Then train CNI-CGAN, described in Section [.2] on all data with pseudo labels predicted by the pre-trained PU classifier. As our CNI- CGAN is robust noisy labels, we leverage the data generated by CNI-CGAN to conduct data augmentation to improve the PU classifier. Finally, we implement the joint optimization for the training of CNI-CGAN and the data augmentation of the PU classifier. We summarize the proce- dure in Algorithm| | and provide more details in Appendix C
Computational Cost Analysis In the implementation of our CNI-CGAN, we need to additionally estimate C, a (K + 1) (K + 1) matrix. The computational cost of this small matrix negligible compared with the updating of discriminator and generator networks, although the estimation of Č is crucial.
Simultaneous Improvement on PU Learning and Generation with Extra Data From the per- spective of PU classification, due to the theoretical guarantee from Theorem 1 CNI-CGAN is capa- ble of learning a clean data distribution out of noisy pseudo labels predicted by the pre-trained PU classifier. Hence, the following data augmentation has the potential to improve the generalization of PU classification regardless of the specific form of the PU estimator. From the perspective of generation with extra data, the predicted labels unlabeled data from the PU classifier can provide CNI-CGAN with more supervised signals, thus further improving the quality of generation. Due to the joint optimization, both the PU classification and conditional generative models able to improve each other reciprocally, as demonstrated in the following experiments.
4 EXPERIMENT
Experimental Setup We perform our approaches and several baselines on MNIST, Fashion-MNIST and 10. We select the first 5 classes MNIST and 5 non-clothes classes Fashion- MNIST, respectively, for K +1 classification (K = 5). To verify the consistent effectiveness of our method in the standard binary PU setting, we pick the categories of transportation tools in CIFAR- 10 as the one-class positive dataset. As for the baselines, the first is CGAN-P, where Vanilla CGAN (Mirza & Osindero 2014) is trained only on limited positive data. Another natural baseline is CGAN-A where Vanilla CGAN is trained on all data with labels given by the PU classifier.
Gencrator Label (ecuracy(%) (%))
MNIST
Generator Label Accuracy(P% (%)
Pashion-MNIST
CIFAR-10
45-
88-
Inception Score
4n-
60-
80
3.5
3.0
40-
OGAN-P OGANA
50
CGAN-P
25
CGAN-P CGAN A
CGAN
20
Ours
38
Oun
Oun
15
10"
101
MNIST
Fashion-MNIST
CIFAR-10
97.5
85
PU Accurncy(5)
v0-
PU Accurncy(5)
35.0
PU Accuracy(1))
88
98.5
80
90.0
E
75-
Original PU
$7.5
Original PU
Original CGAN A
76-
CGAN
$5.0
CGAN
65-
Oues
Oues
Owrs
10a
16
Positive Rate
Positive Rate
Positive Rate
Generation and classification performance of CGAN-P, CGAN-A and Ours on three datasets. Results of CGAN-P (blue lines) on PU accuracy do not exist since CGAN-P generates only K classes data rather than K + 1 categories that the PU classifier needs.
Figure 2:

---

Under review as a conference paper at ICLR 2022
The last baseline is RCGAN-U (Thekumparampil et al. 2018 where the confusion matrix is totally learnable while training. For fair comparisons, we choose the same GAN architecture. Through a line search of param--armettess,
choose к as 0.75, 5.0 and À 0.99 across all the datasets. We set Lo as in Algorithm [ More details about hyper-parameters can be found in Appendix [
Evaluation Metries For MNIST and Fashion-MNIST, racy (Thekumparmmpl et al.] 2018) and PU Accuracy to evaluate the quality of generated images. Generator Label Accuracy compares specified y from CGANs to the true class of the generated ex- amples through a pre-trained (almost) oracle classifier f. In experiments, we pre-trained two K + classifiers with 99.28 99.28% and 98.23% accuracy on the two datasets, respectively. Additionally, the increased PU Accuracy measures the closeness between generated data distribution and test (almost real) data distribution for the PU classification, serving as a key indicator to reflect the quality of generated images. For CIFAR 10, we use both Inception Score (Salimans et al. (016) to evaluate the quality of the generated samples, and the increased PU Accuracy quantify the improvement of generated samples on the PU classification.
mainly
Generator Label Accu-
4.1 GENERATION AND CLASSIFICATION PERFORMANCE
We set the whole training dataset as the unlabeled data and select certain amount of positive data with the ratio of Positive Rate. Figure F the trend of Generator Label Accuracy, Inception Score and PU Accuracy as the Positive Rate increases. It turns out that CNI-CGAN outperforms CGAN-P and CGAN-A consistently especially when the Positive Rate is small, i.e. little positive data. Remarkably, our approach enhances the PU accuracy greatly when exposed to low positive rates, while CGAN-A even worsens the original PU classifier sometimes in this scenario due to the existence of too much label noise given by less accurate PU classifier. Meanwhile, when more pervised positive data are given, the PU classifier generalizes better and then provides more accurate labels, conversely leading to more consistent and better performance for all methods. Besides, note that even though the CGAN-P achieves comparable generator label accuracy on MNIST, it results in lower Inception Score. We demonstrate this in Appendix D
Table I: PU classification accuracy of RCGAN-U and Ours across three datasets. Final PU accuracy represents the accuracy of PU classifier after the data augmentation.
Final PU Accuracy Positive Rates (%)
0.2%
0.5%
1.0%
10.0%
Original PU
68.86
76.75
86.94
95.88
MNIST
RCGAN-U
87.95
95.24
95.86
97.80
Ours
96.33
96.43
96.71
97.82
Original PU
80.68
88.25
93.05
95.99
Pashion-MNISS
RCGAN-U
89.21
92.05
94.59
97.24
Ours
89.23
93.82
95.16
97.33
Original PU
76.79
80.63
85.53
88.43
CIFAR-10 10
RCGAN-U
83.13
86.22
88.22
90.45
Ours
87.64
87.92
88.60
90.69
To verify the advantage of theoretical property for our CNI-CGAN, we further compare it with RCGCN-U (Thekumparampil et al. 2018; Kaneko et al. 2019), the heuristic version of robust gen- eration against unknown noisy labels setting without the theoretical guarantee of optimal condition. As observed in Table D our method outperforms RCGAN-U especially when the positive rate is low. When the amount of positive labeled data is relatively large, e.g., 10.0%, both our approach and RCGAN-U can obtain comparable performance.
Visualization To further demonstrate the superiority of CNI-CGAN compared with the other base- lines, we present some generated images within + 1 classes from CGAN-A, RCGAN-U and CNI- CGAN on MNIST, and high-quality images from CNI-CGAN on Fashion-MNIST and CIFAR-IO, in Figure 3 In particular, we choose the positive rate as 0.2% on MNIST, yielding the initial PU classifier with 69.14% accuracy. Given the noisy labels on all data, our CNI-CGAN can generate more accurate images of each class visually compared with CGAN-A and RCGAN-U. Results of Pashion-MNIST and comparison with CGAN-P on CIFAR-10 10 can refer to Appendix
7

---

Under review as a conference paper at ICLR 2022
MNIST: Positive Rate 0.2%, Initial PU: 69.14%
CNI-CGAN
CGAN-A
RCGAN-U
CNI-CGAN
Fashion-MNIST
CIFAR-I 10
0
O
0
0
0
0 0 0
O
O
O
O
D
/
1
1
1 2 S 4
/
/ 2 3
/ 2 3 4
5 7
1881
7
6 4 3
/
2
2 9
K 3
7 6 4 9
7
2
2 3
2 3 4
2 3 4 8
1
5 2
3
S 4
3
7
6
2
37
4
4 4
4
4
4
4
6
9
9
7
5
4
7
6
9
5
Generator Label Accuracy
39.67%
81.58%
96.33%
Figure 3: Visualization of generated samples three datasets. Rows below the red line represent the negative class. We highlight the erroneously generated images with red boxes on MNIST.
4.2 ROBUSTNESS OF OUR APPROACH
Robustness against the Initial PU accuracy The auxiliary loss can help the CNI-CGAN to learn the clean data distribution regardless of the initial accuracy of PU classifiers. To verify that, we select distinet positive rates, yielding the pre-trained PU classifiers with different initial accuracies. Then we perform our method based on these PU classifiers. Figure 4 suggests that our approach can still attain the similar generation quality under different initial PU accuracies after sufficient training, although better initial PU accuracy can be beneficial to the generation performance in the early phase.
GAN; Generator Label Accuracy (%)
MNIST
GAN; GAN: Generator Label Accuracy (%)
Fashion MNIST
CIFAR 10
100
100
90-
88
80
PU Accuracy(%)
86-
60
60
84
40
40-
82
Initial PU 77.87%
Initial PU 88.27%
Initial PU 79.49%
Initial PU 84.57%
20
Initial PU 91.03%
Initial PU 82.51%
Initial PU 91.28%
Initial PU 94.02%
Initial PU 85 45%
10
10'
10"
10
10
10
10"
10" Number of Training Iterations
10"
10"
Number of Training Iterations
Number of Training Iterations
Figure 4: Tendency of generation performance as the training iterations increase on three datasets.
Robustness against the Unlabeled data In real scenarios, we are more likely to have little knowl- edge about the extra data we have. To further verify the robustness of CNI-CGAN against the unknown distribution of extra data, we test different approaches across different amounts and dis- tributions of the unlabeled data. Partieularly, we consider two different types of distributions for unlabeled data. Type is K+I K+I K+I where the number of data in each class, including
1 where the negative data makes up half of the negative data, is even, while type 1 2.. ......... 2K 2.
all unlabeled data. In experiments, we focus on the PÚ Accuracy to evaluate both the generation quality and the improvement of PU learning. For MNIST, we choose 1% and 0.5% for two settings while opt for 0.5% and 0.2% both Fashion-MNIST and CIFAR-10.
Figure manifests that the accuracy of PU classifier exhibits a slight ascending tendency with the increasing of the number of unlabeled data. More importantly, our CNI-CGAN almost consistently outperforms other baselines across different amount unlabeled data as well as distinct distributions of unlabeled data. This verifies that the robustness of our proposal to the distribution of extra data can be maintained potentially. We leave the investigation on the robustness against more imbalanced situations as future works.
8

---

Under review as a conference paper at ICLR 2022
Distribution Type PU Auet%)) Distriution Type I: Avot%)
100
MNIST
Distrihuton Type 1: Acd(50)
Fashion-MNIST
Distributio Typu 1: Acce
100
CIFAR-10
90-
95
93
90
90
80
85
85-
70
80
80
.21A3-
200A344
75
PSANA
RCGANA
CHANLS
BOGAN4
Can
10'
100
MNIST
Distribution Type 2: PU Acoth))
100
Fashion MNISS
Distributtio Type 2: PU Acce%)
100
CIFAR-10
95
95
.6-
90-
90-
50-
85
85-
Cágini KCCANN
30
75
73
Das
75
70
70
10"
10
Number of Unlabeled Data
Number Unlabeled Data
Number of Unlaboled Data
Figure 5: PU Classification accuracy of CGAN-A, RCGAN-U and Ours after joint optimization across different amounts and distribution types of unlabeled data.
5 RELATED WORKS
Positive-Unlabeled (PU) Learning. Positive and Unlabeled (PU) Learning is the setting where a learner has only access to positive examples and unlabeled data (Bekker & Davis, 2020 Kiryo
iet al. 2017). One related work (Hou et al. 2018) employed GANs (Goodfellow et al. 2014 4) to recover both positive and negative data distribution step away from overfitting. Kato et al. (Kato iet al. 2018) focused on remedying the selection bias in the PU learning. Besides, Multi-Positive and Unlabeled Learning (Xu et al., (017) extended the binary PU setting to the multi-class version, therefore adapting to more practical applications. By contrast, our multi-posinn unlabeled method absorbs the advantages of previous approaches, and in the meanwhile intuitively extends them to fit the differential deep neural networks optimization.
To attain high figh-quality images with both fidelity and di- versity, the training of generative models requires large dataset. To reduce the need of huge amount of data, the vast majority of methods (Noguchi & Harada 2019 Yamaguchi et al. 2019 Zhao et al., 2020) attempted to transfer prior knowledge of the pre-trained generator. Another branch (Lucic et al. 2019) is to leverage self- and supervised learning to add pscudo labels on the in-distribution unlabeled data in order to expand labeled dataset. Compared with this approach, our strategy can be viewed to automatically "pick" useful in-distribution data from total unknown unlabeled data via PU learning framework, and then constructs robust conditional GANs to generate clean data distribution out of predicted label noise. Please refer to more related works in Appendix B
Conditional GANs on Few Labels Data.
6 DISCUSSION AND CONCLUSION
In this paper, we proposed a new method, CNI-CGAN, to jointly exploit PU classification and conditional generation. is, to our best knowledge, the first method of such kind to break the ceiling of class-label scarcity. by combining two promising yet separate methodologies to gain massive mutual improvements. CNI-CGAN can learn the clean data distribution from noisy labels given by a PU classifier, and then enhance the performance of PU classification through data augmentation in various settings. We have demonstrated, both theoretically and experimentally, the superiority of our proposal on diverse benchmark datasets in exhaustive and comprehensive manner. In the future, it will be promising to investigate learning strategies on imbalanced data, e.g., cost-sensitive learning (Elkan 2001), to extend our approach to broader settings, which will further cater to real- world scenarios where highly unbalanced data are commonly available. In addition, the leverage of soft labels in the design of CNI-CGAN is also promising.

---

Under review as a conference paper at ICLR 2022
Ethics Statement. Our designed CNI-CGAN framework interplay with the PU classification and robust generation. which can mitigate the scarcity of class-labeles data. Leveraging extra data may correlate with the privacy issue as the privacy issue still exists in generative models. Thus, a privacy-gune -uracc-guarrnteed version our algorithm can be further proposed in the future to handle the potential privacy issue.
Reproducibility Statement. For the theoretical part, we clearly state the related assumption and detailed proof process in Appendix In terms of the algorithm, implementation is directly adapted from the public one of generative models and PU learning.
REFERENCES
Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey. Machine Learning. 109(4):719-700 2020.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach semi-supervised learning. Advances in Neural Information Processing Systems, pp. 5050-5060, 2019.
Grigorios G Chrysos, Jean Kossaifi, and Stefanos Zafeiriou. Robust conditional generative adver- sarial networks. arXiv preprint 180: 086577. 2018.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Advances in Neural Information Processing Systems, pp. 10541-10551, 2019.
Marthinus Du Plessis, Gang Niu, and Masashi Sugiyama. Convex formulation for learning from positive and unlabeled data. In International conference machine learning, pp. 1386-1394, 2015.
Marthinus Du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. In Advances in neural information processing systems, pp. 703-711, 2014.
Charles Elkan. The foundations of cost-sensitive leaming. In International joint conference on artificial intelligence, volume 17, pp. 973-978. Lawrence Erlbaum Associates Ltd, 2001.
Yixiao Ge, Dapeng Chen, and Hongsheng Li. Mutual mean-teaching: Pseudo label refinery for un- supervised domain adaptation on person re-identification. International Conference on Learning Representations, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- mation processing systems, pp. 2672-2680, 2014.
Jie Gui, Zhenan Sun, Yonggang Wen, Dacheng Tao, and Jieping Ye. A review on generative adver- sarial networks: Algorithms, theory, and applications. arXiv preprint arXiv:2001 00693,, 2020.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, pp- 5767-5777, 2017.
Tianyu Guo, Chang Xu, Boxin Shi, Chao Xu, and Dacheng Tao. Learning from bad data via gener- ation. In Advances in Neural Information Processing Systems, pp. 6042-6053, 2019.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. International Conference on Learning Representations, 2018.
Shohei Hido, Yuta Tsuboi, Hisashi Kashima, Masashi Sugiyama, and Takafumi Kanamori. Inlier- based outlier detection via direct density ratio estimation. In 2008 Eighth IEEE International Conference on Data Mining, pp. 223-232. IEEE, 2008.
Ming Hou, Brahim Chaib-draa, Chao Li, and Qibin Zhao. Generative adversarial learning. In Jérôme Lang (ed.). Proceedings the Twenth-Seventh International Joint Conference on Artificial Intelligence. IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pp. 2255 2255-2261 2261. ijcai.org, 2018. doi: 10.24963/jjcai.20188/ 2018/311.
10

---

Under review as a conference paper at ICLR 2022
Takuhiro Kaneko, Yoshitaka Ushiku, and Tatsuya Harada. Label-noise robust generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2467-2476, 2019.
Masahiro Kato, Takeshi Teshima, and Junya Honda. Learning from positive and unlabeled data with a selection bias. 2018.
Ryuichi Kiryo, Gang Niu, Marthinus du Plessis, and Masashi Sugiyama. Positive-unlabeled learning with non-negative risk estimator. In Advances in neural information processing systems, pp. 1675-1685, 2017.
Kiran Koshy Thekumparampil, Sewoong Oh, and Ashish Khetan. Robust conditional gans under missing or uncertain labels. arXiv preprint 1906v:9906003555 2019.
Mario Lucic, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, and Sylvain Gelly- High-fidelity image generation with fewer labels. nternational Conference on Machine Learning (ICML), 2019.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint 1rXiv:1411.1785 2014.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence,
1999119999993, 2018.
Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2750- 2758, 2019.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in Neural Information Processing Systems, 2016.
Alex Smola, Le Song, and Choon Hui Teo. Relative novelty detection. In Artificial Intelligence and Statistics, pp. 536-543, 2009.
Ke Sun, Bing Yu, Zhouchen Lin, and Zhanxing Zhu. Patch-level neighborhood interpolation: A general and effective graph-based regularization strategy. arXiv preprint arXiv: 1911399 2019.
Kiran K Thekumparampil, Ashish Khetan, Zinan Lin, and Sewoong Oh. Robustness of conditional gans to noisy labels. In Advances neural information processing systems, pp. 10271 2018.
Jun Zhu Tsung Wei Tsai, Tsung Wei Tsai. Countering noisy labels by learning from auxiliary clean labels. arXiv preprint arXiv: 1905. 13305, 2019.
Qin Wang, Wen Li, and Luc Van Gool. Semi-supervised learning by augmented distribution align- ment. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1466-1475, 2019.
Yanwu Xu, Mingming Gong, Junxiang Chen, Tongliang Liu, Kun Zhang, and Kayhan Batmanghe- lich. Generative-diserimnnatve complementary learning. AAAI 2020, 2019.
Yixing Xu, Chang Xu, Chao Xu, and Dacheng Tao. Multi-positive and unlabeled learning. In IJCAI, pp. 3182-3188, 2017.
Shin's ya Yamaguchi, Sekitoshi Kanai, and Takeharu Eda. Effective data augmentation with multi- domain learning gans. arXiv preprint 1rxiv:1912.115597, 2019.
Wei Li Shaogang Gong Yanbei Chen, Xiatian Zhu. Semi-supervised learning under class distribution mismatch. AAAI 2020, 2019.
Bing Yu, Jingfeng Wu, Jinwen Ma, and Zhanxing Zhu. Singent-normal adversarial regularization for semi-super learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10676-10684, 2019.
11

---

Under review as a conference paper at ICLR 2022
Miaoyun Zhao, Yulai Cong, and Lawrence Carin. On leveraging pretrained gans for limited-date generation. arXiv preprint arXiv:2002.11810, 2020.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX: PROOF OF THEOREM 1
Firstl, we recall some definitions. Denote Ty. 2y as the real training and generated samples, respectively. x are the population of all data, and 2y are sampled from )( Yo represents the initial labels for the generator G, while indicates the labels perturhed by C from Vo- The class-prior n, meets n, P(U, = i) = P(O(z+)
i). For rigorous proof Theorem we elahorate it again the appendix.
We assume that the following three mild assumptions can be met: (a) PU classifier is not over- fiting the training data, (b) P(PUs(x,) (zzg) Uo) ((P (c) the conditional sample space disjoint from each other class. Then,
Theorem [
(1) is a permutation matrix the generator with an identity matrix, only happens on rows r where conesponding Ry, E r are equal.
CNI-CGAN is optimal, with the permutation, compared
(2) If pd is an identity matrix and the generator G in CNI-CGAN is optimal, then p'(z, = p² v) where (2r, y) and p" (r, v) are the real and generating joint distribution, respectively.
A.1 PROOF OF (1)
Proof. For general setting, the oracle class of 2y given by label Vg is necessarily equal (Ug() Thus, we consider the oracle class of zg. i.c., Oxx,)) in the proof.
Optimal G. In CNI-CGAN, G is optimal if and only if
p' (U,(()) (xg 0).
(10)
The equivalence of joint probability distribution can further derive the equivalence of marginal distribution, i.c., p" (2+) pd (xg), We define probability matrix C where Ci5 ((PU((z) = (1((x)) where æ are the population data. According to (c), we can apply (()) on both x,- and xy in Eq. 10 Then we have:
P(O(z.) = PU.((,, j) @ ((O(=,) -;
P(O(z,) = 1)P((U)(1,) ( = ;10(x,) = =)
K+1
Σ
P(ys = k. k(0,)) i)P(o jlvo- k. ((zg) =
(11)
T, (c)
P(O(x,) = -1y = k)P( = k)P( = jlyy = k:)
K+1
C
pa Crss
where assumption (a) indicates that (Up( is close to (U) so that P(PUs(a.) = j10(xx) = i) P(PUs(z) = j[O(z) = i). Then the corresponding matrix form follows
1IC IIĆ
(12)
Definition, According to the definition Ò and Law Total Prohahility, we have:
P(y, ((PU)(()))) jly.
n
P(O(a,) = kly, = (() = =100x,,) (0)),,)= = k. =
K+1
(13)
ps -
K+1
n: O
Pa Ckjs
where the last equation is met as p(x,)) is close to p(r) when G' optimal, and thus P(PUp(#g) j1O(z,)
k)) = (() (z0(z) k)) Then consider the corresponding matrix form follows
I1 C
(14)
13

---

Under review as a conference paper at ICLR 2022
12 we have P "1 II, which indicates pa is a general orthogonal matrix. In addition, the element of p is non-negative and the sum of each row is 1. Therefore, we have pa is a permutation matrix with permutation compared with the identity matrix only happens on rows where corresponding Kr, r E r are equal. Particularly, if all яy are different from each other, then permutation operation will not happen, indicating the optimal conditional of ps is the identity matrix.
where II is the diagonal mairix of prior vector и. Combining Eq. 14 and
A.2 PROOF OF (2)
additionally denote y/ as the real label of real sample Tr, i.c., 2/- = (x+)). According to the optimal
condition of G in Eq. 10 we have p" (a,) = =(())) Since we have 15 is an identity matrix, then ((r,) Wy
a.e. Thus, we have p (2n [2 = 1) = pd (1ra) = i). Vi = 1, K + 1. According the assumption (c) and i). In addition, we know that p" (xx)) i)
Eq. 10. have p (x+10(es) = p (1g((((z)))
p (2 Byr i), thus we have p (zrlyr = -) p = 1). Further, we consider the identical class-prior
T1. Finally, we have
p' d' vr = i) p0(xx luo i)sTs p" (3--11 = ()x()(z))
p0(x ( Uv +))((s = +)
(15)
p² (x = ()p(y- = =) = p (solVo s)P(vo = i) p² (2r, Hr) = p" (Ir. Vo).
B APPENDIX: MORE RELATED WORKS
(PU) Learning, Positive and Unlaboled (PU) Learning is the setting where a learner has only access to positive examples and unlabeled data (Bekker & Davis 2020 Kiryo et al. 2017). One related work (Hou et al. 2018) employed GANs (Goodfellow al. (2114) to recover both positive and negative data distribution to step away from overfitting. Kato al. (Kato et al. 2018) focused on remedying the selection bias in the PU learning. Besides, Multi-Positive and Unlabeled Learning Xu et al. 2017) extended the binary PU setting to the multi-class version, therefore adapting to more practical applications. By contrast, our multi- positive unlabeled method absorbs the advantages of previous approaches, and in the meanwhile intuitively extends them fit the differential deep neural networks optimization.
Conditional GANs on Few Labels Data. To attain high-quality images with both fidelity and diversity,
the training of generative models requires a large dataset. To reduce the need of huge amount of data, the
vast majority of methods (Noguchi & Harada 2019 Yamaguchi et al., 2019 Zhao et al. (220)) attempted to transfer prior knowledge of the pre-trained generator. Another branch LLucic et al. (019)) to leverage self-
and supervised learning to add pseudo labels on the in-distribution unlabeled data in order to expand labeled dataset. Compared with this approach, our strategy can be viewed to automatically "pick" useful in-distribution data from total unknown unlabeled data via PU learning framework. and then constructs robust conditional GANs generate clean data distribution of predicted label noise.
Robust GANs. Robust Conditional GANs (Thkumparampil et al. 2018 Kaneko el al. 2019) were pro- posed to defend against class deasndeeenndntt noisy labels. The main idea of these methods is to corrupt labels of generated samples before feeding to the adversarial discriminator, forcing the generator to produce sample with clean labels. Another supplementary investigation (Koshy Thekumparampil et al. 2019) explored the scenario when CGANs get exposed to missing ambiguous labels, while another work (Chrysos et al.) 2018) leveraged the structure of the model in the target space to address this issue. In contrast, the noises in our model stem from the prediction error of a given classifier. We employ the imperfect classifier to estimate the label confusion noise, yielding a new branch of Robust CGANs against "classifier" label noises.
Sepe-Suuervised Learning (SSL). One crucial issue in SSL (Miyato et al. 2018 Yu et al. 2019 Sun
jet al. 2019p is how to tackle with the mismatch of unlaheled and labeled data, Augmented Distribution Align ment (Wag et al. (219)) was proposed to leverage adversarial training to alleviate the bias, but they focused on the empirical distribution mismatch owing to the limited number of labeled data. Further, Uncertainty Aware Self- Distillation (annbei Chen 2019) was proposed to concentrate on this under-studied problem. which guarantce the effectiveness of learning. In contrast, our approach leverages the PU learning to construct the "open world" classification.
Out-Or- cantOOrDistting Do-Dittribution (OOD) Detection OOD Detection is one classical but always vibrant machine learning problem. PÚ learning can he used for the detection of outliers in an unlabeled dataset with knowledge only from collection of inlier data (Hido al. 2008 Smola al. 2009 Another interesting and related
14

---

Under review as a conference paper at ICLR 2022
Table 2: Further evaluation of CGAN-P and Ours from the perspective of Inception Score MNIST and Fashion-MNIST datasets.
Positive Rates
0.75%
1.0%
3.0%
5.0%
10.0%
Inception Score (:: Standard Deviation)
CGAN-P
5.08
5.10-0.03
5.09:00022
5.14.10.03
5.10:10.04
MNIST
Ours
5.60=0.01
5.59=0.02
5.65=10.02
5.52+0001
5.63-t0.02
CGAN-P
4.95=0.03
5.01 0.03
5.04
5.02:10.04
5,00 =0.03
Mushion-MNIST
Ours
4.99 0.02
5.01 0.02
5.03:10.01
5.07 0.02
5.04 0.02
work is Outlier Exposure (Hendrycks et al. 2018), approach that leveraged auxiliary dataset to enhance the anomaly detector based on existing limited data. This problem is similar our generation task, the goal of which is to take better advantage extra dataset, especially outoo--distriuution data, to boost the generation.
Learning from Noisy Labels Rotational- SecoooaallDeuplinngggggg Consistency Regularization (RDCR) (Tsung Wei Tsai 2019) was designed to integrate the consistency-based methods with the sell-supervised rotation task to learn noise-tolerant representations. Mutual Mean-Tecan Mean-Teaching (Ge et al.] 2020) was proposed to refine the soft labels person re-identification task by averaging the parameters of two neural networks In addition, the data with noisy labels can also viewed as bad data. Another work (Guo et al. (2199) provided worst- case learning formulation from bad data, and designed a data generation scheme in an adversarial manner, augmenting data improve the current classifier.
C APPENDIX: DETAILS ABOUT ALGORITHM 1
2017 we utilize the sigmoid loss fag y) 1/(1 + exp(ty)) in the implementation of the PU learning. Besides, denote RT (g: Xi) Rp (9; x3) in the i-th mini-batch. Instructed by the algorithm (Kiryo et al. (2077). r; < we turn to optimize Var: in order to make this mini-batch less overfitting, which is slightly different from Eq.
Similar in Kiryo et al.
D APPENDIX: DETAILS ABOUT EXPERIMENTS
PU classifier and GAN architecture For the PU classifier, employ convolutional layers with dif- ferent numher of filters on MNIST, Fashion-MNIST and CIFAR 10, respectively. For the GAN architecture, we leverage the architecture generator and discriminator in the tradition conditional GANs (Mirza & Osin- dero 2014). To guarantec the convergence of RCGAN-U, we replace Batch Normalization with Instance Batch Normalization. The latent space dimensions of generator are 128, 128, 256 for the three datasets, respectively. WGAN-GP (Gulrajani al. 2017) pursue
As for the optimization of GAN, deploy the avenue desirable generation quality. Specifically, we set update step of discriminator
Fashion-MNIST: Positive Rate 0.3%, Initial PU: 85.41% CNI-CGAN
CGAN-A
RCGAN-U
Generator Label Accuracy
81.17%
94.95%
95.13%
Figure 6: Visualization generated samples from several baselines and ours on FaShion-MNIST.
15

---

Under review as a conference paper at ICLR 2022
CIFAR-I0: Positive Rate 0.3%, Initial PU: 79.46%
CGAN-P
CNI-CGAN
Figure 7: Visualization of generated samples from CGAN-P and ours on CIFAR-10.
Choice of Hyper- apper-prrametess We choose 0.75, 5.0 and À 0.99 across all the approaches. The learning rates of PU classifier CGAN are 0.001 and 0.0001, respectively. the alternate minimization process, we set the update step as I for PU classifier after updating the CGAN, and Lu as in Algorithm [] We used the same and sufficient epoch for all settings (180 opochs for joint optimization) to guarantee the convergence well for fair comparisons.
Further Evaluation of CGAN-P and Ours from the Aspect of Inception Score To better verify our approach can generate more pleasant images than CGAN-P, we additionally compare Inception Score these two methods attain. Specifically, we trained (almost) perfect classifier with 99.21 % and 91.33% accu- racy for MNIST and Fashion-MNIS" respectively. Then we generate 50,000 samples from the two approaches compute Inception Score, the results of which exhibited in Table 2 It turns out that the consistent superiority against CGAN-P P on the Inception Score MNIST, even though the generator label accuracy these two approaches are comparable. Note that the two method obtains the similar Inception Score on Fashion-MNIST, but our strategy outperforms CGAN-P significantly from the perspective of generator label accuracy. Overall, we can claim that our method is better than CGAN-P.
method attain
E APPENDIX: MORE IMAGES
We additionally show some generated images other datasets generated by baselines and CNI-CGAN, shown Figure Note that we highlight the erroncously generated images with red boxes. Specifically, on Fashion- MNIST our approach can generated images with more accurate labels compared with CGAN-A A and RCGAN- U. Additionally, the quality of generated images from our approach are much better than those from CGAN-P that only leverages limited supervised data, shown in Figure 7 on CIFAR-10.
16


Reviews:
[{"review_id": "wCRH7OFchYm", "reviewer": "Reviewer_XLhs", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem is practical and interesting. The paper generally is easy to follow and read. \n\nHowever, my major concern is the novelty of the paper, which is a combination of PU and GAN. It is incremental with limited novelty. \nThere are GAN model for semi-supervised learning. How about comparing with them? This paper only compares with two methods.\nWhat is the benefit of confusion matrix C to get noisy label?\n\n\n\n\n\n", "summary_of_the_paper": "This paper aims to make full use of agnostic unlabeled data to improve classification and generation performance by leveraging Positive-Unlabeled (PU) classification and the conditional generation with extra unlabeled data. They evaluate on some benchmarks in different positive label ratio. ", "summary_of_the_review": "Based on the novelty, I tend to reject the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qdsTlZUMj4Z", "reviewer": "Reviewer_NqyD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n\n- The combination of positive-unlabeled classification and conditional generation with extra unlabeled data is naturally and sounds reasonable. \n- The extensive experimental results indicate the effectiveness of the proposed framework on both classification and generation tasks.\n- The writing is good, with a well-organized structure, clear illustration, rigorous equations.\n\n## Cons\n\n- Lack generation experiments on high-pixel datasets such as ImageNet.\n- It should compare more SOTA PU and GAN methods.\n- The proposed looks complex compared to the baseline methods, contains multiple components such as PU, G, C, D. \n\n", "summary_of_the_paper": "This paper proposes a unified framework to leverage Positive Unlabeled classification and conditional generation with extra unlabled data and benefits the two tasks simultaneously.\nIt also proved the optimal condition of the proposed Classifier-Noise-Invariant Conditional GAN theoretically.\n", "summary_of_the_review": "The proposed training framework which jointly targets PU classification and conditional generation is novel and sound.\nBut the novelty is limited and the lack some key experiments to show the improvements.\nSo I tend to reject the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BR2DTkl-7ZX", "reviewer": "Reviewer_EFMY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The pros and cons of the paper are described below.\n\n## Pros ##\nBridging different generalization methods is a common technique to improve the performance of deep learning, as verified by much previous work. However, it seems to be the fact that utilizing PU and conditional generation is the unique contribution of this work. Such a new perspective does provide new insights to the community and can motivate future research in the field of semi-supervised learning. Meanwhile, combing PU and conditional generation is not an easy task. To this end, the authors designed the model of CNI-CGAN, which introduces several novel designs. The theoretical proof also seems reasonable.\n\n## Cons ##\nFor the empirical evaluation part, the authors only verified their method on several relatively small datasets like MNIST and CIFAR. However, the main claim of this paper is that they can relive the massive labeled data consumption of training deep networks. Thus, it is important to verify the method on some relatively large datasets like ImageNet. The reviewer understands that there may not be a proper data source of unlabeled data or an out-of-distribution data source for ImageNet. However, this fact also suggests that the semi-supervised learning method this paper puts forward may not be easily implemented in practice.\n", "summary_of_the_paper": "This paper targets at relieving the massive labeled data consumption of deep learning through the framework of semi-supervised learning. In particular, it finds out that two training approaches, Positive-Unlabeled classification and the conditional generation, can benefit each other. Jointly conducting these two approaches can push better performance on both tasks, thus eventually achieving better performance with a limited amount of labeled data. The authors combined the two tasks with a new type of GAN network. They further gave the corresponding theoretical proof for this new GAN model and verified its performance on the benchmark datasets.", "summary_of_the_review": "Considering the advantages and disadvantages of this paper, the reviewer recommends weak accept. If the authors can verify their method on ImageNet, the reviewer will raise the rating.\n\nAfter reading the authors' rebuttal as well as the other reviews, I lower my rating because I don't see the experimental results on ImageNet as my expectation. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B60bePyMGAs", "reviewer": "Reviewer_jcSw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The high-level idea that simultaneously conduct PU classification and conditional generation is reasonable.\n2. Experimental results show the effectiveness of the proposed method.\n\nCons:\n1. The motivation is not that clear, especially why adopting multi-class PU learning rather than semi-supervised learning with out-of-distribution data.\n2. Some necessary baseline methods should be compared.", "summary_of_the_paper": "This paper proposes to leverage Positive-Unlabeled (PU) learning and the conditional generation with extra unlabeled data to simultaneously improve classification and generation performance. The authors show that their method is especially effective for handling out-of-distribution unlabeled data.", "summary_of_the_review": "This paper proposes CNI-CGAN to jointly exploit PU classification and conditional generation, so as to address the label sparsity problem. I feel that the high-level motivation is reasonable, but I still have some concerns:\n1. I think the motivation of this paper need more explanations. For the label scarcity problem mentioned in this paper, I feel that some recent semi-supervised learning methods are also applicable. For example, Semi-supervised learning under class distribution mismatch (AAAI 20), Safe deep semisupervised learning for unseen-class unlabeled data (ICML 20) also tackle the out-of-distribution data in semi-supervised learning. Besides, they are naturally deep model which do not need any extensions. In this sense, my question is, why adopting Multi-PU learning (needs extension to deep model) rather than these semi-supervised methods?\n2. The authors wrote \"In classification, a common setting to utilize unlabeled data is semi-supervised learning (Miyato et al., 2018; Sun et al., 2019; Berthelot et al.,2019), which usually assumes that the unlabeled and labeled data come from the same distribution, ignoring their distributional mismatch. In contrast, Positive and Unlabeled (PU) Learning (Bekker & Davis, 2020; Kiryo et al., 2017) is an elegant way of handling this under-studied problem, where a model has the only access to positive samples and unlabeled data.\" Firstly, as mentioned above, there are some recent semi-supervised methods which can well handle the distribution mismatch. Moreover, the authors claim that PU learning can elegantly handle this problem, which I cannot  fully agree. Note that in the setting of classical PU learning (e.g., uPU and nnPU), they also assume P data and U data come from the same underlying distribution, therefore I feel that the nnPU method deployed in this paper cannot well handle the  distribution mismatch problem.\n3. For estimating the transition matrix \\tilde{C}, usually it is a not an easy task in label noise learning. Some assumptions are usually needed, such as anchor point assumption (see Are Anchor Points Really Indispensable in Label-Noise Learning? NeurIPS 19), etc. However, I do not see such pre-set assumptions in this paper, so I doubt about the estimation quality for this matrix.\n4. For experiments, many recent PU learning works are not compared. Besides, another important work \"On Positive-Unlabeled Classification in GAN\" (CVPR 20) should also be discussed and compared. Therefore, I think the empirical study should be enhanced.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper combines discriminative and generative positive-unlabeled learning into a single framework. The reviewers argued the novelty and contributions are not enough for ICLR and unfortunately we cannot accept it for publication.
Decision: Reject