Mining Multi-Label Samples from Single Positive Labels
Authors: Youngin Cho
Daejin Kim
Mohammad Azam Khan
Jaegul Choo
Venue: NeurIPS 2022 Accept (2022)
Keywords: GANs
Markov chain Monte Carlo method
sampling
conditional generation
single positive label
multi-label

Abstract: Conditional generative adversarial networks (cGANs) have shown superior results in class-conditional generation tasks. To simultaneously control multiple conditions, cGANs require multi-label training datasets, where multiple labels can be assigned to each data instance. Nevertheless, the tremendous annotation cost limits the accessibility of multi-label datasets in real-world scenarios. Therefore, in this study we explore the practical setting called the single positive setting, where each data instance is annotated by only one positive label with no explicit negative labels. To generate multi-label data in the single positive setting, we propose a novel sampling approach called single-to-multi-label (S2M) sampling, based on the Markov chain Monte Carlo method. As a widely applicable “add-on” method, our proposed S2M sampling method enables existing unconditional and conditional GANs to draw high-quality multi-label data with a minimal annotation cost. Extensive experiments on real image datasets verify the effectiveness and correctness of our method, even when compared to a model trained with fully annotated datasets.

Mining Multi-Label Samples from Single Positive Labels
Youngin Cho
Daejin Kim
Mohammad Azam Khan'
Jaegul Choo'
KKAIST AI
NAVER WEBTOON AI
{choy10521, , kiddj, azamkhan, (choo) ccooookaisst.co ac.kr
Abstract
Conditional generative adversarial networks (cGANs) have shown superior results in class-coniitioal generation tasks. To simultaneously control multiple condi- tions, cGANs require multi-label training datasets, where multiple labels can be assigned to each data instance. Nevertheless, the tremendous annotation cost limits the accessibility of multi-label datasets in real-world scenarios. Therefore, in this study we explore the practical setting called the single positive setting, where each data instance is annotated by only one positive label with no explicit negative labels. To generate multi-label data the single positive setting, we propose a novel sampling approach called single-t--mult-llaell (S2M) sampling, based on the Markov chain Monte Carlo method. As a widely applicable "add-on" method, our proposed S2M sampling method enables existing unconditional and conditional GANs to draw high-quality multi-label data with a minimal annotation cost. Exten- sive experiments real image datasets verify the effectiveness and correctness of our method, even when compared to a model trained with fully annotated datasets.
Introduction
Since being proposed by Goodfellow el al. []] generative adversarial networks (GANs) have gained much attention due to their realistic output a wide range of applications, e.g., image synthesis 2 3 4. image translation [5] 6 7 8) and data augmentation (9, 10]. As an advanced
task, generating images from a given condition has been achieved by conditional GANs (cGANs) and their variants [2]. To reflect the nature of real data where each data instance can belong to multiple classes, multi-label datasets have been introduced in the applications of cGANs S [3]. In multi-label dataset, each data instance
be specified with multiple attributes. For example, in multi-label facial image dataset, each image is labeled for entire classes such as Black-hate hhir, Smile, and Male. The label for each class is given the corresponding attribute. Despite its usefulness, as claimed in earlier studies [14] [5 16), access multi-label datasets is severely limited in practice due to the difficulty of annotating all attributes. Under these circumstances, a weakly annotated dataset is used as an alternative for cGANs to reduce the annotation cost.
binary value and indicates the presence or absence of
In this paper, we introduce the single positive setting [6],, originally proposed for classification tasks, to conditional generation. Each data instance has a label indicating only the presence of one attribute (i.e., a single positive label) in this setting, and the presence of the rest the attributes remains unknown. For instance, in facial image dataset, all attributes are fully specified in multi-label setting (e.g., Smiling black-haired man) whereas only one attribute is specified in the single positive setting (e.g., Black-hair), The single positive setting allows us not only to reduce the annotation cost, but also to model the intrinsic relationships among classes.
Equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

A 1 Circular
1 Blue
A 10 Circular & Not blue
A 0 0
AUB 1
Not circular Blue
Circular & Blue
GANs S2M sampling
(a) Single positive setting
(b) Non-overlapping classes
(c) Overlapping class
Figure I: Illustration of joint classes where two attributes (circular, blue) are given. (a) Only one of the attributes is specified in the single positive setting. From classes A and B, our proposed S2M sampling method can draw samples from three joint classes: (b) two non-overlapping classes (A \ B and A) and (c) one overlapping class (An B).
To generate multi-label data using only single positive labels, we consider two types of combinatorial classes from two classes, A and B, each corresponding to a single positive label: (1) overlapping class, where the data instances belong to both classes (A n B), and (2) non-overlapping class, where the data instances belong to only one of the classes (A B or B \ A). Figure shows examples of overlapping class and non-overlapping classes when two types of single positive labels are given. By extending this concept, we can consider combinatorial classes, where data instances belong to certain classes but not to the rest. We denote these classes as joint classes. By accessing a joint class, we can generate multi-label samples using only single positive labels. Ideally, we can represent all possible label combinations with at least one positive label.
Several attempts have been made to consider such joint class in generation tasks. Specially designed generative models such as GenPU 17 and RumiGAN [8] were proposed to generate samples from one class while excluding another class. However, these studies deal with only two classes without considering the overlapping class. Recently, Kaneko et al. [19] proposed CP-GAN to capture netween-class relationships in conditional generation. To generate images belonging to n classes, they equally assign 1/n as the class posterior for each selected class. This indicates that CP-GAN generates images that have equal probability of being classified as each class. However, because an image in the real world that belongs to multiple classes does not have equal class posteriors, sample diversity is lost. Consequently, we focus on sampling approach to precisely draw samples from complex distributions that are difficult to directly sample. In recent GANs studies, sampling approaches 120 21 employed the rejection sampling Markov chain Monte Carlo method obtain realistic data. These approaches adopt the sampling process as the post-processing method of GANs and filter generated images using the pretrained classifiers.
In line with these studies, we propose novel sampling framework called single-to- co-multi-llaee (S2M) sampling to correctly generate data from both overlapping and non-overlapping classes. We newly propose a tractable formulation to estimate the conditional density of joint classes. Concretely, we employ several classification networks to estimate the target conditional density and apply the Markov chain Monte Carlo method to pretrained unconditional and conditional GANs. In Figure 2 the conceptual difference against the existing approaches (1s row) and provide the empirical results on a ID Gaussian example (2nd row). As S2M sampling performs at the inference time of generation, it fully preserves the image quality and eliminates the need for changing the objective functions and architectures. We validate the effectiveness of our method compared to the existing models on diverse datasets such MNIST, CIFAR-10 and CelebA. To the best of our knowledge, our approach is the first sampling framework that generates multi-label data from single positive labels. Our contributions can be summarized follows:
we depict
We introduce the single positive setting in the conditional generation task and provide the theoretical framework for estimating conditional densities of joint classes.
We propose novel sampling framework based the Markov chain Monte Carlo method for generating multi-label data from single positive labels.
Through extensive experiments, we show that our proposed S2M sampling method correctly draws multi-labe data while fully preserving the quality of generated images.

---

Dataset
(a) cCANs
(b) CP-GAN
(c) CAN Ours
B
Conceptual examples
1g 6.5)
ACB
ID Gaussian experiments
PDE
class
Ground--ruth (Intersection)
Generated samples
Figure 2: cGANs, CP-GAN and method are compared in a class-overlapping case. 1D Gaussian examples consists of two classes of one-dimensional Gaussian mixtures with one common mode, and each method attempts to generate samples in the overlapping region. For cGANS and CP-GAN, an equal value of 0.5 is provided as labels for the two classes. (a) It not clear how eGANs obtain samples of the class. (b) CP-GAN draws samples from the narrow region. (c) GAN with S2M sampling draws samples accurately without sacrificing diversity.
2 Related Work
Conditional GANs. The aim of conditional GANs (cGANs) [ is to model complex distributions and control data generation by reflecting the label input. Various studies of cGANs have made significant advances in class-conditional image generation by introducing an auxiliary classifier [12] 22]. modifying the architecture [23 21, and applying metric learning [24]. In a reakly-supprviser
setting, GenPU [17] and SuANL (18) specify only two classes and draw samples that belong to one class but not the other. CP-GAN [9] learns to draw samples conditioned on the probability output of the classifier. Given that this model tends draw samples on a limited region of the data space, it is challenging to ensure a variety of samples as shown in Figure 2 In contrast, we propose a sampling method that draws multi-label samples without sacrificing diversity.
Sampling in GANs. Sampling methods are used to improve the sample quality in GANs. Discrimina- tor rejection sampling [20] uses the scheme of rejection sampling and takes samples close to real data by estimating the density ratio with the discriminator. In addition, Metropolis-Hasting adopts the Markov chain Monte Carlo (MCMC) method and calibrates the discriminator to improve the sample quality in high-dimensiona. data space. Discriminator driven latent sampling [25] uses the MCMC method in the latent space of GANs to draw realistic samples efficiently. GOLD estimator (26 uses a sampling algorithm to improve the quality of images for class-conditional generation. While previous studies focus improving the sample quality, our S2M sampling method aims to draw multi-label samples while also improving sample quality.
GAN 21
Methods
3.1 Problem Setting
Let x E X be a data point as a random variable and let Vı.. E {0. 1)" denote its corresponding multi-labels as binary random variables. Here, for every k. Us = indicates that is contained in the k-th class while y/x = indicates that x is not. We consider two index sets, an intersection index set and difference index set J, so that the pair (I can be used as index indicate the class, where data points contained in all classes indicated by 7 but excluded from all classes indicated by Let I be collection of all possible pairs of I and J, defined
(1) where N {1.2, ..., n is finite index set of all classes and P(N) is the power set of N. That is, the intersection index set indicates at least one class and is distinct from the difference index set. Especially for I U N, the class indicated by (I, is called the joint class.
I ((I,)) E P(N) x P(N) : + 0, I. = (}.
Let p(a, V1, 1/, ..., vn)) be the joint probability density function, and let Pdata (x, c) be the joint density
of an observed data point E X and class label e E N such that Pdata (x:(c) = p(z|u- = 1). Given
the class priors Patotaa (c), Te = p = 1) and samples drawn from the class-conditional density Pdota (z/cc) for each 1,2, .... n, goal is to draw samples from the conditional density
(z) = p(=lVi E 1, e J. = 1, 1, = 0),
(2)
for (I,J) and m(I.)) == p(vi E f, E J. = 1, y, = 0) 0. this work, we propose adding mild constraint which will allow our sampling algorithm to derive the target density.
3

---

(a) Single positive setting
Outputs of S2M sampling
A
2L 1-
1-
1, 31
(2)
12,
(3)
J= 0
C(3)
B
1
Figure 3: (a) A dataset with single positive labels is given. (b) S2M sampling can draw samples as multi-label data with two index sets: intersection (I) and difference (J).
Assumption For every j.j e such that + j. if ply: = y; 0) > 0, then supp p(x) y: = 1, Uj = 0) n supp p(zly; 1. Vi 0) = 0.
0) > and ply; yt
Assumption 1 states that no data points are likely to be assigned two mutually exclusive labels, which can be naturally assumed in many practical situations. Figure iillustrates our problem setting.
3.2 Mining Multi-Label Data with S2M Sampling
Naturally, question may arise as to whether the supervision given to us sufficient to obtain multi- label samples. To gain insight into this, we derive useful theorem which provides an alternative formulation for the target density 2
Theorem 1. Let (f(.J) X - [0, oo)),, be an indexed family of non-negative measurable functions X, and let fu == (())),) Then, the following conditions hold:
II, (a) V(IJ) E (I.. Js ((S.N.))
(b) Vi,j E N s.t. i # jusupp ((0,G) Ssupp f(U),)) =
if and only if. everrery
every (I,
(minger ft max fz) minser fi
if
f(,J)
(3)
otherwise
where (-)+ represents the positive part.
Proof. Please refer Appendix A
Let (() pla, vi E Vj E J,y: = 1, 0) for every (T.J) E I. Then, both (a) and (b) in
Theorem hold by the Assumption !
According to the Theorem [ ((I,J) > 0,
((),)) TT.r, (min(mmp(//lu = 1): ie max(T 1)
(0)+ (4)
The alternative formula + shows that the density of the joint class can be derived from the class- conditional densities of the single positive labels. Despite their clear relationship, training a generator that can model the density P(I.J) is non-trivial problem since the formula consists of several implicitly defined conditional densities, class priors, and variable sets. To address this issue, we propose the application sampling approaches upon existing GANs. Interestingly, our sampling framework called S2M sampling allows not only for samples to be drawn from the target density, but also modification of J, and class priors at inference time. The rest of this section introduces the main approaches of our S2M sampling method.
Density Ratio Estimation. We utilize several classification networks compute implicitly defined density ratios. For simplicity, we denote G' as pretrained generator for both unconditional and conditional GANs. produces data by taking a latent and class label e for class-conditional generation and only for unconditional generation. We consider three classifiers D., Dr. and Dj which are obtained by minimizing Lo, L., and L,, respectively, i.e.,
L, [. (2.c) log D,(x))) (2)
D (x))),
(5)
C, (xr [log (c(=)),,
- log (e(c)))),
where PG the distribution of the generated samples by G. The optimal classifiers trained by these losses Dr,, D;, and Dj satisfy the following equations: Dz = Paato (x)) (Panta (x) pa(x)), (elz))) p(zlve 1) (c) pdota (z), D7(clx)) = pc(xlc)po(c)/pe) From D

---

D;, and Dj. we can access the density ratios Paata( (z) p(alye = 1)/Pdata (2), and pa(r)c)/PG(( which will be used to compute the acceptance probability of the MCMC method.
S2M Sampling for Unconditional GANs. We apply Metropolis- Hastingg (MH) independence sampling (17 to draw samples from the complex target distribution pt. The MH algorithm uses a Markov process where each transition from a current state Iy to the next state xx+1 made by an accept-reject step. At each step of MH independent sampling, a new proposal x' is sampled from a proposal distribution q( [x))
q(x') and is then accepted with probability a(x', xx) which given by afx' 2t min (1, pt pe
(=()(((()) We set Tk+1 x' if x' is accepted, and x1+1 Fg. otherwise.
Under mild conditions, the chain Fi:K converges the unique stationary distribution Pt as K
Let G be a generator where the support of pG contains that of P(I,)) To draw samples from P(),)) we let Pt = P(),J) and take multiple samples from as independent proposals, i.e. g(x' [xr] = Pc (1'). Then, the desired acceptance probability a(x', x) can be calculated using D and D,:
(x)
[in n; (d)ta(
Di(il)): max {
(d)ta(() (
T D2 D;(]k) J J +
a(x', ,x) min ( min (1, ((,)(("(
(6) Different from Turner et al. (21, the term Jy ((r,,(xx))) (a')/r(r,)) (I) is added to the acceptance probability formula to draw multi-label samples. To obtain uncorrelated samples, a sample is taken after fixed number of iterations for each chain. The sampling approach allows one to control the parameters I. and " == T (k) without any additional training of the model. A summary of our S2M sampling algorithm provided in Appendix [
S2M Sampling for Conditional GANs. cGANs can provide a proposal distribution close to the target distribution P(r.)), which greatly increases the sample efficiency of the MCMC method. Let c be a class label such that the support of class-conditional density (-(-lc) contains that of Ptr,))- At each step of the MH algorithm, the proposal x' ~ q(x' (x:) pc(x')o (c) is accepted with a probability ae(x', r). The desired a0(2', a) can be calculated as
ru. (
ac(x',z) min ( min
(aun(t) ("
(7)
That is, the sampling method can be adopted to cGANs by additionally computing Di(c))/D((ca)))
Latent Adaptation in S2M Sampling. While our S2M sampling method allows us to draw multi- label samples, the algorithm rejects certain samples through the sampling procedure if the target dis- tribution Pú.J) is significantly different from the generator distribution. Specifically, an independent MH algorithm takes time inversely proportional to the probability of generated sample belonging to the target class. To alleviate this issue, we propose a technique to improve the sample efficiency from past sampling attempts. Initially, certain number of pilot target class samples Fi:m are drawn using S2M sampling, and then the corresponding target latent samples ti:m; 24 = -G(tt) are obtained. Since the latent samples are nearly restricted the latent prior of GANs (e.g., Gaussian distribution), the distribution P. of the target latent can be roughly estimated by fitting a simple probabilistic model us- ing fi. Let PG be the newly obtained generator distribution induced by G. i.e. G(z) r (o(x) for ~ (z). The sample efficiency can be further improved by using PG as the proposal distribution in the MH algorithm since ÅG much close to the target distribution. To run the MH algorithm without retraining the classifiers, we approximate PG
R C-p=(z)/p.
for some
constant C. Then, at each step of the MH algorithm, the proposal x' ~ g(o" (x)) = Pa (x') is accepted
1 For example, the chain of samples is uniformly ergodic if pe/g is bounded (28). 'The equation is derived in the previous latent sampling approaches by the reverse lemma of rejection sampling (25) or the change-of of formula 29
5

---

-3/5
0. Airplane
A: bAngs
B
(3. (3)
1: Automobile
B. Black-hair
A+ NN+
2: Bird
M: Male
Even
3: Cal
N: No beard
All
4:
Smiling
B: Odd digits
Dog
W: Wearing-
Frog
Ipstick
(a) (F)MNIST
(b) CIFAR-7to3
(c) CelcbA-BMS
(d) CalebA-ABMNSW
Figure 4: Experimental settings for (F)MNIST, CIFAR-7103, CelebA-BMS, and CelebA-ABMNSW.
with a probability ,(z',x) which can be calculated as
a(x', x) = min (u.n((())//ol) P(..n (x)/
min (1, (1, (1,Tu,,le)) - (2.())/Ball)) ((.,,) (D); ()p:(3)/
where x' x'= G(z") and x - G(z). Here, we can explicitly compute the density ratio between ps and P. by letting Ps be known probability density function such as a Gaussian mixture. In practice, the target class samples rarely appear in the generated samples, one can consider applying latent adaptation repeatedly. For instance, to draw Black-haired man samples, we can first search the latent space of Man by using latent adaptation, and then perform it in that space again to search the space of Black-haired man. In terms of time complexity, this is performed more efficiently than searching the space of Black-haired man at once. A detailed description of latent adaptation and a discussion about the time complexity are provided in Appendix an2 Appendix [ respectively.
Practical Considerations. We employ three classifiers D.. Dr, and Dy. to compute the acceptance probability used in the MCMC method. For better training efficiency of the classifiers, we use shared layers, except for the last linear layer. Since the classifiers are not optimal in practice, we scale the temperature of classifiers [30] and adjust 7/k to calibrate the sampling algorithm. Detailed settings and the ablation study are provided in Appendix D and Appendix respectively.
Experiments
In this section, we validate that our S2M sampling method properly draws samples within the joint classes. Specifically, we mainly consider two cases in the single positive setting: (i) one class is entirely contained another class, (ii) multiple classes are partially overlapping with each other. In the former setting, we verify that S2M sampling can completely filter out samples of a smaller class, and compare our method to the existing models including GenPU [17] and RumiLSGAN (18) on MNIST and -aShIon-MNIST (FMNIST). In the latter setting, all possible joint classes are assessed with our S2M sampling method. Subsequently, we evaluate how accurately S2M sampling draws samples in these classes compared to CP-GAN [9] and the models trained with a fully annotated multi-label dataset.
To verify whether method can accurately generate samples for each joint class, evaluate accuracy which indicates how many generated images are assigned to the target joint classes by a classifier trained with fully annotated data (i.e., multi-lahel data). Since accuracy itself cannot fully evaluate the distance between the distribution of generated samples and that of real samples, we additionally introduce various metries to evaluate fidelity (i.e., how realistic the generated samples are) and diversity of generated images: (i) Inception Score (IS) [31] (ii) Fréchet Inception Distance (FID) 32) (iii) improved precision and recall [33, and (iv) density and coverage
34
FID and IS are metrics to evaluate the fidelity and the diversity of generated samples using the features of the pretrained Inception- V3 network [35]. A lower FID and a higher IS indicate higher fidelity and diversity of generated samples. Precision and density measure the ratio of generated samples that falls within the manifold of real samples, so that they evaluate the fidelity of generated samples. Contraily, recall and coverage measure the ratio of real samples that falls within the manifold of the generated samples, and thus evaluating their diversity. That is, a higher value of these metrics indicates that the generated samples have a distribution similar to the real one.
Since several metrics for fidelity and diversity MNIST and FMNIST), we instead use the FID score computed from activations of LeNet5 (36) and denote this as FIDÍ, The test set used as reference dataset for evaluating FID. All quantitative results averaged over three independent trials, and the standard deviation is denoted by subscripts.
applicable for non- non-Imaged Iea--meteeeetttte like images (e.g.,

---

MvvsT-Even
FMNIST-BB ven
D
6
O G 3
4
2
4
2
6
6
0
6
7
2
4
8
8
2
2
6
6
2
8
4
O
O
O
6
4
O
R
0
6
0
6
4 3
9
0
S 4
9
4
g
C
RumiLSGAY
CP-GAN
GAN + Ours
RumiL LSGAN
CP-GAN
GAN + Ours
Figure 5: Qualitative results on MNIST-Even and FMNIST-Even.
The qualitative results are randomly sampled in all experiments. Detailed experimental settings, such as architectures and hyperparameters, are provided in Appendix D
4.1 Sampling for Classes with Inclusion
Experimental Settings. We consider a special case of our problem setting, where one class is entirely contained in another class (B C A) as shown in Figure (a). This setting is similar to the
positive-unlabeles setting (37 38) where the positive data is included in the unlabeled data. Under this constraint, GenPU [17]. RumiLSGAN [88. and CP-GAN [19] can be used to generate samples from the oveellallap class (A) Along with these models, we attempt to draw samples from the non- overlapping class by adopting S2M sampling
unconditional WGAN-GP (39]. These experiments 4 were conducted in three settings (See Figure : (i) MNIST-3/5 (A=(3, 5), (=(3)), ) (ii) MNIST-Even
(A= 1, 2, 3, 4, 5, 6, 7, 8, 9), (1, 3, 5, 3,5,,191 9)), and (iii) FMN (A= (A=(Or stintTop 1 Trouser. Pallover: Dress, 4Coat- 5andal- 6shirt: 7 Sneaker. SBag. 9Ankle boor B= (1, 5, 7, 9}). For S2M sampling, samples are obtained 100 MCMC iterations.
Quantitative Results. Table D shows the results of our S2M sampling method and the baselines for the non-overlapping classes. The performance of GenPU is reported only for MNIST-3/5 due to its mode collapse issue [40 41. S2M sampling adopted to unconditional GAN shows promising results in terms of both accuracy and FID', In fact, our results are superior to the existing methods specially designed for generating non-overlapping data such as GenPU and RumiLSGAN. Notably, S2M sampling improves accuracy by 8. 8.5% and 6.8% while decreasing FID' compared to the second- best models on MNIST-Even and FMNIST-Ever respectively. This indicates that S2M sampling accurately samples the images of the non-overlapping classes without being biased to specific mode. Table 1: Results for different models on MNIST and FMNIST.
MNIST-3/5
MNIST-Even
FMNIST-Even
Method
Acc. (%)
FID (4)
Acc. (%)
FID (1)
Acc. (1)
FID (L)
GenPU 1i
99.33-20.56
1.93:11110
RumiLSGAN
x
77 1.55
13.2011119
86.1114483
1.44
91.07:0.88
3.23:1248
CP-GAN 1p
66 66.89211466
19.50:2090
87.87-0040
2.23
81.21
6.14-2.56
GAN Ours
99.52:20.34
0.88:1021
96.37:10.25
0.86::0.2
9..87
2.48
Qualitative Results Figure 5 shows the qualitative results of RumiLSGAN, CPGAN and our method adopted WGAN-GP. As indicated by the red boxes, RumiLSGAN and CP-GAN fail to completely eliminate the samples of the smaller class (e.g., odd digits for MNIST-Ever Even or Sneaker for FMNIST-Even), Conversely, S2M sampling correctly draws samples for the target classes.
4.2 Sampling for Multiple Classes
Experimental Settings. Here, we consider a general case of single positive setting where multiple classes can be overlapping as shown in Figure (b-d), Given classes, at most 2 - joint classes can be obtained, In this setting, we attempt to obtain samples of these joint classes using only single positive labels. We consider cGANs with a projection discriminator (cGAN-PD) 23 AC-GAN [12.
and CP-GAN 19] as the bascline generative models.
Since traditional conditional GANs such as cGAN-PD and AC-GAN are not originally designed for generating joint classes, we naively introduce these models in our setting with slight modifications. Concretely, as a method to generate images belonging to classes, we provide 1/n as the conditional value for each class, following Kaneko et al. [19. In our experiments, we expect these models to have a lower bound performance of our results and indicate their results with asterisks ("). In contrast, conditional GANs trained on a fully annotated dataset where all joint classes are specified can be considered as strong baselines in our settings. cGAN-PD and AC-GAN are trained in this setting,
7

---

Airplane
Bird
Deer
Automobile
Cal
Dog
Frog
CIFAR-7OS
R-M-S M-B-S
S-B-M
B+M-S
M+S-B
B+S-M
B+M+S
CelebA (BMS)
Figure 6: Results of S2M sampling with cGAN-PD on CIFAR-7to3 and CelcbA-BMS. The first row depicts the target joint class.
Table 2: Accuracy (%), FID, IS, precision (P), recall (R), density (D), and coverage (C) for different models on real-world datasets.
Model
Acc. (f)
FID (4)
IS (7)
P())
R (7)
(()
(()
cGAN-PD (Oracle)
23
87.60=0.15
16.55:
8.39
0.73++0.01
0.65
0.89++0.5
0.85
CIFAR-7H 7003
AC-GAN (Oracle)
12
94. 04.399
16.14:20.43
8.44::0.3
0.74::000
0.63-20.00
0.89:10.02
0.84:20.00
cGAN-PD"
23
27.19:0053
21.12:+0.22
7.94
0.71
0.66
0.81+:0.03
0.78
AC-GAN" 121
3..06
26.42:20.82
2.31
0.69:1001
0.66:20.01
0.71::001
0.70
CP GAN 19
42.62:10.54
27.88:42000
7.42400.6
0.71::000
0.66::0.00
0.76:1:000
0.71
GAN+ + Ours
17.65
14.52:+00.55
8.71
0,72++0.00
0.000
0.92+00.2
0.87
eGAN-PD Ours
80.62::2.08
14.14:10.34
9.06:20.18
0.75:10.01
0.662:0.01
1.09::0.12
0.90
cGAN-PD (Oracle)
M
78.35 1.99
00.38:20.55
2.45
0.79
0.48:0.02
1.12::0 18
0.8622002
CelebA BMS
AC- (Orucle)
12
91.57=2043
10.41:1.52
2.40
0.79::002
0.49:0.04
1.18:t0.12
0.86
eGAN-PD' 23
70
10.97:
2.32:1..06
0.79:1001
0.48::0.14
1.19::0:5
0.8622.02
AC- GAN"
12
$2.01:18811
12.09:::.00
2.32:40.10
0.79:10.02
0.49::0.01
1.17:::.00
0.80
CP-GAN 19
87. 2.5
13.38.1
2.43...09
0.78:1001
0.45::0.2
1,0821004
0.8222002
GAN + Ours
85.22::427
10.50:10.97
0.008
0.83++0002
0.48++0.03
1 1.36:t0.08
0.89
eGAN-PD + Ours
90.444::.05
00.63:+0..2
2.46
0.84
0.48
1.40:10022
0.88
eGAN- PD (Oracle)
23
69.61
13.54::2.54
2.44
0.79::0.22
0.42+:0.04
1.15:0.10
0.84
CelchA-AHMNSS
(Omcle)
12
91.37=11144
13.46:1 1:20
2.3020005
0.80
0.3620001
1.17:0000
0.8220.01
"GAN-PD"
PA
15.28:1008
11:09:0:67
2.32:10.03
0.80=:000
0.47:20.03
1.19:0007
0.85
AC-GAN"
12
24.97-1064
11.92:::206
2.44:0006
0.77:1004
0.45:20002
1.11:0.13
0.82220.4
CP-GAN 19
1.109
14.97:+4.18
2.404
0.74-+:0.08
0.46+0003
0.95++0.08
0.77=00.06
GAN + Ours
70.94:12.97
9.77=0022
2.62:10.05
0.81:10001
0.47
1,28:1001
0.87=00011
cGAN-PD Ours
79.74
10.30:10
2.45
0.85:10.01
0.47:10.02
1.45:10.06
0.58:0000
and are denoted oracle models. In the experiments, S2M sampling is adopted on unconditional GAN and cGAN-PD. When adopting S2M sampling to cGAN-PD, a class containing the target joint class is used as the conditional value of cGAN-PD. BigGAN (2) is used as the backbone architecture for all generative models, and only single positive labels are used except for the oracle models.
To demonstrate the effectiveness of our S2M sampling method, we use three real-world image datasets: (i) CIFAR-7to3 that has three classes (A, B, C), each of which contains four original classes of CIFAR-10, 10, i.e. A= (AAirplance, Automobile, Dog, Frog), B=(AAtommbblle Bird, Cat, Frog), ) C=(Cat, Deer, Dog. Frog). ). (ii) CelebA-BMS that consists of human face images, each of which is labeled for one of three attributes (Black-hair. Male, and Smiling), and (ii) CelebA-ABMNSSW that consists of human face images, each of which is labeled for one of six attributes (bAngs, Black-hair, Male, No-beard, Smiling, and Wearing-lipstick As depicted in F up to 23_ 1 or 26 - 1 joint classes. For S2M sampling, samples are obtained at 200 MCMC iterations for CIFAR-7to3 and CeMebA-BMSS. and obtained at 1000 MCMC iterations for CelebA-ABMNSSN
these datasets can have
Quantitative Results. Table summarizes the quantitative results of joint class generation on CIFAR-7to3, CelebA-BMS, and CelebA-ABMMSW, As expected, we can observe that cGAN-PD" and AC-GAN" cannot accurately generate samples from joint classes, which is consistent with the findings of Kaneko et al. [19]. Although CP-GAN shows reasonable accuracy for all experiments,
8

---

Non-overlapping class (B-M-S)
Overlapping class (B+M+S)
Sereoeae probo
Accuswcy (85)
Accuriny
MCMC MCMC 1
MCMC
MCMC Htertion
MCMC zeration
GAN + S2M Sampling
CAN + S2M Sampling w/ Latent Adaptation
Figure 7: Accuracy (%). FID, and acceptance probability (%) per MCMC iteration for S2M sampling with and without latent adaptation.
suffers from generating diverse samples. As discussed in Section I the high FID and low coverage support that CP-GAN tends to generate samples in a narrow scope of real data space.
In all experiments, our S2M sampling method adopted to cGAN-PD consistently outperformes the non-oracle baselines in terms of accuracy and diversity, e.g. in CIFAR-7to3, our method improves accuracy and FID of CP-GAN by 38% and 13.74, respectively. Such improvements verify that our S2M sampling method generates correct samples for joint classes, and also preserves diversity within them. Despite a large sample space of GAN, S2M sampling adopted to unconditional GAN also surpasses non-oracle baselines in terms of both fidelity and diversity while achieving reasonable accuracy. Surprisingly, despite the fact that the oracle models are trained with fully annotated data, S2M sampling shows comparable performance against those models using only single positive labels. Moreover, as an post- post-processing method, S2M sampling fully preserves the image quality of GANs and shows the highest fidelity in terms of precision and density.
Qualitative Results. Figure 6 depicts the results of S2M sampling for every possible joint classes on CIFAR-7to3 and CelebA-BMS. The results show that S2M sampling can draw diverse high-quality images from the distributions of all joint classes. Furthermore, S2M sampling is model-agnostic and does not require the retraining of GANs; thus, it can be easily integrated with various GANs including otate-of-the-art GANs. We depict more qualitative results of adopting our S2M sampling method to unconditioanl GAN and pretrained StyleGANv2 in Appendix [E.3 and Appendix
Analysis on Latent Adaptation. To demonstrate the effect of latent adaptation, we tested our latent adaptation technique in overlapping class (B+M+S) and non-overlapping class (B-M-S) on the CelebA-BMS dataset. We first perform S2M sampling on unconditional BigGAN these classes and draw 10k pilot latent samples obtained 100 MCMC iterations. Using these latent samples, a Gaussian mixture of eight components is fit as a new latent of GAN (See Appendix B.2) Due the discrepancy between the generator distribution and the target distribution, S2M sampling originally shows a low acceptance probability, as shown in Figure 7 In this case, applying latent adaptation increases the acceptance probability and significantly improves both accuracy and FID at the early stage of iterations. This indicates that the generator distribution becomes closer to the target distribution by adopting latent adaptation, resulting an accelerated convergence speed.
5 Limitations
In general, as the number of attributes increases, the number of each joint class samples in the train- ing dataset are drastically reduced. This may cause two primary problems: (i) training GANs to be able to generate each joint class samples becomes chal- lenging, and S2M sampling cannot draw samples in which GANs do not generate, and (ii) the com- putation time of the sampling procedure increases. In this case, it may be necessary to apply the latent adaptation technique to shorten MCMC chains. In principle, we can reduce the sampling time complex- ity to be proportional to the number of attributes (See Appendix C).
mining camples (k)
30
30
25
# tovining le
Q1
20
20
Q2
15
Q3
10
5 10 30
60
A+B
+32
+N
+5
+W
Joint class index
Attributes
Figure 8: (Left) Distribution of the number of training samples for each joint class on CelebA-ABMNSW. (Right) As attributes are added, the quartiles of the number of samples for each joint class significantly decrease.

---

As shown in Figure 8 the number of training samples in joint classes rapidly decreases. Because of this, six attributes most are used on the CelebA dataset for experiments. Almost half of the joint classes have fewer than 100 training samples when using six attributes (i.e., CelebA-ABMNSW), Nevertheless, our method handles larger number of attributes compared the existing methods. We believe that our work can be further extended to larger attributes with the sufficient amount of data.
6 Conclusion
In this study, we investigate the single positive setting in the class-conditional generation task and propose a novel sampling framework called S2M sampling. We demonstrate that our proposed S2M sampling method can accurately draw high quality multi-label samples in the single positive setting, To improve sampling efficiency, we also introduce the latent adaptation technique. We believe that our theoretical framework provides better understanding for beeakly-supervieee multi-label data generation. Our S2M sampling method has the potential to be applied to a wide variety of generative among others.
models as well, including clustering based GANs [4] and diffusion models
Acknowledgments and Disclosure of Funding
This work was supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (No. 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST) and No. 2021-0-01778, Development of human image synthesis and discrimination technology below the perceptual threshold). This work was also supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) (No. NRF-2022R A2B5B02001913).
References
[1] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2014.
[2] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Proc. the International Conference on Learning Representations (ICLR), 2019.
[3] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In Proc. the International Conference on Learning Representations (ICLR), 2018.
[4] Taesung Park, Ming- Yu Liu, Ting-Chum Wang, and Jun-Yan Yan Zhu. Semantic image synthesis with spatially-adapie normalization. In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), 2019.
[5] Ming-Yu Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2017.
[6] Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proc. of the IEEE international conference on computer vision (ICCV), 2017.
[7] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-imat tomee translation with conditional adversarial networks. In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), 2017.
[8] Yunjey Choi, Min-Je Choi, Munyoung Kim, Jung- Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), 2018.
[9] Abhinav Shrivastava, Abhinav Gupta, and Ross B. Girshick. Training region-based based object detectors with online hard example mining. Proc. of the IEFE conference on computer vision and pattern recognition (CVPR), 2016.
10

---

[10] Christopher Bowles, Liang Chen, Ricardo Guerrero, Paul Bentley, Roger N. Gunn, Alexander Hammers, David Alexander Dickie, Maria del C. Valdés Hernández, Joanna M. Wardlaw, and Daniel Rueckert. GAN augmentation: Augmenting training data using generative adversarial networks. CoRR, abs/1810. 10863, 2018.
[1]] Mehdi Mirza and Simon Osindero. 0ts//411.1188. 2014.
Conditional generative adversarial nets.
CoRR,
[12] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In Proc. the International Conference on Machine Learning (ICML), 2017.
[13] Yu-Jing Lin, Po-Wei Wu, Che-Han Chang, Edward Y. Chang, and Shih- Liao. Relgan: Multi- domain tonge-to-mmae translation via relative attributes. In Proc. of the IEEE international conference on computer vision (ICCV), 2019.
[14] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Proc. of the European Conference on Computer Vision (ECCV), 2014.
[15] Agrim Gupta, Piotr Dollár, and Ross B. Girshick. LVIS: dataset for large vocabulary instance segmentation. In Proc. of the IEEE conference computer vision and pallern recognition (CVPR), 2019.
[16] Elijah Cole. Oisin Mac Aodha, Titouan Lorieul, Pietro Perona, Dan Morris, and Nebojsa Jojic. Multi-label learning from single positive labels. Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), 2021.
[17] Ming Hou, Brahim Chaib-draa, Chao Li, and Qibin Zhao. Generative adversarial positive- unlabelled learning. Proc. the International Joint Conference on Artificial Intelligence (IJCAI), 2018
[18] Siddarth Asokan and Chandra Sekhar Seelamantula. Teaching a GAN what not to learn. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2020.
[19] Takuhiro Kaneko, Yoshitaka Ushiku, and Tatsuya Harada. Class-distinet and class-mutual image generation with gans. In Proc. of the British Machine Vision Conference (BMVC), 2019.
[20] Samanch Azadi, Catherine Olsson, Trevor Darrell, Ian J. Goodfellow, and Augustus Odena. Discriminator rejection sampling. In Proc. the International Conference on Learning Represen- tations (ICLR), 2019.
[21] Ryan D. Turner, Jane Hung, Eric Frank, Yunus Saatchi, and Jason Yosinski. Metropolis-hastings generative adversarial networks. In Proc. the International Conference on Machine Learning (ICML), 2019.
[22] Mingming Gong, Yanwu Xu, Chunyuan Li, Kun Zhang, and Kayhan Batmanghelich. Twin auxiliary classifiers GAN. CoRR, 0bs//197.002690 2019.
[23] Takeru Miyato and Masanori Koyama. cgans with projection discriminator. In Proc. the International Conference Learning Representations (ICLR), 2018.
[24] Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2020.
[25] Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your GAN is secretly an energy-based model and you should use discriminator driven latent sampling- In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2020.
[26] Sangwoo Mo, Chiheon Kim, Sungwoong Kim, Minsu Cho, and Jinwoo Shin. Mining GOLD samples for conditional gans. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2019.
11

---

[27] Luke Tierney. Markov chains for exploring posterior distributions. The Annals of Statistics, 22(4):1701-1728, 1994.
[28] K. L. Mengersen and R. L. Tweedie. Rates of convergence of the Hastings and Metropolis algorithms. The Annals of Staristics, 24(1):101 121, 1996.
[29] Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Reparameterized sampling for generative adversarial networks. In Machine Learning and Knowledge Discovery in Databases. Research Track- European Conference, 2021.
[30] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proc. the International Conference on Machine Learning (ICML., 2017.
[31] Tim Salimans, Ian J. Goodfellow, Wojeiech Zaremba, Vicki Cheung. Alec Radford, and Xi Chen. Improved techniques for training gans. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2016.
[32] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vish- wanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems, 2017.
pages
[33] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2019.
[34] Muhammad Ferjad Nacem, Scong Joon Oh, Youngjung Uh Yunjey Choi, and Jaejun Yoo. Reli- able fidelity and diversity metries for generative models. In Proc. the International Conference on Machine Learning (ICML), 2020.
[35] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), 2016.
[36] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324 1998.
[37] François Denis. PAC learning from positive statistical queries. In ALT, Lecture Notes in Computer Science, 1998.
[38] François Denis, Rémi Gilleron, and Fabien Letouzey. Learning from positive and unlabeled examples. Theor. Compur. Sci., 2005.
[39] Xiang Wei, Boqing Gong. Zixia Liu, Wei Lu, and Liqiang Wang. Improving the improved training of wasserstein gans: A consistency term and its dual effect. In Proc. the International Conference on Learning Representations (ICLR), 2018
[40] Hui Chen, Fangqing Liu, Yin Wang, Liyue Zhao, and Hao Wu. A variational approach for learning from positive and unlabeled data. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2020.
[41] Florent Chiaroni, Ghazaleh Khodabandelou, Mohamed-Chene Rahal, Nicolas Hueber, and Frédéric Dufaux. Counter-examples generation from a positive unlabeled image dataset. Pattern Recognition, 107:107527, 2020.
[42] Arantxa Casanova, Marlène Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero- Soriano. Instance-conditiones GAN. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2021.
[43] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2020.
[44] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2021.
12

---

[45] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. the International Conference on Learning Representations (ICLR), 2015.
[46] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proc. of the IEEE conference computer vision and pattern recognition (CVPR), 2018.
[47] Jae Hyun Lim and Jong Chul Ye. Geometric GAN. CoRR, abs//790,,,894, 2017.
[48] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normaliza- tion for generative adversarial networks. In Proc. the International Conference on Learning Representations (ICLR), 2018.
[49] Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), 2020.
[50] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 2008.
[51] Pavlin G. Policar, Martin Strazar, and Blaz Zupan. Embedding to reference t-sne space addresses batch effects in single-cell classification. In Discovery Science 22nd International Conference, DS, 2019.
[52] Ninarch Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. CoRR, abs/I 908 09635, 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] We provide the codes in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] We reported the standard deviation in the experimental results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] We used publicly available benchmark
datasets: MNIST, FMNIST, CIFARIO. 10, CelebA, and CelebA-HQ.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We provide the codes in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes] The benchmark datasets used in our paper are widely used for research purpose.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] The benchmark datasets used in paper widely used by others and do not contain any offensive contents.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "wNDOSkjosM", "reviewer": "Reviewer_UrwN", "summary": "This paper proposes a method to generate multi-label images by using a (conditional-)GAN architecture. Because it is quite expensive to annotate a large dataset with multi-label annotations, this paper focuses on the single positive setting where each image is annotated with a single positive label. The paper proposes a novel sampling framework based on the Markov chain Monte Carlo method for generating multi-label images from single positive labels. The proposed method shows good performances in two settings: sampling for classes with inclusion and sampling for multiple classes.", "questions": "- I wonder what was the motivation to focus only on the single positive setting instead of a more generic setting e.g. at least a single positive label. For example, OpenImage dataset has a lot of partially-labeled images.", "limitations": "I think the authors have adequately addressed the limitations and potential negative societal impact of their work", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n+ I think the idea of generating multi-label images is interesting. As mentioned in the paper, building large scale multi-label datasets is expensive so the approach of using partially labeled images looks very promising. Each image is annotated with a single positive label.\n+ The problem formulation with intersection index set and difference index set seems novel.\n+ The paper proposes a novel sampling framework based on the Markov chain Monte Carlo method for generating multi-label data from single positive labels.\n+ The proposed method is evaluated in two settings:\n  - sampling for classes with inclusion, where one class is entirely contained in another class. The proposed method shows better results on MNIST-3/5, MNIST-Even, and FMNIST-Even than RumiLSGAN and CP-GAN.\n  - sampling for multiple classes, where multiple classes are overlapping. The proposed method shows better results on CIFAR-7to3, CelebA-BMS, and CelebA-ABMNSW than existing models like CP-GAN.\n\n  For both settings, some visual results are shown and multiple metrics are used to evaluate the model performances.   \n\n+ The description of the limitations is interesting because it gives some insights about some potential next steps to improve the proposed method.\n\n**Weaknesses:**\n\n- The proposed approach seems to be specific to GAN, and cannot be used directly in other generative models like VAE. \n- There is no information on how the examples in ​​Figure 5 and 6 are chosen.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lMkUCIXflPX", "reviewer": "Reviewer_NQQq", "summary": "This paper proposed a sampling method based on MCMC to sample multi-label data from GANs trained with single positive labels. The authors prove that the target distribution can be derived from the conditional densities of single positive labels. Then they use several classifiers to estimate the density ratio and Metropolis-Hastings, an MCMC-based sampling method to draw samples. The method is technically sound and achieves good restuls in empirical studies.", "questions": "Why did the authors choose Metropolis-Hastings rather than other MCMC methods? Is there any reason or advantage?", "limitations": "The authors adequately addressed the limitations in Section 5 and the potential negative societal impact in Appendix F.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. The problem is interesting and valuable for reducing annotation costs.\n2. The sampling method is novel and can be applied to any existing GANs.\n3. The formulations and derivations are clear.\n4. The empirical studies, including experiments on 1-D Gaussian mixtures and real image datasets, well support the method's superiority with a clear margin.\n\nWeakness\n1. As discussed in Limitations, the method is hard to scale up for many attributes. The authors may consider moving some conclusions of complexity analysis from the appendix to the main paper.\n2. The experiment setting of sampling for multiple classes could be clearer with some examples. For instance, in CIFAR-7to3, class A includes images of airplanes, automobiles, dogs, and frogs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "g1VgaUqcTLY", "reviewer": "Reviewer_wf99", "summary": "This paper proposes a sampling method based on MCMC called single to multiple sampling (S2M). A few applications for this “add-on” method on unconditional and conditional GAN is applied to show the effectiveness of the proposed method.", "questions": "The main questions for this paper are :\n\n(1) Lack of sufficient comparison with other methods in this area in both theoretical study and experimental analysis (2) the motivation is not clear (especially for low data problem) (3) the insufficient experimental validation", "limitations": "N.A.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nThe proposed method seems to be able to help resolving the multiple label learning from single positive samples to some extent, which may serves as an interesting supplement to the existing methods, although it is not clear how good it is compared to other methods.\n\nWeakness: \n(1) One of the main concerns for this paper is that on the topic of multiple label learning from single positive samples, there are already quite a few solid study from previous work. For instance, in the reference [1]:\nCole et al “Multi-Label Learning from Single Positive Labels”, CVPR 2021.\nThe submission fails to connect their work to the reference [1]. Thus, it is not clear the importance of the S2M method for improving this type of problem. For example, how is the performance comparison of the proposed method vs the techniques described in [1] including “label smoothing, expected positive regularization, pseudo-negative sampling, etc” in both theoretical analysis and experimental study.\nWithout such important study, the novelty and significance of the paper does not stay on the safe ground.\n\n(2) It is also not clear how does the proposed method scale to large dataset. \n\n(3) While the paper proposes an interesting application to enhance the results with S2M sampling, the overall depth of the paper clearly falls below the Neurips acceptance bar. It will be nice that the author can motivate their method better. Does the S2M method provide any advantages compared to label smoothing, pseudo-negative sampling etc ?\n(4)  As we know, the sampling based method usually needs to accumulate sufficient number of samples to be effective, however, typically in the regime of multi-label learning from single positive labels, the data/labels are limited. Thus it is questionable that practically, how useful the method can be.\n(5) The experimental validation is not sufficient and convincing as it only considers a special case, which makes the application domain quite limited\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7OZeqCjSt_h", "reviewer": "Reviewer_E5Ue", "summary": "This paper focus on a setting that multi-label with only one positive label and try to generate the multi-label data with generative adversarial network (GAN). The proposed method considers both unconditional and class conditional version by a pre-trained GAN in a sampling strategy. The experimental results shows that the proposed method gets a promising result under the single positive setting.", "questions": "Please see my main review.", "limitations": "The work can only apply to the small datasets or small class space datasets and the experiments need to be further improved.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nOverall, the problem of multi-label with single positive label is interesting.\n\nThe paper is well written and organized.\n\nThe idea of using single class densities to provide an estimate for the multi-class target distribution is interesting.\n\nWeaknesses:\n\nThe datasets used in the experiments can be relatively small. It is better to adopt some large datasets to validate the effectiveness of the proposed methods.\n\nSince the paper focuses on the multi-label with singe positive label setting, it is better to use some datasets with large label space to see the performance the proposed method.\n\nI suggest the author to add an experiment about varying the label space of the data and show how the performance changes to further validate the multi-label data generation performance.\n\nOverall, this paper is novel to some extent. It is better that more promising results from some large datasets.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept