Generative Modeling Helps Weak Supervision (and Vice Versa)
Authors: Benedikt Boecking
Nicholas Roberts
Willie Neiswanger
Stefano Ermon
Frederic Sala
Artur Dubrawski
Venue: ICLR 2023 poster (2023)
Keywords: generative model
weak supervision

Abstract: Many promising applications of supervised machine learning face hurdles in the acquisition of labeled data in sufficient quantity and quality, creating an expensive bottleneck. To overcome such limitations, techniques that do not depend on ground truth labels have been studied, including weak supervision and generative modeling. While these techniques would seem to be usable in concert, improving one another, how to build an interface between them is not well-understood. In this work, we propose a model fusing programmatic weak supervision and generative adversarial networks and provide theoretical justification motivating this fusion. The proposed approach captures discrete latent variables in the data alongside the weak supervision derived label estimate. Alignment of the two allows for better modeling of sample-dependent accuracies of the weak supervision sources, improving the estimate of unobserved labels. It is the first approach to enable data augmentation through weakly supervised synthetic images and pseudolabels. Additionally, its learned latent variables can be inspected qualitatively. The model outperforms baseline weak supervision label models on a number of multiclass image classification datasets, improves the quality of generated images, and further improves end-model performance through data augmentation with synthetic samples.

Published as a conference paper at ICLR 2023
GENERATIVE MODELING HELPS WEAK SUPERVISION (AND VICE VERSA)
Benedikt Boecking' Nicholas Roberts Willie Neiswanger
Stefano Ermon* Frederic Sala Artur Dubrawski
*Carnegie Mellon University Univerritty of Wisconsin Sttanoord University
ABSTRACT
Many promising applications of supervised machine learning face hurdles in the acquisition of labeled data in sufficient quantity and quality, creating an expensive bottleneck. To overcome such limitations, techniques that do not depend ground truth labels have been studied, including weak supervision and generative modeling. While these techniques would seem to be usable in concert, improving one another, how to build an interface between them is not well-understood. In this work, we propose model fusing programmatic weak supervision and generative adversarial networks and provide theoretical justification motivating this fusion. The proposed approach captures discrete latent variables in the data alongside the weak super- vision derived label estimate. Alignment of the two allows for better modeling of sample-dependent accuracies of the weak supervision sources, improving the estimate of unobserved labels. It is the first approach to enable data augmentation through weakly supervised synthetic images and pseudolabels. Additionally, its learned latent variables can be inspected qualitatively. The model outperforms baseline weak supervision label models on a number multiclass image classifi- cation datasets, improves the quality of generated images, and further improves end-model performance through data augmentation with synthetic samples.
INTRODUCTION
How can we get the most out of data when we do not have ground truth labels? Two prominent paradigms operate in this setting. First, programmatic weak supervision frameworks use weak sources of training signal to train downstream supervised models, without needing access to ground- truth labels (Riedel et al. 2010 Ratner al. 2016) Dehghani al. [017] Lang & Poon 2021). Second, generative models enable learning data distributions which can benefit downstream tasks, e.g. via data augmentation or representation learning, in particular when learning latent factors of variation (Higgins et al. 2018 Locatello et al. 2019 Hu et al., 2019). Intuitively, these two paradigms should complement each other, each can thought as different approach to extracting structure from unlabeled data. However, to date there is no simple way to combine them.
Fusing generative models with weak supervision holds substantial promise. For example, could yield large reductions in data acquisition costs for training complex models. Programmatic weak supervision replaces the need for manual annotations by applying so-called labeling functions to unlabeled data, producing weak labels that are combined into a pseudolabel for each sample. This leaves the majority of the acquisition budget to be spent on unlabeled data, and here generative mod- eling can reduce the number of real-world samples that need to be collected. Similarly, information about the data distribution contained in weak label sources may improve generative models, reducing the need to acquire large volumes of samples to increase generative performance and model discrete structure. Additionally, learning with weak labels may enable targeted data augmentation, allowing for class-conditional sample generation despite not having access ground truth.
"boecking @ emmuedd,, awd@cmu.edu
nick1 rroberts @ww.wwii.e wis.wisc.uuu fredsala@ cs. wss.wic.edu iniiswanger @es. www..sstaoo.dddd... edu, ermon @ www.seefsssss....e..

---

Published as a conference paper at ICLR 2023
The main technical challenge is to build interface between the core models used the two approaches. Generative adversar- ial networks (GANs) (Goodfellow et al. 2014), which focus on in this work, have at least a generator and a discrimi- nator, and frequently additional auxiliary models, such as those that learn to disentan- gle latent factors of variation (Chen et al.] 2016). In programmatic weak supervision, the label model is the main focus. It is nec- essary to develop an interface that aligns the structures learned from the unlabeled data by the various components.
Class-conditional image generation by the pro- posed WSGAN based on weakly supervised CIFARIO sub- set with 30k samples. Here, WSGAN uses a StyleGAN2 basc architecture and we keep the discrete code in each row fixed.
Figure 1:
We introduce weakly-superrised GAN (WS- GAN), simple yet powerful fusion of weak supervision and GANs visualized in Fig. 2 and provide a theoretical justification that motivates the expected gains from this fusion. Our WSGAN approach is related to the unsupervised InfoGAN (Chen et al. (2016) generative model, and also inspired by encoder-basee ture in the data, and our approach ensures alignment between the resulting variables by learning projections between them. The proposed WSGAN offers a number of benefits, including:
label models as in (Cachay et al.] 2. These techniques expose struc-
Improved weak supervision: We obtain better-quality pseudolabels via WSGAN's label model, yielding consistent improvements in pseudolabel accuracy up to 6% over established programmatic weak supervision techniques such as Snorkel (Ratner et al. (020).
Improved generative modeling: Weak supervision provides information about unobserved labels which can be used to obtain better disentangled latent variables, thus improving the model's generative performance. Over 6 datasets, our WSGAN approach improves image generation by an average of 5.8 FID points versus InfoGAN. We conduct architecture ablations and show that the proposed approach can be integrated into state-of-the-art GAN architectures such as StyleGAN (Karras et al., 2019) (see Fig. D. achieving state-of-the-ar image generation quality.
Data augmentation via synthetic samples: WSGAN can generate samples and corresponding label estimates for data augmentation (e.g. Fig. [0), providing improvements of downstream classifier accuracy of up to 3.9% in our experiments. The trained WSGAN can produce label estimates even for samples, real or fake, that have no weak supervision signal available.
2 BACKGROUND
We propose fuse weak supervision with generative modeling the benefit of both techniques, and first provide brief overview. A broader review of related work is presented in Section 5
Weak Supervision Weak supervision methods that use multiple sources of imperfect and partial labels (Ratner al. 2016 2020; Cachay et al. 2021), sometimes referred as programmatic weak supervision, seek to replace manual labeling for the construction of large labeled datasets. Instead, users define multiple weak label sources that can be applied automatically to the unlabeled dataset. Such sources can be heuristics, knowledge base look-ups, off-the-shelf models, and more. The technical challenge is to combine the source votes into a high-quality pseudolabel via label model. This requires estimating the errors and dependencies between sources and using them to compute a posterior label distribution. Prior work has considered various choices for the label model, most of which only take the weak source outputs into account. A review can be found in Zhang et al. 22021] 2022). Instead, our label model produces sample dependent accuracy estimates for the weak sources based on the features of the data, similar to Cachay et al. (2021).
Generative Models and GANs Generative models are used to model and sample from complex distributions. Among the most popular such models are generative adversarial networks (GANs) (Goodrellow et al. 2014). GANs consist of a generator and discriminator model that play minimax game against each other. Our approach builds off InfoGAN (Chen et al. 2016). which adds an auxiliary inference component to learn disentangled representations via set of latent factors of variation. We hypothesize that connecting such discrete latent variables to the label model should yield benefits for both weak supervision and generative modeling.
2

---

Published as a conference paper at ICLR 2023
WSGAN 151
Ee-gete)log (ogg))
Feenttal Aogto log(1 (20(=
X
Bexpte) (1t,, ([G(+,
Fi
Alprean:
LM
á Bogget a ((PP) (((0)) FLZAAA)
Lia (
Fs
Figure 2: The proposed WSGAN models discrete latent variables in X via a network Q, while learning a generator C and discriminator D. A label model L uses weak supervision votes and weights estimated by A produce pseudolabels. WSGAN aligns the pseudolabels with the discrete structure learned by Q.
3 THE WSGAN MODEL
We first deseribe our proposed reakly-superviiee GAN (WSGAN) model, visualized in Fig. and then provide theoretical justification for the model fusion. We work with unlabeled samples X E X C Rd drawn from a distribution Dx. We want achieve two goals with the samples X. First, in generative modeling, approximate Dx with model that can be used to produce high-fidelity synthetic samples. Second, in supervised learning. we wish to use X to predict labels Y {1,2..... C). where (X,Y) is drawn from distribution whose marginal distribution is Dx- However, in the weak supervision setting, we do not observe Y. Instead, we observe m labeling functions (LFs) A fo...., ..... C)" n that provide imperfect estimates of Y for subset of the samples. These LFs vote on a sample x; to produce an estimate of the label Ay(x.) E {l,.... C} or abstain (i.e. vote) with O. The goal is to combine the LF estimates into pscudolabel Y that can be used to train a supervised model (Ratner et al. 2016). While weak supervision and generative modeling function over number of modalities, this work focuses images. Note that LF construction for image tasks is more challenging than for text tasks (cf. Section
3.1 PROPOSED METHOO
To improve generative performance and the weak supervision- seasred pseudolabels, we propose a model that consists of number of components. Because wish to ensure that component models benefit each other, our architecture aims for the following characteristics: (I) A generative model component that learns discrete latent factors of variation from data and exposes these externally, (II) weak supervision label model component that makes predictions of the unobserved label by aggregating the weak supervision votes, using sample-dependent weights, (III) models that connect the components. Our design choices are made to satisfy those goals.
of interface
GAN Architecture We write G for the generator; its goal is learn mapping to the image space based on input consisting of samples z from a noise distribution Pz( along with set of latent factors of variation we are targeting classification setting, we restrict ourselves to discrete b. The output of G are samples æ; these are consumed by a discriminative model D, which estimates the probability that sample came from the training distribution rather than G. Furthermore, Q which learns to map from a sample z to the discrete latent code 6. We denote the standard GAN objective by V(D,G), and the InfoGAN objective by IV(D,G,Q) (Chen et al. 2016):
p(b), following the ideas introduced in InfoGAN (Chen al. 2016). Because
define auxiliary model
min max V(D,G) = EE [log(D(=))] + Ez=p(z).6>p(() [log(1 - D(G(2, G D
(1)
min max IV(D, = V(D,G)+ (=sp(())bx=(()) [[b,, ([G((((G)))))))) 6))]],
(2)
G.Q
D
where is an appropriate loss function, such as cross entropy, and a trade-off parameter. Equation 2 aims maximize the mutual information between generated images and b, while G continues to fool the discriminator D, leading to the discovery of latent factors of variation.
Weak Supervision Label Model The purpose of the label model is to encode relationships between the LFs and the unobserved label y. enabling us to produce an informed estimate of y. In prior work, the model is often factor graph (Ratner et al. 2016 2019 Fu et al.) 2020 Zhang al. 2022) with potentials Φ (x). y) and dy.u (x, (z) (x) capturing the degree of agreement between an LF AÚ and y correlations between LFss Xj and Ap. We define the accuracy potentials

---

Published as a conference paper at ICLR 2023
(5() v) - 1 y) in related work. Each potential o, is associated with an accuracy
parameter f.. Once we obtain estimates of fj. we can predict y from the LFs À via
expl Ex 90 Lo(A)a ΣgEy exp exp(_x= (,0,(L)(2))0))
A (e,......,
This is softmax over the weighted votes of all LFs, which derives from the factor graph introduced in Ratner et al. (2016). Note that related work only models the L.F outputs to learn 0, ignoring any additional information in the features x. However, the structure in the input data J. is crucial to our fusion. For this reason, we define modified label model predictor in the spirit of [Cachay et al. (202T). has local accuracy parameters (sample-dependent values encoding the accuracy of each )₃) via an accuracy parameter encoder A(x)) Rd
RT.
This variant is given by:
exp(Ex= m A ((0)) (A)x Egeyexp( exp(E ((0))))((()))
AkE E (1.....c}.
softmax over the LF votes by class, weighted by the accuracy encoder output. Note that, while A(z) allows for finer- Amer-grained adjustments of the label estimate Y, the estimate still anchored the votes of LFs which represent strong domain knowledge and are assumed to be better than random.
Learning the Label Model The technical challenge of weak supervision is to learn the parameters of the label model (such as 0g above) without observing y. Existing approaches find parameters under label model that (i) best explain the LF votes while (ii) satisfying conditionally independent relationships (Ratner et al., 2016 2019 Fu et al. 2020). The features x are ignored; it is assumed that all information about y is present in the LF outputs. Instead, we promote cooperation between our models by ensuring that the best label model is the one which agrees with the discrete structure that the GAN can learn, and vice versa. The intuition is that, as each of the generative and label models learn useful information from data, this information can-if aligned correctly-be shared to help teach the other model. To this end, note that the sampled variable can only be observed for generated images, not for real images. Nonetheless, Q can be applied to real-world samples to obtain a prediction of the latent b. Crucially, in the weak supervision setting we observe the LF outputs. enabling us derive label estimate for each real image LA (A) Y, which can be aligned with the predicted code to guide Q on real data, and vice versa.
Interface Models and Overall WSGAN Objective We introduce the following interface models to map between the estimates of b and y. Let Fi [0, 1] - [0, 1] and F2 (0, 1] + [0. 1] An effective choice for Fj and F2 are linear models with softmax activation function. To achieve agreement between the latent structure discovered by the GAN's auxiliary model Q as well as by the label model LA, via the LFs, we introduce the following overall objective, ensuring that mapping exists between the latent structures on the real images in the training data:
(4) with hyperparameter 3 and loss function I, such as the cross entropy. Pseudocode for the added loss term can be found in Algorithm [ In our implementation, we let D. Q and A share convolutional layers and define distinct prediction heads for each. For LA we detach the features from the computation graph before passing them to multilayer perceptron (MLP), followed by sigmoid activation function. Thus, the WSGAN method only adds a small number of additional parameters compared to a basic GAN or InfoGAN.
min max (V + G.Q.A.Fi.F& D
+ B E.R,AAAD,... (MFF(QQ()) ( (A()) A
common in related GAN work,
Improving Alignment Initializing the label model LA such that it produces equal weights for all LFs results in strong baseline estimate of Ý, as users build LFs to be better than random. Initializing LA, in this way, it can act as teacher in the beginning and guide Q towards the discrete structure encoded in the LFs. We find that adding a decaying penalty term that encourages equal label model weights in early spochs-while not necessary to achieve good performance-almen always improves latent label estimates. Let i 2 denote the current epoch. We propose to add the following linearly decaying penalty term for an encoder A that uses sigmoid activation function:
C/(i x 7 + A(x) - Ix 0.51| where is decay parameter. In our experiments we set 7 = 1.5.
4

---

Published as a conference paper at ICLR 2023
Augmenting the Weak Supervision Pipeline with Synthetic Data Given WSGAN model trained according to Eq. 4 we can generate images via G to obtain unlabeled synthetic samples ž. To obtain pscudolabels for these images have at least one and sometimes two options. When LFs can be applied to synthetic images, we obtain their votes X()) X and apply our WSGAN label model (A). However, in many practical applications of weak supervision, some LFs are applied to images directly. but rather to metadata or an auxiliary modality such as text (cf. Section With WSGAN, we can obtain pseudolabels via Fi (Q())) for samples that have no LF votes, using the trained WSGAN components Q and F1. in essence transferring knowledge from Q to the end model. Note that the quality of these synthetic pseudolabels hinges on the performance of Q, which can conceivably improve with the supply of weakly supervised well as entirely unlabeled data.
3.2 THEORETICAL JUSTIFICATION
In this section, we provide theoretical results that suggest that there is provable benefit to combining weak supervision and generative modeling. In particular, we provide two theoretical claims justifying why weak supervision should help generative modeling (and vice versa): (1) generative models help weak supervision via a generalization bound on downstream classification and (2) weak supervision improves a multiplicative approximation bound on the loss for a conditional GAN using the unob- served true labels labels-namely, we extend the theoretical setup and noisy channel model of the Robust Conditional GAN (RCGAN) claims can be found in Appendix l
(TeEAA) (Thekumparampil al., 2018). Formal statements and proofs of these
Claim (1) Assume that we have ni unlabeled real examples where our label model fails to produce labels, i.e. all LFs abstain on these Y4 points. This is typical issue in weak supervision, as sources often only vote on a small proportion of points. We then sample enough synthetic examples from our generative model such that we obtain n2 synthetic examples for which our label model does produce labels; this enables training of downstream classifier on synthetic examples alone with the following generalization bound:
log(1/6)
sup
(R RD(f) 29 + fEF
2nz + B,G's + B.V BBVV2exp(-mel
where R is the Rademacher complexity of the function class. The first two terms standard. The third term is the penalty due to generative model usage; any generative model estimation result for total variation distance can be plugged in. For example, for estimating mixture of Gaussians, G (4" /n, )t/2 which depends on the number of mixture components k and dimension d. The last term is the penalty from weak supervision with m LFs whose accuracy is a better than chance; this implies that generated samples can help weak supervision generalize when true samples cannot.
Claim (2) Noisy labels from majority vote improve the multiplicative bound on the RCGAN loss given in Theorem 2 of TThhkumparmmpl et al. (018). Let P and Q be two distributions over X' x {0, 1) and let PaY and QMv be the corresponding distributions with noisy labels generated by majority vote over m LFs. Let d= QMv) be the RCGAN loss with noisy labels generated by majority vote and let ex be the mean error of each of the m LFs. Using majority vote with m 0.5 log(1/ex)/ (1 <x)² LFs, we obtain an exponentially tighter multiplicative bound on the noiseless RCGAN loss:
ds ΦMv) S ()( < 1- eexp -2m ((-=)))
-
dy (P Quav)
< (1 22x) -1 QMv).
This means that weak supervision can help RCGAN more-accurately learn the true joint distribu- tion, even when the true labels are unobserved. The full analysis is provided in Appendix F
4 EXPERIMENTS
Our experiments on multiple image datasets show that the proposed WSGAN approach is able to take advantage of the discrete latent structure it discovers in the images, leading better label model performance compared to prior work. The results also indicate that weak supervision as used by WSGAN can improve image generation performance. In the spirit of democratizing AI, aim to
5

---

Published as a conference paper at ICLR 2023
Table 1: Datasets and labeling function (LF) characteristics used evaluate the proposed WSGAN. Acc denotes accuracy, and Coverage denotes the proportion of samples where the LF does not abstain.
Dataset
#Classes
#LFs
#Samples
Mean LF Acc
Min LF Acc
Max LF Acc
Mean Coverage
LF Type
AwA2- -A
10
29
6726
0.504
0.053
0,850
0.104
Attribute beuristics
AwA2
10
0.548
0.116
0.131
Attribute heuristics
DomainNet
10
4
6369
0.493
0.416
0.684
1,000
Domain transfer
MNIST
10
30000
0.564
0.047
SSL., finetuning
PashionMNIST
10
23
30000
0.773
0,542
0,949
0.047
SSL, finctuning
GTSRB
43
100
22640
0.837
0.609
0,949
0.007
SSL, finetuning
CIFARIO-A
10
20
30000
0.773
0.624
0,896
0.061
Syathetic
CIFARIO-B B
10
20
30000
0.736
0.531
0.912
0.042
SSL,, finetuming
keep the complexity of our experiments manageable, to ensure accessible reproducibility. Therefore, conduct our main experiments with simple DCGAN base architecture. As ablation, also adapt StyleGAN2-AD. (Karras et al. 2020) to WSGAN, showing that the proposed method can be integrated with other GAN architectures achieve state-of-the-art image generation and label model performance. Please see the Appendix for additional details and experiments as well as a link to code.
4.1 SETUP
Datasets Table shows key characteristics of the datasets used in our experiments, including information about the different LF sets. We conduct our main experiments with the Animals with Attributes (2 (Mazzetto et al.) (021a), DomainNet (Peng et al.. 2019), the German Traffic Sign Recognition Benchmark (GTSRB) (Stallkamp et al.. 2012), and CIFARIO (Krizhevsky) (209)) color image datasets, as well as with the gray-scale MNIST (LeCun et al. 1998) and Fashion (iiao al.) 2017) datasets. We use a variety of types of weak supervision sources for these datasets (see Appendix B for more dataset details). The LF types we cover are:
Domain transfer: classifiers are trained on images in source domains (e.g. paintings), and the trained classifiers are then applied to images in target domain (e.g. real images) to obtain weak labels. This I.F type is used our DomainNet experiments, following Mazzetto et al. (2021a). Attribute heuristics: we use these LFS in our AwA2 experiments. Attribute classifiers are trained on a number of seen classes of animals. Given these weak attribute predictions, we use the known attribute relations and a small amount of validation data to train shallow decision trees to produce weak labels for a set of unseen classes of animals.
SSL-based: using image features learned on ImageNET with SimCLR (Chen et al. 2020), we fine-tune shallow multilayer perceptron classifiers on small sets of held-out data to produce weak labels for our datasets.
Synthetic: these simulated LFs, used in some of our CIFARIO experiments, are unipolar LFs based on the corrupted true class label. To this end, random errors are introduced to the class label to achieve sampled target accuracy and propensity.
Models We study two versions of the proposed WSGAN model: (1) WSGAN- which uses an accuracy parameter encoder A(x), that takes in an image x and outputs an accuracy weight vector for the label model. (2) WSGAN-Vector, baseline which learns parameler vector that is used to weigh LF votes and is not sample-dependent.
For our main experiments, G, D follow a simple DCGAN (Radford et al. 2015) design. All networks trained from scratch and we the same hyperparameter settings all experiments. For our architecture ablation, we adapt StyleGAN2-ADA (Karras et al. 2020) to create StyleWSGAN. See Appendix for implementation details and parameter settings.
We compare WSGAN to the following label model approaches: (I) Snorkel (Ratner et al. 2016 2020): a probabilistic graphical model that estimates LF parameters by maximizing the marginal likelihood using observed LFs. (II) Dawid-Skene (Dawid & Skene 1979): model motivated by the crowdsourcing setting. The model, fit using expectation maximization, assumes that error statistics of sources are the same across classes and that errors are equiprobable independent of the true class. (II) Snorkel MeTaL (Ratner et al. 2019): Markov random field (MRF) model similar to Snorkel which uses a technique complete the inverse covariance matrix of the MRF during model fitting, and also allows for modeling multi-task weak supervision.
(IV) FlyingSquid (FS) (Fu et al. 2020): based on label model similar Snorkel, FS provides a closed form solution by augmenting it to set up binary Ising model, enabling scalable model fitting. (V) Majority Vote (MV): A standard scheme that uses the most popular LF output as the estimate of the true label.

---

Published as a conference paper at ICLR 2023
Table 2: Average posterior accuracy of various label models on training samples with at least one LF vote. We highlight the best result in blue and the second best result in bold.
Dataset
MV
DawidSkene
McTal.
FS
Snorkel
WSGAN- Vector
WSGAN-
AwA2 0.631
0.607
0.632
0.615
0.641
0.647
0.681
AwA2
0.623
0.548
0.582
0.602
0.605
0.645
0.699
DomainNet
0.614
0.658
0.487
0.635
0.499
0.661
0.643
MNIST
0.775
0.729
0.766
0.773
0.766
0.782
0,813
FashionMNIST
0.735
0.717
0.730
0.734
0,729
0.737
0.744
GTSRB
0.816
0.619
0.815
0.671
0.814
0.815
0.823
CIFARIO-A
0.827
0.850
0.806
0,800
0.807
0,850
0.874
CIFARTO-B 0.716
0.677
0.708
0.708
0.707
0.725
0.731
Evaluation Metries As common in related work, label model performance is compared based on the pseudolabel accuracy the models achieve on the training data, since programmatic weak supervision operates in transductive setting. Weighted Fl and mean Average Precision provided in Appendix E To compare the quality of generated images, we use the Fréchet Inception Distance (FID) on color images, which has been shown be consistent with human judgments (Heusel et al.) 2017 7 and is used to measure performance of current state-of-the-art GAN approaches (Karras et al. 2021 To show the improvement in alignment of the auxiliary model Q's predictions the discrete latent code b with the latent labels y, we track the Adjusted Rand Index (ARI) between the two.
4.2 RESULTS
We first discuss results comparing label model and image generation performance, before presenting the use of WSGAN for augmentation of the downstream classifier with synthetic samples. We repeat each experiment at least three times and average the results in our tables.
4.2.1 LABEL MODEL AND IMAGE GENERATION
Label Model Performance Table 2 shows a comparison label model performance based on the accuracy of the posterior on the training data, without the use of any labeled data validation sets. WSGAN-encoder largely outperforms alternative label models, while the simpler WSGAN-veecor model performs competitively well. These results hold according additional metrics provided in Appendix C Results with standard deviations over random runs are provided in the Appendix in Table 11 indicating that many differences are significant.
Discrete Latent Code Comparison Figure shows the evolving ARI between the ground truth and the auxiliary model Q's prediction of the latent code on real data during model training. The figures show a large improvement in Q's ability to uncover the unobserved class label structure when comparing WSGAN InfoGAN, which is expected as WSGAN can take advantage of the weak signals encoded in LFs, while InfoGAN is completely unsupervised.
Image Generation Performance Table compares FID of generated color images, suggesting that WSGAN models do take ad- vantage of the weak supervision signal to improve Q, thereby improving the quality Info- GAN using the same base DCGAN archi- tecture. WSGAN has a lower FID only the GTSRB dataset, likely due to GTSRB's class imbalance and the dataset difficulty (43 classes, <23k samples).
Table Color image generation quality (mean FID). The best scores are highlighted in blue.
Dataset
InfoGAN
WSGAN-V
WSGAN-E
AwA2 A
41.62
36.74
34.71
of generated images compared
AwA2
41.62
36.79
34.52
DomainNet
53.98
50.16
44.35
GTSRB
69.67
75.27
73.96
CIFARIO-A
28.93
25.70
22.71
CIFAR10-B
33.50
26.17
24.41
4.2.2 DATA AUGMENTATION
We record the change in test accuracy for a ResNet-18 (He et al. 2016 end model when adding 1. 1,000 synthetic WSGAN-encoder images augment each dataset. While the increases are modest, the process is beneficial and does not require additional human labeling or data collection efforts. Adding larger amounts of synthetic samples did not lead further increases, possibly due to the limited image quality achieved by the basic DCGAN design explored in this section.
7

---

Published as a conference paper at ICLR 2023
0..4
0.4
0.4
ARI
0.4
0,0
0.0
0.0
0.0
50
100
150
200
50
100
150
200
50
100
150
200
25
50
73
100
GTSRB
CIFARIO-A
CIFARIO-B
FashionMNIST
0.4
0.4
0.5
ART
0.4
0.2
0.2
WSGAN
0.0
30
200
100
150
200
50
100
100
Epach
Epoch
Epocs
Epoch
Figure 3: Adjusted Rand Index of the unobserved and the code predictions (xx) real images Weak supervision allows WSGAN to better uncover latent y compared to an unsupervised InfoGAN.
Synthetic Images with Labeling Function Votes The last column in Table displays test accuracy in- creases by applying LFs À to synthetic images τ. We obtain pseudolabels via LA(a) (A(T). We observe a modest average increase of 1.38%.
Table 4:
Test accuracy increase when aug- menting downstream model training with 1,000 synthetic images and pseudolabels (PLs). Synthetic PL. are obtained via Fi (Q)())), LF PLS via L(z) (A()))
Synthetic Images with Synthetic Pseudolabels We can create pseudolabels with Fi (Q(()))) e.g. when I.Fs cannot be applied to synthetic images. With this, the second column of Table 4 shows an average increase in test accuracy of 1 1%, and up to 2.4%. We do not observe larger increases in accuracy by adding more generated images. Figure 10 shows small number of generated images along with syn- thetic pseudolabel estimates. While Fi (Q(x)) could conceivably be used as a downstream classifier, the choices of network architecture are then constrained as it shares convolutional layers with D.
Dataset
Synthetic PLs
LF PLs
AwA2 A
0.88%
0.79%
AwA2 B
2.40%
3,90%
DomainNet MNIST
2.31%
1.50%
1.60%
1.71%
FashionMNIST
0.29%
0.34%
GTSRB
0.40%
0.02%
CIFARIO- 10-R
0.04%
CIFARIO-B 0.30%
Synthetic Data Quality Checks In addition to visually inspecting some generated samples and checking if conditionally generated samples reflect the target labels, we recommend checking the class balance in the pseudolabels of synthetic images before adding them downstream training set, as mode collapse in trained GAN can potentially be diagnosed this way.
4.2.3 NETWORK ABLATION - STYLEWSGAN
We apply StyleWSGAN weakly supervised LSUN scene categories (Yu et al. 2015), and to our CIFARIO- dataset, please see Appendix C.2 for details. The results demonstrate WSGAN's complementarity with other GAN architectures and that it scales images of higher resolution. On weakly supervised LSUN scene category images with resolution of 256 by 256 pixels, StyNeWSGAN achieves mean FID of 7.54 (samples visualized in Fig. (8), while an unconditional, tuned StyleGAN2- ADA (Krras et al. 2020) achieves an FID of 8.41. On CIFAR10- 10-B, StyleWSGAN achieves a mean FID of 3.79 (see generated images in Fig. 9). while also attaining high label model accuracy of 0.736 (compare with Table 2). The unsupervised StyleGAN2-ADA with the optimal, tuned settings identified in Karras et al. (2020), achieves an average FID of 3.85 on this subset. An unsupervised StyleInfoGAN that created achieved mean FID of 4.13.
RELATED WORK
Programmatic Weak Supervision Data programming (DP) (Ratner al. 2016 popular weak supervision framework in which subject matter experts programmatically label data through multiple noisy sources of labels known as labeling functions (L.Fs). These I.Fs capture partial knowledge about an unobserved ground truth variable at better than random accuracy. In DP, label model combines LF votes to provide an estimate of the unobserved ground truth, which then used to train end model using a noise-aware loss function. DP has been successfully applied to various domains including medicine (Fries et al. 2019 Dunnmon et al. 2020 Eyuboglu et al. 2021) and industry applications (Ré et al. 2020: Bach et al. 2019). Many works offer DP label models with improved properties, e.g., extensions to multitask models (Ratner et al.) 2019), better computational efficiency (Fu et al. 2020), exploiting small amounts of labels as in semi-supervised
8

---

Published as a conference paper at ICLR 2023
learning settings (Chen et al.
2021 Mazzetto et al. 202 b), end-to-end training (Cachay et al.
2021), interactive learning [Boecking et al. 2021). p. or extensions to structured prediction settings
(Shin et al. 2022). See Zhang et al. (2022) for more detailed survey.
Programmatic Weak Supervision and Images Our main focus is on applications of weak super- vision to image data. On images, imperfect labels are often obtained from domain specific primitives and rules Vaarma & Ré, 2018; Fries et al. 2019). rules defined on top of annotations by surrogate ) rules defined meta-data (Li Denton et al.] 2013) or rules applied
models (Varma& & Réj 22188 Chen al. 2019| Hooper al.
2021),
& Fei-Feij 20100 Chen & Guptal 2015; Izadinia et al. 2015
t0 second paired modality such as text (Joulin al. 2016
Wang al. [0177] Irvin et al. 2019
Boecking et al. 2021 Dunnmon et al. 2020: Saab et al.
2019 Eyuboglu et al. 2021).
Among the numerous existing approaches to generative modeling, in this work we focus on generative adversarial networks (GANs) (Good] fellow et al. 2014). We are particularly interested in work that aims to learn disentangled repre- sentations (Chen et al. 2016 Lin et al. 2020) that can align with class variables of interest. Chen al.] (2016) introduce InfoGAN, which learns interpretable latent codes. This is achieved by max- imizing the mutual information between fixed small subset of the GAN's input variables and the generated observations. Gaabaay & Hoshen (2020) present a unified formulation for class and content disentanglement as well as a new approach for supervspe content disentanglement. Nie (ct al. (2020) study semi-supervised high-resolution disentanglement learning for the state-of-the-ar StyleGAN architecture. A potential downside to modeling latent factors in generative models is decrease in image quality of generated samples that has been noted when disentanglement terms are added (Burgess al. 2018 Khrulkov al. [02T).
Generative Models and Disentangled Representations
Prior work has studied how to integrate additional information into GAN training, in particular ground truth class labels (Mirza & Osindero| 2014; Salimans et al.) 2016 Odena 2016 Odena al. 2017] Brock et al. 2019 [Thekumparampil et al.
2018 Miyato & Koyama 2018 Lučié et al. 2019). also
considering noisy scenarios (Kaneko et al. 2019). However, in the programmatic weak supervision setting, having multiple noisy sources of imperfect labels that include abstains present large hurdles to similar conditional modeling. Some prior work uses other weak formats of supervision to aid specific aspects of generative modeling. For example, Chen & Batmanghelich (2020) propose learning disentangled representation using user-provided ground-truth pairs. Yet, prior work does not fuse programmatic weak supervision frameworks and generative models, and so are limited to one-off techniques to solely improve generative models.
Using GANs for Data Augmentation An exciting application of GANs is generate additional samples for supervised model training. The challenge is to produce sufficiently high-quality samples. conditional GAN to generate synthetic images of tomato plant leaves for a disease detection task. GANs for data augmentation are also popular in medical imaging YYi et al.) 2019 Motamed et al. 2021 Hu al. 2019). For example, [Hu et al. (2019) use an InfoGAN-lit model to learn cell-level representations in histopathology, Motamed et al. (2021) augment radiology data, and Pascual al. (2019) generate synthetic epileptic brain activities.
For example, Abbas al. (2021)
6 CONCLUSION
We studied the question of how to build an interface between two powerful techniques that operate in the absence of labeled data: generative modeling and programmatic weak supervision. Our fusion of the two, a weakly supervised GAN (WSGAN), defines an interface that aligns structures discovered in its constituent models. This leads to three improvements: first, better quality pseudolabels compared to weak supervision alone, boosting downstream performance. Second, improvement in the quality of the generative model samples. Third, it enables data augmentation via generated samples and pseudolabels, further improving downstream model performance without additional burden on users.
Standard failure cases of GANs such as mode collapse still apply to the proposed approach. However, we do not observe that WSGAN is more susceptible to such failures than the approaches we compare to. For future work, we are interested in other modalities, exploiting for instance generative models for graphs and time series. Further, motivated by the performance of WSGAN, we seek to extend the underlying notion of interfaces between models to a variety of other pairs of learning paradigms. Limitations of the proposed approach include common GAN restrictions such as the types of data that can be modeled and the number of unlabeled samples required to fit distributions, and also known difficulties of acquiring weak supervision sources of sufficient quality for image data.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was partially supported by a Space Technology Research Institutes grant from NASA's Space Technology Research Grants Program and the Defense Advanced Research Projects Agency's 's award FA8750-17-2-01300 This work was also supported in part by NSF (#1651565, #CCF2106707) and ARO (W911NF2110125), and the Wisconsin Alumni Research Foundation (WARF).
REFERENCES
Amreen Abbas, Sweta Jain, Mahesh Gour, and Swetha Vankudothu. Tomato plant disease detection using transfer learning with c-gan synthetic images. Computers and Electronics in Agriculurre, 187:106279, 2021.
Hassan Ashtiani, Shai Ben-David, Nicholas JA Harvey, Christopher Liaw, Abbas Mehrabian, and Yaniv Plan. Nearly tight sample complexity bounds for learning mixtures gaussians via sample compression schemes. In Proceedings of the 32nd International Conference Neural Information Processing Systems. pp. 3416-3425, 2018.
Stephen H Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alex Ratner, Braden Hancock, Houman Alborzi, al. Snorkel drybell: A case study in deploying weak supervision at industrial scale. In Proceedings of the 2019 International Conference on Management of Data, pp. 362-375, 2019.
Benedikt Boecking, Willie Neiswanger, Eric Xing. and Artur Dubrawski. Interactive weak super- vision: Learning useful heuristics for data labeling. In International Conference on Learning Representations, 2021.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference Learning Representations, 2019.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 3-vae. arXiv preprint PrXivv/804.03555 2018.
Salva Rühling Cachay, Benedikt Boecking, and Artur Dubrawski. End-to-end weak supervision. Thirty-fifth Conference on Neural Information Processing Systems, 2021.
Junxiang Chen and Kayhan Batmanghelich. Weakly supervised disentanglement by pairwise similar- ities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3495-3502, 2020.
Mayee Chen, Benjamin Cohen- Cohen-Wang, Stephen Mussmann, Frederic Sala, and Christopher Ré. Comparing the value of labeled and unlabeled data method-oo-mmoments latent variable estimation. In International Conference Artificial Intelligence and Statistics, pp.
PMLR, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597-1607. PMLR, 2020.
Vincent Chen, Sen Wu, Alexander J Ratner, Jen Weng, and Christopher Ré. Slice-based learning: programming model for residual learning in critical data slices. In Advances in Neural Information Processing Systems, pp. 9392-9402, 2019.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, 2016.
Xinlci Chen and Abhinav Gupta. Webly supervised learning of convolutional networks. In Proceed- ings of the IEEE International Conference Computer Vision, pp. 1431-1439, 2015.
Alexander Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28 (1):20-28, 1979.
10

---

Published as a conference paper at ICLR 2023
Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W. Bruce Croft. Neural ranking models with weak supervision. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 65-74, Association for Computing Machinery, 2017.
Emily Denton, Jason Weston, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. User conditional hashtag prediction for images. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1731-1740, 2015.
John Duchi. Lecture notes for statistics 311/elctriccal engineering 377. URL: https://stanford. httpaa////sat3/////eee/e////ee// pnttessslltntts...ooooe pdf. Last visited on, 2:23, 2016.
Jared A. Dunnmon, Alexander J. Ratner, Khaled Saab, Nishith Khandwala, Matthew Markert, Hersh Sagreiya, Roger Goldman, Christopher Lee-Messer, Matthew P. Lungren, Daniel L.. Rubin, and Christopher Re. Cross-modal data programming enables rapid medical machine learning. Patterns, 1(2):100019, 2020.
Sabri Eyuboglu, Geoffrey Angus, Bhavik N Patel, Anuj Pareek, Guido Davidzon, Jin Long, Jared Dunmmon, and Matthew P Lungren. Multi-task weak supervision enables anatomically-resovee abnormality detection in whole-body fdg-pet/ct. Nature Communications, 12(1):1-15, 2021.
Jason A Fries, Paroma Varma, Vincent S Chen, Ke Xiao, Heliodoro Tejeda, Priyanka Saha, Jared Dunnmon, Henry Chubb, Shiraz Maskatia, Madalina Fiterau, et al. Weakly supervised classification of aortic valve malformations using unlabeled cardiac mri sequences. Nature Communications, 10 (1):1-10, 2019.
Daniel Y. Fu, Mayee F. Chen, Frederic Sala, Sarah M. Hooper, Kayvon Fatahalian, and Christopher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods, 2020.
Aviv Gabbay and Yedid Hoshen. Demystifying inter-class disentanglement. In International Conference on Learning Representations, 2020.
Ian Goodfellow, Jean Pouget-Abadie. Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672-2680, 2014.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Berhaard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv: 2018
Sarah Hooper, Michael Wornow, Ying Hang Seah, Peter Kellman, Hui Xue, Frederic Sala, Curtis Langlotz, and Christopher Ré. Cut out the annotator, keep the cutout: better segmentation with weak supervision. International Conference on Learning Representations, 2021.
B. Hu, Y. Tang, F. I. C. Chang, Y. Fan, M. Lai, and Y. Xu. Unsupervised learning for cell-level visual representation in histopathology images with generative adversarial networks. IEEE Journal of Biomedical and Health Informatics, 23(3):1316-133 1328,, 2019. doi: 10. 10111/99..201112222222222
Jeremy Irvin. Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Guurea-Ieuus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 590-597, 2019.
Hamid Izadinia, Bryan C Russell, Ali Farhadi, Matthew D Hoffman. and Aaron Hertzmann. Deep classifiers from image tags in the wild. In Proceedings of the 2015 Workshop on Community- Organized Multimodal Mining: Opportunities for Novel Solutions, pp. 13-18, 2015.
11

---

Published as a conference paper at ICLR 2023
Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data, In European Conference on Computer Vision, pp- 67-84. Springer, 2016.
Takuhiro Kaneko, Yoshitaka Ushiku, and Tatsuya Harada. Label-noise robust generative adver- sarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2467-2476, 2019.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data, Advances Neural Information Processing Systems, 13:12104-1211 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alies-free generative adversarial networks. Advances in Neural Information Processing Systems, 34:852-8653 2021.
Valentin Khrulkov, Leyla Mirvakhabova, Ivan Oseledets, and Artem Babenko. Disentangled repre- sentations from non-disentangled models. arXiv preprint arXiv:2102.06244, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, Ontario, 2009.
Hunter Lang and Hoifung Poon. Self-supervised self-supervision by combining deep learning and probabilistic logic. In In Proceedings of the Thirty Fifth Annual Meeting of the Association for the Advancement of Artificial Intelligence (AAAI), 2021.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-25 2324, 1998.
Li-Jia Li and Li Fei-Fei. Optimol: automatic online picture collection via incremental model learning. International Journal of Computer Vision, 186 2010.
Zinan Lin, Kiran Thekumparampil, Giulia Fanti, and Sewoong Oh. Infogan-cr and modelcentrality: Self-supervised model training and selection for disentangling gans. In International Conference on Machine Learning. pp. 6127-6139. PMLR, 2020.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf. and Olivier Bachem. On the fairness of disentangled representations. Advances in Neural Information Processing Systems, 32, 2019.
Mario Lučié, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, and Sylvain Gelly. High-fidelity image generation with fewer labels. In International Conference on Machine Learning, pp. 4183-4192. PMLR, 2019.
Alessio Mazzetto, Cyrus Cousins, Dylan Sam, Stephen H Bach, and Eli Upfal. Adversarial multi class learning under weak supervision with performance guarantees. In International Conference on Machine Learning, pp. 7534-7543. PMLR, 2021a.
Alessio Mazzetto, Dylan Sam, Andrew Park, Eli Upfal, and Stephen Bach. Semi-supervised ag- gregation of dependent weak supervision sources with performance guarantees. In International Conference on Artificial Intelligence and Statistics, pp. 3196-3204. PMIR, 2021b.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. In International Conference Learning Representations, 2018.
Saman Motamed, Patrik Rogalla, and Farzad Khalvati. Data augmentation using generative adver- sarial networks (gans) for ganebased detection pncumonia and covid-19 in chest x-ray images. Informatics in Medicine Unlocked, 27, 2021.
12

---

Published as a conference paper at ICLR 2023
Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debnath, Anjul Patney, Ankit Patel, and Animashree Anandkumar. Semi-supervised stylegan for disentanglement learning. In International Conference on Machine Learning, pp. 7360-7369. PMLR, 2020.
Augustus Odena. Semi-supervised learning with generative adversarial networks. arXiv preprint 588,, 2016.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary PMLR, 2017.
classifier gans. In International Conference on Machine Learning, pp.
Damián Pascual, Amir Aminifar, David Atienza, Philippe Ryvlin, and Roger Wattenhofer. Synthetic epileptic brain activities using gans. Machine Learning for Health (MLAH) at NeurlPS, 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1406-1415, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv preprint arXiv:1511.0644 2015.
Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederie Sala, Shreyash Pandey, and Christopher Ré. Training complex models with multi-task weak supervision. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4763-4771, 2019.
Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. The VLDB Journal, 29(2):709-730, 2020.
Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. In Advances in Neural Information Processing Systems, pp. 3567-3575, 2016.
Christopher Ré, Feng Niu, Pallavi Gudipati, and Charles Srisuwananukorn. Overton: A data system for monitoring and improving machine-learedd products. In Proceedings of the 10th Annual Conference on Innovative Data Systems Research, 2020.
Sebastian Riedel, Limin Yao, and Andrew McCallum. Modeling relations and their mentions without labeled text. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 148-163. Springer, 2010.
Khaled Saab, Jared Dunnmon, Roger Goldman, Alex Ratner, Hersh Sagreiya, Christopher Ré, and Daniel Rubin. Doubly weak supervision of deep learning models for head ct. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 811-819. Springer, 2019.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in Neural Information Processing Systems, 29: 2234-2242, 2016.
Changho Shin, Wunfred Li, Harit Vishwakarma, Nicholas Roberts, and Frederic Sala. Universalizing weak supervision. International Conference on Learning Representations, 2022.
Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks. 0, 2012.
Kiran K Thekumparampil, Ashish Khetan, Zinan Lin, and Sewoong Oh. Robustness of conditional gans noisy labels. In Advances in Neural Information Processing Systems, volume 31, 2018.
Paroma Varma and Christopher Ré. Snuba: automating weak supervision label training data. Proceedings of the VLDB Endowment, 12(3):223-236 2018.
Xiaosong Wang. Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks weakly-supervee classifi- cation and localization of common thorax diseases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2097-2106, 2017.
13

---

Published as a conference paper at ICLR 2023
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXin:1008.07777, 2017.
Xin Yi, Ekta Walia, and Paul Babyn. Generative adversarial network in medical imaging: A review. Medical Image Analysis, 58:101552, 2019.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of scale image dataset using deep learning with humans in the loop. arXiv preprint orXiv:/506.000
2015.
Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang. Mao Yang, and Alexander Ratner. Wrench: A comprehensive benchmark for weak supervision. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Jieyu Zhang, Cheng- Yu Hsich, Yue Yu, Chao Zhang, and Alexander Ratner. A survey on program matic weak supervision. arXiv preprint arXiv:2202.05433, 2022.
14

---

Published as a conference paper at ICLR 2023
APPENDIX
Algorithm 1 Pseudocode for the proposed WSGAN loss term which is added to the basic InfoGAN loss. Input images and LFs are assumed be filtered to only contain samples with least non-abstaining I.F vote.
input: Batch of real images X and one-hot encoded LFs A, label model L, networks Q, F1, F2. WSGAN mode m, number of classes C, current epoch i, 7 decay parameter. 6. Z = Q(X) # get predicted code and image features if m "vector":
= Ao() # WSGAN-vector; get weight vector.
else:
0 = A(Z.delach() # WSGAN-encoder: predict weights using image features Z.
T. # Get label estimate from labelmodel using weights 0.
# Compute cross-entropy losses
loss = celoss(F) [1 (b), j.detach()
loss += celoss(F2(t)) b b.detach())
loss += C/(i x 7 1) (asemmsseeee) 0, I x 0.5) # add decaying loss keeping weights uniform.
return loss
A IMPLEMENTATION DETAILS AND COMPLEXITY
Code for WSGAN can be found at httea//gi.t////////o/oooo/oooood
(WS)GAN Models The following design choices were used for the experiments conducted with simple DCGAN base networks (as opposed to the settings used in the StyleGAN ablations).
Generator G, Discriminator D, and auxilliary model Q: Figures 5 and f show the simple DC- GAN (Radford et al. 2015) based generator and discriminator architectures we use in WSGAN for experiments with 32 x 32 images. As mentioned in the main paper, Q and D are neural networks that generally share all convolutional layers, with final fully connected layer to output predictions. We follow the same structure in our experiments. We set the dimension of the noise variable z to 100, and of equal to the number of classes. We sample from normal distribution and b from uniform discrete distribution.
Accuracy Encoder A: For WSGAN-Vector A simply a parameter vector of the same length as the number of labeling functions. For VYSGAN-Encodee we use image features obtained from the shared convolutional layers of Q and D, which we detach from the computational graph before passing them on to an MLP prediction head. For images with 32 x 32 pixels, the feature vector obtained from the shared convolutional layers is of size 512 16. The MLP head of A set to have three hidden layers of size (256, 128, 64), with ReLU activations, and an output layer the size of the number of labeling functions followed by a sigmoid function. We did not observe significant changes in performance when we change the MLP to be shallower wider. However, for large numbers of LFs, one should consider increasing the width the MLP.
Mappings F1, F2: We set F1 and F2 to each be simple linear models with softmax at the output, and set the input and output size of each to the number of classes.
(WS)GAN Training We use the same hyperparameter settings for all datasets. We train all GANs for maximum of 200 epochs. We use batch size of 16 and find that a lower batch size leads to more frequent convergence of the generator and discriminator. We also conducted ablation experiments with a batch size of 8 and 32 and found no significant difference in FID image generation quality or label model accuracy. For WSGAN, we use four optimizers, one for each of the different loss terms: discriminator training, generator training, the Info loss term, and the WSGAN loss term. We use Adam for all optimizers and set the learning rates as follows: 4 x 10- for D, 10 for G, 1 x 10 for the info loss term, and 8 x 10 for the WSGAN loss term. We follow the same settings for InfoGAN training for the components shared with WSGAN.
15

---

Published as a conference paper at ICLR 2023
(WS)GAN Training and Failure Cases While WSGAN is still susceptible the common GAN failure cases of its base networks, such as mode collapse, we empirically find WSGAN training to be more stable than training a GAN that also learns discrete latent code but uses weak supervision signals (InfoGAN), despite the high level of noise in our weak supervision sources. InfoGAN failed to converge more frequently.
To help train the DCGAN networks successfully, we find that employing discriminator label flipping (randomly calling tiny percentage of real samples fake and vice versa) and label smoothing (adding small amounts of noise to the real target of 1.0 and fake target of 0.0) stabilizes and improves GAN training. Despite employing these tricks, we were unable to avoid occasional convergence failures. Fortunately. monitoring the generator and discriminator losses, inspecting the quality of generated images, or tracking image quality metrics such as FID allows one to easily discard failed runs or to pick model checkpoints from earlier iterations before failure, without requiring labeled data.
StyleWSGAN Model Setup and Training We adapt StyleGAN2-ADA Karras et al., 2020) to build StyLNWSGAN Model as well as a StyleInfoGAN. The generator architecture follows the same approach as a class-conditionss d-dimensional vector via linear layer and then concatenated with the original latent code, after each is normalized. This concatenated vector is then passed to the StyleGAN mapping network. We find the relationship between the number of layers of the StyleGAN mapping network and the size of the embedded sampled code d to be crucial for SoyleWSGAN, When the mapping network is too shallow, as in the tuned CIFAR 10 settings in (Karras et al. 2020), a large d can lead to training instability for StyleWSGAN and StyleInfoGAN.
StyleGAN generator: the sampled code is embedded to
We use separate optimizer settings for each loss term, and set the learning rate for the Info term (added term of Equation 2 and the WSGAN term (added term of Equation plus decay penalty) to a factor of 2/10 of the base learning rate in StyleGAN. This results in a learning rate of 0.0005 for the added WSGAN terms in our experiments, while maintain learning rate of 0.0025 for the original StyeIGAN terms. Due to the use of different learning rates in the separate optimizers, the added loss terms are not scaled, and the hyper-parameters a, 3 are set to
For the CIFARIO 10 experiments we largely follow the settings used in (Karras et al. 2020): no style mixing, no path length regularization, no ResNet D. However. we increase the depth of the mapping network from 2 to 6, we decrease the size of the code embedding to 200, and continue training until the discriminator has seen total of 50M real images. mapping network of depth 4, and code embedding size of 50 also lead to good performance, performing only slightly worse measured by both FID and label model accuracy.
For the LSUN experiments, we train StyleWSGAN until the discriminator has seen total of 35M real images, and the baseline StyleGAN2-AII that we compare to trained until the discriminator has seen a total of 50M real images. We largely follow the settings used for 256 x 256 images in (Karras et al. 2020). but disable style mixing and path length regularization. We set the size of the discrete code embedding to 50.
End Model Training For all datasets, we train a ResNet-18 (He et al. 2016) for 100 epochs, using Adam and a learning rate scheduler. The learning rate scheduler uses a small validation set to make adjustments to the learning rate.
Image Augmentation We use the following random image augmentation functions during DC- GAN and endmodel training for color images: random crop and resize (cropping out maximum height/width of 13%), random sharpness adjustment (p 0.2), random Gaussian blur (p- = 0.1), and random color jitter.
Label Models To compare to related work, we use implementations of label models made available via WRENCH (Zhang et al. 202T).
Complexity WSGAN shares the same operations as InfoGAN and adds some additional steps on real samples that have at least one LF vote, which slightly increases the required computation. Recall that C' denotes the number of classes, m the number of LFs, and x an image of a real sample. Further, let nhe denote the number of samples that have at least one weak label vote from any LF,, let
16

---

Published as a conference paper at ICLR 2023
Layer (type:depth-idx)
Output Shape
Paran
InfoltDiscriminator
-Sequential: 1-1
[16, 512, 4, 41
Conv2d: 2-1
[16, 64, 32, 32]
1,792
LakkyReUU: 2-2
[16, 64, 32, 32]
Conv2d: 2-3
[16, 64, 16, 16]
68,
eakyReLU: 2-4
[16, 64, 16, 16]
Conv2d: 2-5
[16, 128, 16, 16]
73,856
eakyReLU; 2-6
[16, 128, 16, 16]
Conv2d: 2-7
[16, 128, 8, 8]
262, 272
eakyReLU; 2-8
[16, 128, 8, 8]
Conv2d: 2-9
[16, 256, 8, 8]
295, 168
eakyReLU; 2-10
[16, 256, 8, 8]
Conv2d: 2-11
[16, 256, 4, 4]
1,048,832
eakyReLU: 2-12
[16, 256, 4, 4]
Conv2d: 2-13
[16, 512, 4,
1,18
LeakyReLU: 2-14
[16, 512, 4,
Dropout: 2-15
[16, 512, 4, 4]
Sequential: 1-2
[16, 1]
Conv2d: 2-16
[16, 1,
8,192
Figure 4: The DCGAN discriminator architecture used in our experiments with 32 32 images.
Layer (type:depth-ide)
Output Shape
Paran
DCGeneratorThree
Linner:: 1-1
Sequential:
[16, 32, 32]
ConvTranspose2:: 2-1
[16, 512, 4, 4]
981, 632
BatchNorm2d: 2-2
[16, 512, 4,
1,824
ReLU: 2-3
[16, 512, 4, 4]
ConvTranspose2d: 2-4
[16, 256, 8, 8]
2, 2,897,4488
BatchNorm2d:
[16, 256, 8,
ReLU: 2-6
[16, 256, 8, 8]
ConvTranspose2d: 2-7
[16, 128,
526,416
BatchNorm2d: 2-8
[16, 128, 16, 16]
256
ReLU: 2-9
[16, 128, 16, 16]
ConvTranspose2d: 2-10
[16,
136,136
BatchNorm2d: 2-11
[16, 64, 32, 32]
128
ReLU: 2-12
[16,
ConvTranspose2d: 2-13
[16, 3, 32, 321
1,731
Tanh: 2-14
[16, 3, 32, 32]
Figure 5: The DCGAN generator architecture used in our experiments with 32x32 images.
obtain image features and the discrete code prediction, and a denote the number of steps for a forward pass through the MLP A. For forward pass, WSGAN increases the complexity compared to InfoGAN in each epoch by (n,, (a + 9 m + 20 C'm + 8))). Note that may be eliminated for the forward pass through careful implementation as the image features are already obtained for the basic InfoGAN update. In practice, in our experiments the computational overhead, including for additional data loading of the LFs, leads to modest increase runtime (measured bps, denoting batches per second) of the weakly supervised WSGAN over the unsupervised InfoGAN, follows. InfoGAN: 14bps, WSGAN Encoder: 7.8bps, WSGAN Vector: 8.6bps (NVIDIA RTX A6000, batch size 16). In terms of parameters, WSGAN shares the same generator G and discriminator components D.Q as InfoGAN, and adds additional label model parameters. The overall number of parameters in our experiments with 32 32 images InfoGAN 6.7M, WSGAN 8.8M.
g denote the number of steps required for forward pass through
17

---

Published as a conference paper at ICLR 2023
outomabb to 10:28)
hamae 23.331
merso 10,55
aupptae
automedde
10.301
airitana
thd 20.00
home 201
Figure 6: Some synthetic images and pseudolabels generated by the proposed WSGAN with DCGAN base-architecture, learned from weakly supervised CI We note WSGAN able to generate images and estimate their lahels, even for images where no weak supervision sources provide information (see end of Section 3.1 for details).
B DATASET DETAILS
CIFARIO contains 32x32 color images of 10 different classes. We create different subsets of CIFARIO. One set (used for experiments CIFARIO-C.D) uses the full training set of CIFARIO (minus 300 samples held out for downstream validation), while the second (used for experiments CIFARIO-A,B,E.7 is random subset of 30,000 training images.
MNIST and FashionMNIST both contain 28x28 grayscale images, which we resize to 32x32. For both, use a random sample of 30,000 images from the training data for our experiments. SSL-based labeling functions are fine-tuned on small, random subsets of the remaining training data of each dataset.
GTSRB contains 64x64 color images of German traffic signs. We use 22,640 random images from the full training dataset during our experiments, while random subsets of the remaining images in the original training data are used finetune the SSL-based labeling functions.
The original DomainNer (Peng al. 2019) dataset contains 345 classes of images in 6 different domains [1] As our dataset, following Mazzetto et al. (2021a) we use the images in the real domain and select the 10 classes with the largest number of instances in this domain. Because of the small size of the resulting dataset, we resize the images to 32 x 32 in our experiments.
Animals with Attributes 2 (AwA2) Mazzetto et al. 202 la is image dataset with known general attributes for each class, divided into 40 seen and 10 unseen classes. Because of the small size of the resulting dataset once LFs are created, resize the images to 32 32 in our experiments.
LSUN scene categories see Section
for details.
B.1
LABELING FUNCTION DETAILS
Synthetic: based the true class label, we create synthetic, unipolar LFs via the following procedure: for each LF, we sample class label, error rate, and a propensity (i.e., the percentage of samples where the LF casts a vote, also referred as coverage). Given the target label, we then sample true positives and false positives at random to achieve the desired LF accuracy and propensity.
Domain transfer: these LFs are used in our DomainNet dataset experiments. We fol- low Mazzetto et al. (2021a). and derive weak supervision sources for a multiclass classifica- real images contained in the DomainNet (Peng et al. 2019) dataset. First, we set our target domain to real images and select the 10 classes with the largest number of instances in this domain. As LFs, we then train classifiers using the selected classes within the remaining five domains, and apply these trained classifiers the unseen images in the target domain of real images to obtain weak labels.
tion task
Auribute heuristics: we create two sets of LFs for the Animals with Attributes (AwA2) (Mazzetto et al. a image classification dataset. Following Mazzetto al.
Real, painting, sketch, clipart, infograph, quickdraw.
18

---

Published as a conference paper at ICLR 2023
CIFARIO-C
CIFARIO-E
0.4
0.4
ARI
0.2
0.2
0.0
0.0
0
50
100
150
200
50
100
150
200
COFARIO-D
CIFARIO-F
0.4
0.4
ARI
InfoGAN
0.2
0.2
WSGAN
0.0
0.0
0
100
150
200
50
100
150
200
Epoch
Epoch
Figure 7: We here show the Adjusted Rand Index (ARI) for the additional CIFAR experiments. The plots show the ARI between the unobserved class label y and the discrete code prediction by the auxiliary model (xx) on real image z, during training. Weak supervision allows WSGAN to better uncover the latent class structure compared to an unsupervised InfoGAN.
(2021b)a). we train one-vs-rest attribute classifiers using the 40 seen classes of the Aw dataset. These classifiers are applied to the 10 unseen classes to produce weak attribute labels. At this stage, we discard attribute classifiers which perform worse than random. We create an 85% train/ cain//alidationntess split of the 10 unseen classes which we use to define decision trees to produce weak labels on the bases of weak attribute predictions. We create the 29 unipolar LFs for AwA2-A by training 3 one-vs-rest decision trees per each the 10 classes on 100 random samples from the training set. To create a slightly easier set, we create the 32 unipolar LFs used in AwA2-B by training 80 decision trees, retaining one random tree specializing each class, and then selecting all remaining ones where validation accuracy is higher than 0.65.
SSL-based: The base representations are learned on unsupervised ImageNET with Sim- CLR (Chen et al. 2020). The trained network used to obtain features for our image datasets. We then train shallow MLP networks on a few hundred held-out samples to predict a randomly sampled target label at a randomly sampled target accuracy. The accuracy is validated to be within range of the target accuracy another small amount of held-out data. Thus, during their creation these unipolar LFs are never trained or evaluated on the WSGAN training data the downstream test data.
C ADDITIONAL EXPERIMENTS
C.1 WSGAN WITH DCGAN BASE-ARCHITECTURE
Table 5: Additional datasets and labeling function (LF) characteristics used to evaluate the proposed
WSGAN model. Acc denotes accuracy, while Coverage denotes the number of samples where the LF does not abstain.
Dataset
#Classes
#LFs
#Samples
Mean LF Acc
Min LF Acc
Max LF Ace
Meun Coverage
LF Type
CIFARIO-C
10
20
49,700
0.747
0.621
0.879
0.048
Synthetic
CIFARIO-D
10
40
49,700
0.760
0.621
0.898
0.052
Synthetic
CIFARIIEE
Synthetic
CIFARIO-F
10
40
30,000
0.728
0.531
0.912
0.046
SSL, finetuning
For further evaluation of WSGAN with a DCGAN base architecture, we created additional weakly supervised image datasets based on CIFAR10 by varying the number of samples and the type
19

---

Published as a conference paper at ICLR 2023
Table 6: Additional datasets to evaluate WSGAN with DCGAN base architecture. This table shows average posterior accuracy of various label models on training samples with at least one LF vote. We highlight the best result in blue and the second best result in bold.
Dataset
MV
DawidSkene
MeTal
FS
Snorkel
WSGAN-
WSGAN-Encoder
CIFARIO-C
0.762
0.778
0.751
0.764
0.757
0.778
0.796
CIFARIO-D
0.831
0.861
0.819
0.805
0.812
0.854
0.865
CIFARIO-E
0.865
0.902
0.845
0.827
0.849
0.898
0.917
CIFARIO-F
0.687
0.601
0.682
0.677
0.678
0.691
0.702
Table Additional datasets: color image generation quality measured by average Fréchet Inception Distance (FID). The best scores for each dataset are highlighted in blue.
Dataset
InfoGAN
WSGAN-V
WSGAN-E
CIFARIO-C
33.64
24.11
26.00
CIFAR10-D
33.64
24.09
23.78
CIFARIO-E
28.93
21.97
22.63
CIFARIO-F 33.50
24.59
22.54
of labeling function, see CIFARIO dataset details in Table 5 The proposed WSGAN approach outperforms related approaches in these experiments well. The label model accuracy results shown in Table 6 while additional metrics including F1 are shown in Section E Image generation quality results are provided in Table 7 Finally, a comparison between the latent discrete variable of WSGAN and InfoGAN is given in Figure 7 which shows how the Adjusted Rand Index evolves between the unobserved class labels and the latent discrete variable modeled by auxiliary model Q
C.2 STYLEWSGAN
Please see Section A for implementation details and hyperparameter settings our StyleGAN experiments. Dataset statisties for this section are shown in Table g
LSUN scene categories To test the proposed WSGAN on higher resolution images with Style- GAN base architecture, we create a balanced subset of the LSUN scene categories dataset (Yu et al.) The dataset contains 10 classes (i.e. 10 different scene categories) and we center-crop and resize images to 256 by 256 pixels. We sample an equal number of images from each of the 10 classes for final dataset size of 1,212,270 images. As weak supervision sources, we create 30 SSL--based LFs by training classifiers on small amounts of held-out data using image features learned via self-supervised learning, as described in Section B.T
2015).
StyleWSGAN achieves an average FID of 7.54 on this dataset. An unconditional StyleGAN2-ADA achieves an FID of 10.3 with the settings for 256 by 256 images set in (Karras et al.) 2020), and an FID of 8.41 when we turn off path length regularization and style mixing. Note that unconditional StyleGAN results LSUN images with lower FID scores reported in related work generally obtained by training on a single LSUN scene or object category, rather than on multiple categories simultaneously, in our experiments, which results in more challenging setup.
The average WSGAN labelmodel accuracy on this weakly supervised LSUN dataset is 0.766. Other label models obtain the following average accuracies: DawidSkene 0.765, Majority Vote 0.76. FlyingSquid 0.739, Snorkel MeTaL 0.740, and Snorkel 0.728.
CIFARIO First, Figure g shows synthetic images by StyleWSGAN on the weakly supervised CIFARI0-B dataset, which uses SSL-based LFs. These LFs are quite noisy, with mean LF accuracy of 0.736, which reflected in the noisy classcooniitinall samples that can be inspected in Figure 9 On this dataset, StyleWSGAN achieves mean FID of 3.79, while also attaining high label model accuracy of 0.736. The unsupervised StyleGAN2-ADD with the optimal, tuned settings identified in (Karras al. 2020) achieves average FID of 3.85 on this subset.
20

---

Published as a conference paper at ICLR 2023
Figure 8: Synthetic images learned by StyleWSGAN on a weakly supervised subset of the LSUN scene category dataset.
We create an additional weakly supervised CIFARIO with lower noise LFs, to see such setting can lead to results that better than the state-of-the-an art (SOTA) unsupervised image generation quality on the full CIFAR10 10 dataset reported in (Karras et al.) 2020). For this experiment, we create LFs by randomly introducing errors and abstains to the ground- -ruth vector. For the LFs, we minimum accuracy of 0.8 and a maximum accuracy of 0.95 and create 20 LFs. This dataset contains 48000 samples, has mean LF accuracy 0.888, and a mean coverage of 0.102 (meaning that an LF on average abstains on ~ 89.8% of the dataset). For this dataset, StyleW achieves an FID of 2.84, which is better than the SOTA unsupervised result reported in (Karras al. 2019) of 2.92 FID on the full 50k CIFAR 10 samples, but shy of the performance of the conditional StyleGAN (Karras et al. 2019) which uses projection discrimination and has access to all ground-truth labels and achieves and FID of 2.42.
Table Additional datasets used evaluate StyleWSGAN. Ace denotes accuracy, while Coverage denotes the number of samples where an LF does not abstain.
Dataset
#Classes
#L.s
#Samples
Mean LF Ace
Min LF Ace
Max LF Acc
Mean Coverage
LF Type
CIFARIO- low noise LFs
10
20
48,000
0.888 0.736
0.816 0.624
0.949 0.873
0.102 0.098
Synthetic SSt-based
SUNN categories
10
30
1,212,270
21

---

Published as a conference paper at ICLR 2023
Figure 9: Synthetic images learned on the CIFARIO-B subset by StyleWSGAN, which a version of our WSGAN that built on StyleGAN2-ADA rather than simple DCGAN as in our main experiments.
D ADDITIONAL BASELINES
We compare against two additional baselines. First, we train a generative model that is conditioned on pseudolabel information with the aim of improving image generation performance; we use pseudolabels provided by established weak-supervision label models this role. Second, we use basic generative model to produce synthetic samples that augment downstream classifier (with weak labels provided by outputs weak supervision sources applied to the synthetic images). These two baselines represent the straightforward way to use weak supervision to improve generative modeling (and vice-versa). We observe that such naive combinations struggle compared to our proposed approach, further motivating the importance of the interface in WSGAN.
22

---

Published as a conference paper at ICLR 2023
Table 9: A comparison using an ACGAN with pseudolabels. Image generation quality measured by average Fréchet Inception Distance (FID). The best scores for each dataset are highlighted in blue.
Dataset
InfoGAN
ACGAN
ACGAN (crisp)
WSGAN-V
WSGAN-E
AwA2 A
41.62
51.32
47.21
36.74
34.71
AwA2-
41.62
53.73
50.03
36.79
34.52
DomainNet
51.88
61.96
47.32
51.16
45.6
CIFARIO-A
28.93
79.15
25.53
25.7
22.71
CIFARIO-C 33.64
36.53
26.61
24.11
26.0
CIFARIO-D 33.64
36.81
45.1
24.09
23.78
CIFARIO-E 28.93
80.43
33.05
21.97
22.63
Table 10: Baseline comparisons using an InfoGAN to create synthetic images, applying LFs to the synthetic images, and then using established label models synthesize the weak labels in to a pseudolabel resulting in weakly labeled fake images. The table shows the change in test accuracy by augmenting the downstream classifier training data with such 1,000 synthetic images and corresponding pseudo labels. Experiments are conducted subset of the datasets where labeling functions can be applied to synthetic images.
Dataset
WSGAN
InfoGAN Snorkel
InfoGAN + DawidSkene
AwA2 A 0.79%
-0.63%
-1.26%
AwA2 B
3.90%
-1.01% %
-1.77%
DomainNet
1.50%
0.02%
-3.14%
D.1 CONDITIONAL IMAGE GENERATION WITH PSEUDOLABELS OR RAW WEAK SUPERVISION VOTES
As an additional GAN baseline to our proposed WSGAN, we slightly adapt an Auxiliary Classifier Generative Adversarial Network (ACGAN) (Odena et al. 2017) condition it pseudolabels. The ACGAN is run on all data, but the auxiliary loss on real data with pseudolabels is only used for samples where least one labeling function does not abstain. We create two versions: (I) using probabilistic pseudolabels with a soft cross-entropy loss, and (2) using hard/crisp labels with a cross-entropy loss. provide the strongest possible baseline in this experiment, we obtain the pseudolabels via the Dawid-Skene label model as it attains the best performance datasets compared to other related label models. Results are provided in Table 9 showing that this baseline approach is frequently unable to overcome the noise in the pseudolabels to improve over the InfoGAN results, and that does not perform better than WSGAN with an encoder. Furthermore, much more difficult train these models and they frequently failed to converge.
average over all
We also attempted to train different types of conditional GANs (ACGAN and a GAN with projection discrimination) conditioned on the raw weak supervision votes, but were unable to obtain reasonable performance as the models failed to converge.
D.2 DATA AUGMENTATION FOR DOWNSTREAM CLASSIFICATION WITH SYNTHETIC IMAGES
In this experiment, we augment the training for downstream classifier with synthetic images. As baselines, we generate synthetic images F with an InfoGAN, and then apply the image labeling functions to the generated images to obtain LF votes >(ā). Pseudolabels for the synthetic images then obtained by fitting label models to the real training data and then applying the label models to the labeling function outputs on the synthetic data. Table compares InfoGAN + Snorkel and InfoGAN DawidSkene baselines to the improvements in test accuracy obtained by using WSGAN and shows that we were unable to obtain improvements in downstream accuracy with the baselines, possibly due to performance of the InfoGAN on these small datasets.
23

---

Published as a conference paper at ICLR 2023
Table 11: In this table, we include standard deviations for the posterior accuracy of various label
models on training samples with at least one LF vote, computed over five random runs. Due to limited computational budget, we were unable to accumulate five runs for all datasets and model combinations,
Dataset
DawidSkene
MeTaL
FS
Snorkel
AwA2
0.607(:0.029)
0.632(=0.002)
0.615(#0.003)
0.641(10.001)
0.681(#0.011)
DomainNet
0.658(10.000)
0.635(=0.000) 0.487(+0.004)
0.499(10.015)
0.643 0033)
MNIST
0.729(+0.000)
0.766(+0.001)
0.773(+0.000)
0.766(+0.001)
0.813(+0.004
FNShionMNIST
0.717(=0.002)
0.730(+0.001)
0.734(=0.001)
0.729(+0.001)
0.744(#0.002
GTSRB
0.619(=0.001)
0.815(#0.002)
0.679(#0.001)
0.814(#0.000)
0.823
CIFARIO-A
0.806(10.001) 0.850(10.001)
0.800(10.000)
0.807(10.002)
0.874(10.002
CIFARIO-B
0.677(10.000)
0.708(10.001)
0.708(10.000)
0.707(10.000)
0.731(10.004)
Table 12: Weighted mean average precision of various label models on training samples with at least L. vote. We highlight the best result in blue and the second best result in bold.
Datasct
MV
DawidSkene
McTaL
FS
Snorkel
WSGAN-
WYSGAN-Encodee
AwA2 A
0.616
0.661
0.653
0.627
0.653
0.672
0.737
AwA2 -
0.591
0.652
0.662
0.642
0.668
0.681
0.743
DomainNet
0.599
0.702
0,630
0.654
0.621
0.679
0.795
MNIST
0.684
0.772
0.784
0.765
0.785
0.792
0,870
MNSST
0.620
0.712
0.691
0.686
0.692
0.703
0.742
GTSRB
0.718
0.731
0.761
0.714
0.772
0.768
0,808
CIFARIO-A
0.796
0,866
0.855
0.838
0.854
0.878
0,912
CIFAR 10-B
0.594
0.659
0.658
0.631
0.666
0.678
0.732
CIFARIO-C
0.664
0.763
0.758
0.737
0.751
0.780
0.825
CIFARIO-D
0.788
0.896
0.889
0.876
0.878
0.901
0.908
CIFARIO-E
0.880
0.954
0,942
0.924
0.940
0.950
0.959
CIFARIO-F
0,561
0.658
0.66
0.647
0.659
0.675
0.708
E ADDITIONAL METRICS
We provide additional metrics for the label model comparisons shown in Table 2 in the main paper. Again, results are averaged over random runs. Table 13 shows the weighted score, an average all classes weighted by the support of each class. Table 12 shows weighted mean average precision, a metric that summarizes the precision-recall curve across all classes. We compute the average precision individually for each class (one vs. rest) and then aggregate the scores by summing them weighted by the support of each class to produce the weighted mean average precision score.
Table 13: Weighted Fl score of various label models on training samples with at least one LF vote. The F1 is computed separately for each class and then averaged weighted by the support of each class. We highlight the best result in blue and the second best result in bold.
Dataset
MV
DawidSkene
MeTaL
FS
Snorkel
WSGANN-Vector
WSGAN-Eneoder
AwA2 A
0.641
0.665
0.62
0.619
0.636
0.637
0.684
AwA2 B
0.604
0.661
0.58
0.597
0.593
0.664
0.672
DomainNet
0.603
0.655
0.443
0.622
0.468
0.654
0.634
MNIST
0.756
0.716
0.746
0.755
0.746
0.764
0.795
FasHionMNISTT
0.706
0.691
0.698
0.705
0.698
0.710
0.715
GTSRB
0.802
0.616
0.800
0.628
0.799
0.801
0,811
CIFAR 10-A
0.824
0.850
0.797
0.796
0.798
0.851
0.872
CIFAR 10-B
0.712
0.672
0.702
0,703
0.702
0.720
0.727
CIFARI0-C
0.726
0.741
0.723
0.713
0.718
0.738
0.759
CIFAR I0-D
0.809
0.839
0.791
0.784
0.781
0.834
0.844
CIFARIO-E
0.864
0,901
0.843
0.825
0.839
0.899
0.916
CIFARTO-F
0.684
0.609
0.677
0.675
0.674
0.688
0.699
24

---

Published as a conference paper at ICLR 2023
F THEORETICAL JUSTIFICATION
We provide additional setup and proofs for our two theoretical claims.
F.1 CLAIM (1)
Our goal is to derive a generalization bound; that is, an upper bound on [Rp Rp In words, this is the gap between the loss on sample drawn from the true distribution and the empirical loss we obtained by training on the supervised dataset with unlabeled data sampled from the generative model.
Mixture of Gaussians Recall that D is the joint distribution of the unlabeled and labeled points. Let's call the unlabeled data marginal distribution Dx. Then, we make the assumption that Dx is mixture of & Gaussians. Here, there is some relationship between the mixtures and the two classes, but we need not further specify it. Using the result (Ashtiani et al. 2018), we get that the number of samples needed to learn Dx up to e in total variation distance is (kkd/
Note that in fact this expression hides some polylogarithmic terms. However, for simplicity, we'r 'r going to ignore these terms and just pretend that the necessary bound cgkdd /E", where eg is some constant for learning density.
Based on this, we'l 'll make the following assumption. We perform density estimation on nı samples from Dx and obtain some model g such that distribution of (we'll 'll abuse notation and just refer to this as the model itself g) and Dx satisfies
cok n1
div sdv
(5)
So now we have control over one marginal (the unlabeled data). Let's work on the conditional term next.
Majority Vote For simplicity, let's assume that majority vote the aggregation scheme for the m labeling functions. We make the following assumptions. The labeling functions have accuracy 1/2 + a, for some E (0, 1/2), in the following sense. For any datapoint (X,Y). ), the probability of a labeling function guessing the value of Y correctly is 1/2 + a, and the probability of any guessing wrong 1/2 a. This holds for all values of X. Note: these are very strong assumptions.
The probability that we make mistake, e.g., that majority vote aggregates votes to O when Y 1 or vice-versa given by the binomial CDF F(m/2, m, a 1/2), which has the following simple bound that follows from Hoeffding's 's inequality,
m//2 -2m a + 1/2
)) = =exp( -mma2.
F(m/2, m. + 1/2) & exp
m
With the above, Dyix is Bernoulli random variable, directly upper bound the total variation distance between Dy x and Dyixi
drv Drix) S exp(-2ma²).
(6)
Joint Distribution Now we have some control over the generative model's 's error (from the density estimation bound) and some control over the label recovery (from the above bound resulting from majority vote). Now we put it together. First, we write down some useful inequalities between the total variation distance and the Hellinger distance (Duchi 2016) (Prop 2.10). These are, for densities
Dha q) S ////222mp 2dTv(p..))
and
drv(p. Dhol(p. Dree (p. 2/2/4.
(8)
25

---

Published as a conference paper at ICLR 2023
We use p the density for D and q the density for D, and write p1(2)p2(blz). qi (z)gg2)))z) First, using Eq. 5 and Eq. we have that
(
Deel (Dx, 9) I (2d
(9)
n1
Then, using Eq. 66 and Eq.
we get
Dhe D D(Ix) &
(drx(Dr Dy(x) & /2exp( 2ma²) = V2exp(-maa (10)
Next,
Dia(D, D) - / (V (2) dyddz
/ ( -atp:=/wahhaoooaa///////a//aaaa- dydr
2 2-2 2 / ( (z)
/ Vp((z)g((t)
2 2
Daall Dra (22)
dx
2 /
(vi(s)(((t))
v2
2 exp(-ma"
Note that here, use the fact that our bound holds for all conditional distributions regardless of Continuing,
(Da(D,D) / (D((T))((T)
v2 exp(- -maa)
1-
V2
=2 201 Dnei(Pi,9h)
exp( (m-mmam)
2 - 2 ( 1- ( (Acgkd2
$
22
1
exp( (op(-maa)
m
2
(cckkd22
((ccket)
+ 2
exp(- exp(-maa).
n1
171
Now we apply Eq. (8) to get the bound back into the total variation distance setting. We have
(D D) s (na DI/1 Dra (D, (D,
4oc
-(ooket)
(Gkkd)
+ +2eexp(mmal
exp(-ma?)
nh
11
4ookdd
<2eep((mma)
(11)
11
Bounding the Risk The final task is to bound the risk. First, suppose we are training classifier chosen from a function class F, trained on n2 independently-drann data points. Then, a standard result is that with probability at least 1- 6,
log(1/8)
sup [ - RD( < 2 FEF
(12)
2n12
Here, R is the Rademacher complexity of the function class. However, the above is for training on samples from the true distribution. Instead, we can write
R Rp = _ - P. + R. - RRp
_ Rool+ + R. R.pl
26

---

Published as a conference paper at ICLR 2023
For the right-hand term, we have the following:
/ ((((z)) q( y)
D - RR = =
- (edTv((,
Then, putting this together with the expression Eq. into Eq. [12). we get that, with probability at least 1 6,
sup [ - :RD(11 - (sup SEF
IR - R + [R Rp
FEF
log(1/6)
2 22R +
+ BedTv(D,
2n12
log(1/0)
(Aeckd n1
$ 22R +
+ Be
+ BB BeV2exp(-ma's
(13)
2n2
Interpreting the Bound
In Eq.
13
saw that
log(1/0)
( + Br
3/
SEF [f) RD(f)
sup
29 +
+ +BB2eexp(-ma))
2n12
Y1
Now let's interpret this result piece-by-piece. The terms are the following
The Rademacher complexity of the function class, which is present in the standard general-
ization bound.
An estimation error term as a function of how much data we have to train our classifier n2-- It has the standard rate 1 /n2.. Again, this is standard in any bound.
A penalty term due the generative model usage. It tells how much we lose by training on generated data rather than (unlabeled) data from the true distribution. scales as 1/4 where nı is the number of samples of unlabeled data used train the generative model. Note also the dependence on the number of mixture components and dimension.
A penalty term due to weak supervision. (pseudo) labels rather than true labels; note that the penalty scales exponentially in the number of labeling functions m, but is slowed down by small a, as our accuracies are a better than random.
tells us what we lose by using estimated
F.2
CLAIM (2)
Our proof of claim (2) uses the setting of Thekupparampi et al. (2018), which introduces RCGAN. RCGAN is conditional GAN architecture that corrupts the label before passing them to the discriminator by passing the true labels through a noisy channel. The authors provide multiplicative approximation bound between the GAN loss under the unobserved true labels and the loss under noisy labels. This noisy channel model acts nice model of the label generating process of weak supervision. Using this noisy channel model, we can control the amount of label corruption to match that of weak supervision.
Following the setup of TThekumparmppi et al. 2018), we define a function that multiplies a one-hot encoded true label vector by right- stochasti matrix C' 2 where Ci,- P(ū (vx) this is our noisy channel. This induces a joint distribution Px,y for the examples æ and noisy labels ÿ from the conditional distribution defined by C. We restate the theorems of interest from TTekumparampii et al.) (2018) here, and proceed to adapt them to our problem setting.
Theorem 1. (Multiplicatiiee bound on the total variation distance from Thekumparampil et al. 2018 Let Рx, and Qx be two distributions over X {0,1) }} and let Рx, and Q Qx be the corresponding distributions with noisy labels from C. CfC is full-rank, then
drv P) S d < || drv (P, Q)
(14)
27

---

Published as a conference paper at ICLR 2023
Theorem [ says that the total variation distance between the true noisy distribution and the noisy generated distribution from RCGAN approximate its noiseless counterpart up to a factor of ||C='Ile<- Our goal is to construct Cawy model the noise from weak supervision (in particular, majority vote) and show that it leads to a tighter bound than when we directly plug in the labels from single L.F To begin, consider the following parameterization of C, with (0,
into Theorem
1 - - -
- C. - c. - -
(15)
1-c
Here, denotes the labeling error for each class. Given this parameterization, obtain the following expression for ||C= l
| 10000
e
E
(16)
1 -
0cc
[[(1 - E)3 -
2) 2
1-c -c
(17)
-c 1 1-c
= (1 11-26 2c)
(18)
Note that C, is full-rank as it has a finite inverse. It is also clear that || is a monotonically increasing function of That is to say that if we do something to decrease the labeling error c, then ||C,"Il || also decreases and we obtain a tighter bound. We will go on to derive an expression for the labeling error under majority vote with m LFs, eMV, and show that it is smaller than the labeling error from a single LF,, ex- Namely, we want find a condition where MV S ex holds and that majority vote leads an improved Theorem hound.
Proposition 1. (Total variation version.) Let eMY be the labeling error from majority vote from m LFs, where log(1/cx), whose individual labeling errors each cy. Then the following holds 2()-ex)²
drv(PMv, @Mv) S QTv(P.Q) S l1C drv(PMv. Qmv) S I1CT Iloo drv(PMrv QMv).
Proof. We begin by deriving an upper bound on eNv. We have LFs { that each produce incorrect predictions with probability ca 2 a, using a defined in Claim (1). Now, need to show that the probability of producing incorrect predictions using majority vote with more label functions, 1m has error eNV < ex. Define the event that A; is incorrect follows:
= I[Ax * y), then E |I| 2/] Ex. Using this, we apply Hoeffding's 's bound to exV.
m
CMV P
2
meX
(19)
2 mcx)
I exp
(20)
m
2)
exp
-2m
2 - EA
(21)
Next, we plug the bound from 21 into 18 obtain the following expression for ||C=_ Il
||- lloo = (1 -22Mv)
(22)
(22m (=-aa) 2) 19)²
< 2 2xxp
-Ea
(23)
To complete the proof, we need the following to hold: ||C==] 1l,0 < || 1loo. but due to the monotonicity of ||CC==llo lloo. it is sufficient to show that exv S EX.
Recall that <x < exp (-2m (1 - =xx²²) so if we set exp (-2m (1 -ex)²) S Ex we obtain the
minimum number of label functions, m, required to ensure <MV S Ex.
28

---

Published as a conference paper at ICLR 2023
2 Ex )² EA
1
exp
-2m
log( /x2)
7 m
2 2 (2)
Plugging 23 into Theorem
1
we obtain the following
drv frv, s (dv((,, Q) < |o drv (Pvv, Qmv)
t> )))
< 2exp
-2m
drv Qmv)
< (1 22x) - drv( (Pav: QMv)
1 (Pv(PMv,
which completes the proof.
Notice that the proof of Proposition does not depend on total variation distance beyond the dependence on Theorem [ As such, Proposition can be stated more generally in terms of the Integral Probability Metric induced by the GAN discriminator F using Theorem 2 of Thhhumparampil al. (2018):
dy @Mv) S dr(P,Q) & |C dP @Mv) S 1oo dy((Pvv Qmv).
Finally, notice that Proposition is made in terms of d (PMv, QMv) and not in terms of d (Px, Qx). We can show that as the number of LFs approach infinity, we recover the distance under the clean labels: dF(P, Q). Applying Theorem]] to majority vote and a single LF results the following two expressions:
dy (PMv, (PM.) < d)((.Q) < | dy (Pav, @uvv
(24)
dF((.)) s s 10 d= Q2x)
(25)
Rearranging terms, we obtain the following
15 dr(P,Q)
11 100 I || 1000
dy (PMv. QMv)
and
15 (x(P.Q) $ || 1bx 1b0.
dy Qx)
Notice that || boo has no dependence since it's single LF, but ||C l0 approaches 1 as m -> 00:
lim 15 dz(P,QQ
S 11 S || Iloc
m0000
d=(Pvv, QMv)
()(P.Q)
$1 11
dy (PMV, Qmv)
= dy dy Q) = dy (Pav. Qmv).
Hence obtain stronger bound the number of LFs increases.
29

---

Published as a conference paper at ICLR 2023
F.3 EXTENSIONS
For the sake of clarity, we make several simplifying assumptions in Claims (1) and (2). Both claims the simplest possible aggregation strategy for weak supervision-mmajorii vote, and analysis in Claim (1) involves the use of a Gaussian mixture model-a less complex object of study compared GAN, We can directly extend both analyses use more sophisticated weak supervision label models instead of majority vote, and different generative models, which should lead to improved bounds at the expense of more complex claim statement. We note, additionally, that neither of the claims attempt to provide deep insight into the benefits of jointly learning the generative model and label model-buut this can done with a slightly more careful analysis.
30

---

Published as a conference paper at ICLR 2023
Speod Bisk 50 10.42)
Go smighe (9.0))
Baand shead 69.20
Go Gossniglt (D67)
No owarkking oi on
50
Custrnction
Tmii
1006
Gesigó
70
Figure 10: Images pseudolabels generated by the proposed WSGAN (with simple DCGAN architecture) on the weakly supervised GTSRB dataset. WSGAN can estimate labels even for images where no weak supervision sources provide information (see end Section 3.1..
4
4
4
4
4
4
4
4
4
4
9
9
9
9
9
9
9
9
9
9
3
3
3
3
3
3
3
3
3
3
0
0
0
0
0
0
0
Figure 11: random set of MNIST images generated by WSGAN-E (using a DCGAN), with the discrete latent random variable kept fix for each row of images.
G ADDITIONAL IMAGES
Here we provide additional generated images in Figures TO are generated by WSGAN with DCGAN base architecture, where the discrete latent variable passed to the generator is kept the same each row.
These random images
31

---

Published as a conference paper at ICLR 2023
Figure 12: random set FashionMNIST images generated by WSGAN-E (using DCGAN), with the discrete latent random variable kept fix for each row of images.
Figure 13: A random set of Domainnet images generated by WSGAN-E (using a DCGAN), with the discrete latent random variable kept fix for each row of images. Note that this dataset is particularly challenging for a GAN as our dataset has fewer than 7,000 images, resulting in considerably lower quality of synthetic images compared to GTSRB for example.
Figure 14:
random set of AwA2 images generated by WSGAN-E (using DCGAN), with the discrete latent random variable kept fix for each row of images. Note that this dataset is particularly challenging for GAN, as this weakly supervised dataset has fewer than 7,000 images, resulting in considerably lower quality of synthetic images compared to GTSRB for example.
32


Reviews:
[{"review_id": "fRHRMdVuoRt", "reviewer": "Reviewer_kuyc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper has tackled the problem of simultaneously dealing with generative modeling and weak supervision. The authors have claimed that the generative model and weak supervision mutually benefit each other. The proposed method (WSGAN) has improved weak supervision, generative modeling, and data augmentation. \nTechnical challenges arise from developing the interface between the label model (weak supervision) and the generative model and fusing them. The challenges are solved by the proposed WSGAN and theoretical justification is provided in addition to empirical evidence. ", "summary_of_the_review": "The paper is well-written and a significant problem is solved in a novel way. \nThe method is mathematically sound and comprehensive experiments are performed.  ", "strength_and_weaknesses": "The paper is well-written and straightforward to follow. Technical terminology is adequately provided and checking the mathematics for soundness is relatively easy. While the theoretical justification (bounding in the appendix) was not carefully checked, it looked very interesting and it is definitely helpful for interested readers. \n\nBoth weak supervision and generative model are significant to the community to reduce the huge cost of labeling data. Combining them to further improve the labeling process matters to both academic researchers and industrial practitioners. This interesting problem is well-studied in the paper and a powerful and effective solution (WSGAN) is provided and justified theoretically and empirically. \n\nOne weakness is that the generated images are mostly low-resolution. While StyleGAN architecture is tried, the results in Figure 8 are still too small. I am wondering if the authors have plans to generate more realistic images. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "cGDEY6rIK2x", "reviewer": "Reviewer_ERZw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a generative modeling setup for weak supervision in images. The claim is that by doing so, there are improvements both to generative modeling and to the pseudolabeling qualities of the model, and they work in concert. They do this through proofs and experimental evaluations. \n\nThe setup is a GAN architecture based on InfoGAN, with hooks added for weak supervision. The InfoGAN latent code c is to be used as pseudolabel, trained by 'aligning' with the label generated by the labeling model LM through a loss construction. During training, c is produced by the synthetic image produced by the generator. However, during inference, one passes an actual test image to the encoder Q. If the GAN is trained properly, it is assumed that Q will work with 'real' images as well. \n\nFor evaluations, the authors show improvements as measured in FID (for image quality improvements) and posterior accuracy, F1 score (for labeling model) over a number of datasets, tasks and competitive models. ", "summary_of_the_review": "The paper presents an approach to add weak supervision to GAN modeling, and shows that the addition is useful for both pseudo-labeling and generative modeling. The presentation is clear, and results seem convincing. ", "strength_and_weaknesses": "+ The use of generative modeling with a disentangling setup like InfoGAN is a creative way to generate pseudolabels. \n+ The evaluations show clear improvements over other approaches \n- Failure cases for GAN generations not discussed. \n- I also think that the applicability of the approach might be slightly limited in other areas such as speech, or NLP. \n- Some visuals of what the latent space is learning would be illuminating.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S3JKifAZJ9", "reviewer": "Reviewer_vpDG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To tackle the problem of lacking sufficient ground-truth labels in practical applications, this paper proposed a paradigm to fuse the generative model and programmatic weak supervision method, which are often used separately to tackle this issue in previous literatures. The fusion is mainly achieved by encouraging an agreement between the prediction label from the weak-supervision label model and the inferred discrete latent code discovered with the InfoGAN. The agreement essentially means the labels predicted by the weak-supervision label model and InfoGAN are the same up to some permutation.\n\nThe authors demonstrate that the proposed fusion and alignment are technically solid, by theoretically showing that: 1) the loss gap between the original dataset and augmented dataset with generated samples is controlled by penalty from labeling function accuracy, indicating positive benefits of weak supervision; 2) the RCGAN loss with noisy labels generated by multiple labeling functions has a tighter bound than that by a single labeling function. And these two theoretical results together make it possible for weak supervision and generative modeling to cooperate each other.\n\nExperiments are conducted to investigate the impacts on performance from four aspects: 1) label model accuracy; 2) classifier output and label model output alignment; 3) image generation quality; 4) augmented images by generator. Reported results show considerable performance boost for both InfoGAN and label model components as well as better alignment between their outputs than vanilla InfoGAN. Moreover, the proposed WSGAN enables pseudo-labeled samples generation with trained generator and classifier / label model, allowing for augmented training dataset without extra data collection or labeling to improve the downstream classifier performance.\n ", "summary_of_the_review": "The paper is generally clearly written and provides a novel perspective to incorporate weak supervision into generative modeling framework. Extensive experiments are conducted to demonstrate the effectiveness of the proposed fusion approach.", "strength_and_weaknesses": "Strengths\n1) The idea of fusing InfoGAN that discovers discrete latent factors and programmatic weak supervision based label model is interesting, and could also be very helpful for many practical tasks.\n2) The experimental results are comprehensive and convincing. A considerable number of experiments are conducted to demonstrate the model's effectiveness on both weak supervision labeling and generative modeling, in addition to the ability of generating samples with corresponding label estimates.\n3) Theoretical analyses are presented to support the claim that generative model and weak-supervision label model can help each other.\n\nWeaknesses\n1) Theoretical statement of Claim 2 is not precise enough lacking necessary explanation for related assumptions and analyses.\n2) The descriptions to Table 4 is confusing. It is stated that an improvement in test accuracy up to 3.9% (or a modest increase of up to 2.4% later) was achieved. But there are many different datasets, why you only focus on the performance changes on one dataset?\n3) Although the paper has demonstrated that the proposed method is the first to enable pseudo-labeled data augmentation, the idea of generating synthetic images with pseudo-labels is not novel in current generative models (e.g. GMM based deep generative models). And more experiments need to be conducted to demonstrate better qualities of generated pseudo-labeled samples compared to other generative methods.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster