Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper has tackled the problem of simultaneously dealing with generative modeling and weak supervision. The authors have claimed that the generative model and weak supervision mutually benefit each other. The proposed method (WSGAN) has improved weak supervision, generative modeling, and data augmentation. 
Technical challenges arise from developing the interface between the label model (weak supervision) and the generative model and fusing them. The challenges are solved by the proposed WSGAN and theoretical justification is provided in addition to empirical evidence. 
strength_and_weaknesses: The paper is well-written and straightforward to follow. Technical terminology is adequately provided and checking the mathematics for soundness is relatively easy. While the theoretical justification (bounding in the appendix) was not carefully checked, it looked very interesting and it is definitely helpful for interested readers. 

Both weak supervision and generative model are significant to the community to reduce the huge cost of labeling data. Combining them to further improve the labeling process matters to both academic researchers and industrial practitioners. This interesting problem is well-studied in the paper and a powerful and effective solution (WSGAN) is provided and justified theoretically and empirically. 

One weakness is that the generated images are mostly low-resolution. While StyleGAN architecture is tried, the results in Figure 8 are still too small. I am wondering if the authors have plans to generate more realistic images. 
clarity,_quality,_novelty_and_reproducibility: As mentioned above, the paper is clear and well-written.
The tackled problem is novel and interesting to the community. 
As code and model are not provided, reproducibility is in question and I am wondering if the authors have plans to release the code and the model.
summary_of_the_review: The paper is well-written and a significant problem is solved in a novel way. 
The method is mathematically sound and comprehensive experiments are performed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a generative modeling setup for weak supervision in images. The claim is that by doing so, there are improvements both to generative modeling and to the pseudolabeling qualities of the model, and they work in concert. They do this through proofs and experimental evaluations. 

The setup is a GAN architecture based on InfoGAN, with hooks added for weak supervision. The InfoGAN latent code c is to be used as pseudolabel, trained by 'aligning' with the label generated by the labeling model LM through a loss construction. During training, c is produced by the synthetic image produced by the generator. However, during inference, one passes an actual test image to the encoder Q. If the GAN is trained properly, it is assumed that Q will work with 'real' images as well. 

For evaluations, the authors show improvements as measured in FID (for image quality improvements) and posterior accuracy, F1 score (for labeling model) over a number of datasets, tasks and competitive models. 
strength_and_weaknesses: + The use of generative modeling with a disentangling setup like InfoGAN is a creative way to generate pseudolabels. 
+ The evaluations show clear improvements over other approaches 
- Failure cases for GAN generations not discussed. 
- I also think that the applicability of the approach might be slightly limited in other areas such as speech, or NLP. 
- Some visuals of what the latent space is learning would be illuminating.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper reads well enough, but it took me some time to figure out how the generative setup could be used for inference as well. Perhaps that ought to be emphasized for clarity

Quality and Novelty: I felt that the use of the disentangling GAN setup to be an interesting, novel approach and the results are convincing. However, it seems to me that some more insight into what the latent space learns (as we are seemingly producing a discrete label) through some visuals would be illuminating.

Reproducibility: I think the algorithmic components are properly laid out. However, I am not sure how easy it is to reproduce the experiments and whether the training dynamics are stable - especially, across the many datasets and experiments that the paper reports. Can the authors comment on this aspect?


summary_of_the_review: The paper presents an approach to add weak supervision to GAN modeling, and shows that the addition is useful for both pseudo-labeling and generative modeling. The presentation is clear, and results seem convincing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: To tackle the problem of lacking sufficient ground-truth labels in practical applications, this paper proposed a paradigm to fuse the generative model and programmatic weak supervision method, which are often used separately to tackle this issue in previous literatures. The fusion is mainly achieved by encouraging an agreement between the prediction label from the weak-supervision label model and the inferred discrete latent code discovered with the InfoGAN. The agreement essentially means the labels predicted by the weak-supervision label model and InfoGAN are the same up to some permutation.

The authors demonstrate that the proposed fusion and alignment are technically solid, by theoretically showing that: 1) the loss gap between the original dataset and augmented dataset with generated samples is controlled by penalty from labeling function accuracy, indicating positive benefits of weak supervision; 2) the RCGAN loss with noisy labels generated by multiple labeling functions has a tighter bound than that by a single labeling function. And these two theoretical results together make it possible for weak supervision and generative modeling to cooperate each other.

Experiments are conducted to investigate the impacts on performance from four aspects: 1) label model accuracy; 2) classifier output and label model output alignment; 3) image generation quality; 4) augmented images by generator. Reported results show considerable performance boost for both InfoGAN and label model components as well as better alignment between their outputs than vanilla InfoGAN. Moreover, the proposed WSGAN enables pseudo-labeled samples generation with trained generator and classifier / label model, allowing for augmented training dataset without extra data collection or labeling to improve the downstream classifier performance.
 
strength_and_weaknesses: Strengths
1) The idea of fusing InfoGAN that discovers discrete latent factors and programmatic weak supervision based label model is interesting, and could also be very helpful for many practical tasks.
2) The experimental results are comprehensive and convincing. A considerable number of experiments are conducted to demonstrate the model's effectiveness on both weak supervision labeling and generative modeling, in addition to the ability of generating samples with corresponding label estimates.
3) Theoretical analyses are presented to support the claim that generative model and weak-supervision label model can help each other.

Weaknesses
1) Theoretical statement of Claim 2 is not precise enough lacking necessary explanation for related assumptions and analyses.
2) The descriptions to Table 4 is confusing. It is stated that an improvement in test accuracy up to 3.9% (or a modest increase of up to 2.4% later) was achieved. But there are many different datasets, why you only focus on the performance changes on one dataset?
3) Although the paper has demonstrated that the proposed method is the first to enable pseudo-labeled data augmentation, the idea of generating synthetic images with pseudo-labels is not novel in current generative models (e.g. GMM based deep generative models). And more experiments need to be conducted to demonstrate better qualities of generated pseudo-labeled samples compared to other generative methods.

clarity,_quality,_novelty_and_reproducibility: 1) Clarity. The paper is well written and organized, provides a clear explanation of problem definition, proposed methods and experimental results. Theoretical analysis of the proposed method is also included.
2) Quality. The paper is technically sound, with well supported results of weak supervision based generative model learning.
3) Novelty. The paper provides a novel idea of introducing InfoGAN into label model design, which leverages superior generative power of GAN to model the label distribution.
4) Originality. The proposed fusion is interesting and novel.

summary_of_the_review: The paper is generally clearly written and provides a novel perspective to incorporate weak supervision into generative modeling framework. Extensive experiments are conducted to demonstrate the effectiveness of the proposed fusion approach.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses:  
The paper discusses the relationship generative modeling and weak supervision. The paper is generally well written and has clear technical and empirical support. The discussion phase was useful in helping improve the paper, although the reviewer scores remained the same. In general I think ICLR readers will find this an interesting and accessible piece of work.

justification_for_why_not_higher_score: This is a good paper, but there wasn't overwhelming enthusiasm for this amongst the reviewers. 
justification_for_why_not_lower_score: This is a solid piece of work.
Decision: Accept: poster