Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focus on a setting that multi-label with only one positive label and try to generate the multi-label data with generative adversarial network (GAN). The proposed method considers both unconditional and class conditional version by a pre-trained GAN in a sampling strategy. The experimental results shows that the proposed method gets a promising result under the single positive setting.
strengths_and_weaknesses: Strengths:

Overall, the problem of multi-label with single positive label is interesting.

The paper is well written and organized.

The idea of using single class densities to provide an estimate for the multi-class target distribution is interesting.

Weaknesses:

The datasets used in the experiments can be relatively small. It is better to adopt some large datasets to validate the effectiveness of the proposed methods.

Since the paper focuses on the multi-label with singe positive label setting, it is better to use some datasets with large label space to see the performance the proposed method.

I suggest the author to add an experiment about varying the label space of the data and show how the performance changes to further validate the multi-label data generation performance.

Overall, this paper is novel to some extent. It is better that more promising results from some large datasets.

questions: Please see my main review.
limitations: The work can only apply to the small datasets or small class space datasets and the experiments need to be further improved.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed a sampling method based on MCMC to sample multi-label data from GANs trained with single positive labels. The authors prove that the target distribution can be derived from the conditional densities of single positive labels. Then they use several classifiers to estimate the density ratio and Metropolis-Hastings, an MCMC-based sampling method to draw samples. The method is technically sound and achieves good restuls in empirical studies.
strengths_and_weaknesses: Strength
1. The problem is interesting and valuable for reducing annotation costs.
2. The sampling method is novel and can be applied to any existing GANs.
3. The formulations and derivations are clear.
4. The empirical studies, including experiments on 1-D Gaussian mixtures and real image datasets, well support the method's superiority with a clear margin.

Weakness
1. As discussed in Limitations, the method is hard to scale up for many attributes. The authors may consider moving some conclusions of complexity analysis from the appendix to the main paper.
2. The experiment setting of sampling for multiple classes could be clearer with some examples. For instance, in CIFAR-7to3, class A includes images of airplanes, automobiles, dogs, and frogs.
questions: Why did the authors choose Metropolis-Hastings rather than other MCMC methods? Is there any reason or advantage?
limitations: The authors adequately addressed the limitations in Section 5 and the potential negative societal impact in Appendix F.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a method to generate multi-label images by using a (conditional-)GAN architecture. Because it is quite expensive to annotate a large dataset with multi-label annotations, this paper focuses on the single positive setting where each image is annotated with a single positive label. The paper proposes a novel sampling framework based on the Markov chain Monte Carlo method for generating multi-label images from single positive labels. The proposed method shows good performances in two settings: sampling for classes with inclusion and sampling for multiple classes.
strengths_and_weaknesses: **Strengths:**

+ I think the idea of generating multi-label images is interesting. As mentioned in the paper, building large scale multi-label datasets is expensive so the approach of using partially labeled images looks very promising. Each image is annotated with a single positive label.
+ The problem formulation with intersection index set and difference index set seems novel.
+ The paper proposes a novel sampling framework based on the Markov chain Monte Carlo method for generating multi-label data from single positive labels.
+ The proposed method is evaluated in two settings:
  - sampling for classes with inclusion, where one class is entirely contained in another class. The proposed method shows better results on MNIST-3/5, MNIST-Even, and FMNIST-Even than RumiLSGAN and CP-GAN.
  - sampling for multiple classes, where multiple classes are overlapping. The proposed method shows better results on CIFAR-7to3, CelebA-BMS, and CelebA-ABMNSW than existing models like CP-GAN.

  For both settings, some visual results are shown and multiple metrics are used to evaluate the model performances.   

+ The description of the limitations is interesting because it gives some insights about some potential next steps to improve the proposed method.

**Weaknesses:**

- The proposed approach seems to be specific to GAN, and cannot be used directly in other generative models like VAE. 
- There is no information on how the examples in ​​Figure 5 and 6 are chosen.

questions: - I wonder what was the motivation to focus only on the single positive setting instead of a more generic setting e.g. at least a single positive label. For example, OpenImage dataset has a lot of partially-labeled images.
limitations: I think the authors have adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a sampling method based on MCMC called single to multiple sampling (S2M). A few applications for this “add-on” method on unconditional and conditional GAN is applied to show the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths: 

The proposed method seems to be able to help resolving the multiple label learning from single positive samples to some extent, which may serves as an interesting supplement to the existing methods, although it is not clear how good it is compared to other methods.

Weakness: 
(1) One of the main concerns for this paper is that on the topic of multiple label learning from single positive samples, there are already quite a few solid study from previous work. For instance, in the reference [1]:
Cole et al “Multi-Label Learning from Single Positive Labels”, CVPR 2021.
The submission fails to connect their work to the reference [1]. Thus, it is not clear the importance of the S2M method for improving this type of problem. For example, how is the performance comparison of the proposed method vs the techniques described in [1] including “label smoothing, expected positive regularization, pseudo-negative sampling, etc” in both theoretical analysis and experimental study.
Without such important study, the novelty and significance of the paper does not stay on the safe ground.

(2) It is also not clear how does the proposed method scale to large dataset. 

(3) While the paper proposes an interesting application to enhance the results with S2M sampling, the overall depth of the paper clearly falls below the Neurips acceptance bar. It will be nice that the author can motivate their method better. Does the S2M method provide any advantages compared to label smoothing, pseudo-negative sampling etc ?
(4)  As we know, the sampling based method usually needs to accumulate sufficient number of samples to be effective, however, typically in the regime of multi-label learning from single positive labels, the data/labels are limited. Thus it is questionable that practically, how useful the method can be.
(5) The experimental validation is not sufficient and convincing as it only considers a special case, which makes the application domain quite limited

questions: The main questions for this paper are :

(1) Lack of sufficient comparison with other methods in this area in both theoretical study and experimental analysis (2) the motivation is not clear (especially for low data problem) (3) the insufficient experimental validation
limitations: N.A.
review_title: null

Meta-review: metareview: This paper addresses the GAN generation with multi-label condition variables problem by a single-to-multi-label generation method. More specifically, given only one positive label for each image, this paper proposes a MCMC sampling method to generate combinations of labels and reduce the cost for data annotation. This is a challenging problem and the authors have made clear assumptions under which the sampling can be successful. The experimental results demonstrate the effectiveness of the proposed method.


Overall, the paper is novel and interesting.  I would recommend acceptance of this paper given the novelty of the idea and the technical soundness. However, I would suggest that the authors could narrow the scope of “multi-label” to multiple attributes. It is unclear whether the proposed method can handle the general multi-label problem in which multiple objects could appear in an image.

confidence: Certain
Decision: Accept