Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The ultimate goal of the paper is to train an unsupervised generative model, in a semi-supervised manner, as a synthetic training example generator for training segmentation models. Prior works (eg. DatasetGAN) show that one can label a few synthetic images from a pre-trained generative model, and train a segmentation model (called annotation model) over the intermediate representation produced by the generative model. Then, every time a sample is taken from the generative model, the corresponding labels can be produced by applying the annotation model to the corresponding intermediate representations. This provides a valuable way to produce synthetic labelled training examples. However, (some of) these existing approaches require annotating (a limited set of) generated samples for training the annotation model. This is problematic as one needs to re-label such images every time the generative model changes.  

This paper aims to address this problem by learning the annotation model over annotated real images instead of annotated synthetic images. To this end, the paper proposes a meta-learning approach. The main idea is to learn the annotation model such that when a (student) segmentation model is trained over the generative model outputs and their labels given by the annotation model, the resulting student model’s loss shall be low on real annotated images. This meta-learning idea is implemented as an approximation by using the Gradient Matching Loss.

main_review: Strengths:
- The paper makes a valuable improvement over the DatasetGAN-like approaches by removing the need for annotating synthetic samples. 
- The motivations and the proposed solution is clearly explained in the paper. 
- The approach also appears to be an effective alternative for traditional semi-supervised learning. The experimental comparison to semi-supervised methods mostly appear to be convincing enough.
- The paper also reports promising results in comparison to DatasetGAN, though the results are a bit mixed in terms of the performance.

Weaknesses:
- SemanticGAN's relevance is not clearly discussed, despite the experimental comparisons made in Section 4. In particular, it should be clarified in intro/related work, whether this work proposes an alternative approach to what SemanticGAN does, or whether they differ in terms of the data/annotation requirements. Please clarify.
- The paper seems to have missed a closely related work:  Gradient Matching Generative Networks for Zero-Shot Learning (CVPR 2019). This CVPR’19 paper appears to introduce the “gradient matching loss" (the same core formulation, and possibly its name as well) for training conditional generative models, as an efficient approximation to inner loop optimization for meta-learning. It trains synthetic train data-generating conditional generative models, such that when a classification model trained over the synthetic samples the resulting model’s loss is low on limited labelled samples, which is clearly closely. While the overall differences are significant enough (eg training conditional generator vs annotator only, segmentation vs zero-shot classification), the core idea, the use of Gradient Matching Loss and the overall goal of leveraging generative models for labelled data generation via meta-learning principles appear to be closely related, therefore, need to be discussed in the paper.
- The experimental comparison to DatasetGAN is not fully clear (to me). DatasetGAN uses the annotated synthetic samples. Which labelled examples do you use for the proposed approach in these comparison experiments? It would have been valuable to do this in two ways: (i) train over the synthetic images used in DatasetGAN experiments, and (ii) train over alternative real training images.
- The paper does not discuss whether it can be used on complex scenes, given that it is typically much more difficult to obtain unsupervised generative models of complex scenes.
- It would have been very valuable to see an evaluation that shows the detailed analysis on the annotation cost vs final segmentation accuracy (in mIoU), when real images are labelled & used with the proposed approach, in comparison to labelling synthetic images to be used in DatasetGAN. Currently it is unclear how many more real images need to be annotated to match the DatasetGAN's sample generation performance, given that DatasetGAN performs better in several cases (Table 1).   
- Fig 2: what happens if you train fine-tuned the model only on real annotated images, using the same training procedure, starting with pre-trained backbone, with optimized hyper-parameters?
- How would it perform if you were to train the segmentation model solely on the available real samples, apply it to synthetic images for pseudo-labeling the pixels? This baseline also seems to be missing.
- Please clarify at which stage(s) the synthetically generated images are being used in the semi-supervised experiments (Figure 2). Do you use the synthetic images as the unsupervised image set?
- How would it perform if you were to actually do the inner loop optimization? While the computational advantages are clear, the difference is not empirically evaluated. In particular, how would the approach work with a 1st-order / kth-order MAML-based loss? 

summary_of_the_review: The paper proposes a reasonable and interesting meta-learning approach towards leveraging (pre-trained) generative models for constructing synthetic labelled training sample sets. The overall approach is interesting, the results are promising yet the paper (in text and in experimental analysis) seems to have various weakness, as discussed above. 

While I do *not* consider that the existence of all suggested experimental evaluations is a pre-condition for acceptance, the current experimental analysis seem to be underwhelming in several aspects. In addition, all the textual weaknesses & ambiguities definitely need to be addressed/clarified in the comments/revision.

— Post revision/rebuttal update —
I have gone over all the reviews, responses and the updates in the paper. The revision and responses address my concerns. The text and experimental evaluation have been improved on several aspects, though there is still room for improvement in experimental analysis. Overall, based on all positive improvements, I increase my recommendation score from 5 to 6.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method to solve the problem that generally requires large-scale labeled datasets to train deep learning models. The proposed method is designed to solve the nested-loop optimization problem (with an annotator that generates a label and a student network that predicts a label) based on a pre-trained GAN that generates images. In particular, the authors solve this problem through a gradient matching approach, and they claim it is much more efficient than the existing end-to-end gradient-based meta-learning approach. This method was applied to the semi-supervised part segmentation task, and its effectiveness was verified through various experiments.

main_review: This paper has the following Strengths.

- The problem covered by the paper is important, and a good direction is proposed based on clear motivation.
- Experiments were sufficiently conducted to verify the effectiveness of the proposed method.
- The paper is written in an easy-to-understand manner through detailed formulas and figures.

However, the following concerns exist.

1. Comparison with the end-to-end gradient-based meta-learning approach
    - In section 3.2, it is stated that the nested-loop optimization problem can be solved through existing end-to-end gradient-based meta-learning approaches. And the authors claim that those methods are complex and expensive, so they suggest an efficient gradient matching approach. However, I cannot find any experiment to show how efficient and effective the proposed method is compared to them, and this is my biggest concern.
2. Images in batch to get gradient matching loss
    - According to Algorithm 1, the images used to compute each gradient appear to be identical. (The indices of $x_i$ for $B_l$ and $B_g$ are the same.) However, in Figure 1, the two images for each gradient are different, causing confusion.
3. Pascal-{Horse, Aeroplane} dataset
    - For this paper, images of Horse and Aeroplane were selected from the Pascal Part segmentation dataset for the experiment. Is there any particular reason for choosing these two classes (among 20 classes)?

summary_of_the_review: This paper proposes an intuitive method based on clear motivation. In addition, sufficient experiments have been conducted to prove it, and the paper is written in an easy-to-understand manner. However, one big concern (#1 above) remains, so I give my initial rating as 6.

**[Comments After Author Response]** I appreciate the efforts of the authors for the detailed rebuttal. The authors have well addressed my concerns (other reviewers' as well), so I raise my rating to 8. I strongly recommend that the authors will reflect those contents in the final version.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work deals with the problem of training a part segmentation network by automatically synthesizing pairs of images and annotations. The authors propose a training method of an annotator given the well-pretrained generator model. The formulation is well-motivated and converged to the gradient matching loss. 

This work extends [Zhao et al., ICLR 2021] by combining the idea of DatasetGAN and RepurposeGAN in an interesting way.
The proposed approach can favor the unlimited number of annotated data with only limited supervised real data + the pre-trained GAN.

The proposed method shows noticeable improvement over the other baselines and the competing methods in scarcely labeled data regimes (especially semi-supervised learning).

main_review: <Pros>


- The authors argued that the proposed method has a few benefits over the prior arts. The limitations of the prior arts (DatasetGAN and RepurposeGAN) are discussed: 1) they are only applicable to generator-specific generated data, and 2) they need manual re-labeling when the generator is changed. In contrast, the proposed method does not have such problems.


- In addition, this work does not require manual labeling of generated images, but requires manually labeled real data. 


- The learned annotator is student-agnostic


- A naive counterpart could be the GANinversion baseline. It is well-compared in the supplementary.


- The authors tackle a good scope range that has not been dealt with similar works, DatasetGAN and RepurposeGAN.


- Diverse experimental setups



<Cons>


- The mathematical derivation and formulation are primarily borrowed from [Zhao et al., ICLR 2021]. In this work, the synthetic dataset in [Zhao et al., ICLR 2021] is parameterized by GAN and the learnable annotator. This is on the borderline.


- While the authors argued that the proposed method significantly outperforms the semi-supervised competing methods, but the data regime is limited to low-data regimes. The effectiveness of this approach is unknown for the large semi-supervised data regimes (sufficient labeled data + very large unlabeled data).


- While the proposed approach has merits, its practicality is questionable in that the approach assumes the high fidelity pre-trained GAN. It would be very interesting to see how much robust the proposed method is with immaturely pre-trained GANs (i.e., low-quality GAN).


- The experiment settings are a bit biased to favorable settings to the proposed methods. Including the above concern, the experiment setups of DatasetGAN and RepurposeGAN are less considered. Are there specific reasons? Also, large-scale experiments and diverse balance setups could have been regarded.


<Questions>

- In the 1st paragraph of "Comparison to DatasetGAN", the authors mentioned that "FID cannot depict the disentanglement of generator features." What is the relationship of the proposed approach with disentanglement?


- The authors argued the non-necessity of labeling generated images as a benefit of the proposed method (including the one in "Comparison to DatasetGAN", Sec. 4). However, the proposed method requires the labeled real dataset as well. In order to validate the argument, the amount of the manual annotation cost (the number of labeled data) should be compared.



<Minor comments>


- The notation convention: In the paper, $\mathbf{\hat y}$ denotes manual annotation, while $\mathbf{y}$ denotes the predicted output. Conventionally, it it more often the other way around.


- $\mathcal{D}_l$ is abused for both a real labeled dataset and a labeled dataset of generated images, which may lead to confusion. It would be clearer to distinguish the notation.

- In Algorithm 1, the annotator and segmentor (called as student) are alternatingly optimized by each other. In this manner, the term student is misleading, since the role of the student is altered.

- Typos and grammars: There are many grammar errors, especially with articles, singular and plural forms.

summary_of_the_review: The submission has a few limitations, including the limited demonstration of its effectiveness in low-data regimes and grammatic errors. However, this work extends [Zhao et al., ICLR 2021] by combining the idea of DatasetGAN and RepurposeGAN in an interesting way. The merits of this work seem to outweigh the demerits.

===== After rebuttal ===== 
This reviewer stays at the same rating because the merits of this work seem to outweigh the demerits even after the discussion phase.
It would have been good to show the limitation and break-down point analyses in the submission.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors mainly propose a gradient-matching-based method for part segmentation to reduce the annotation cost. Based on the DatasetGAN, the proposed model also used the Style GAN family to generate high-quality images and remove the human annotations on a handful of synthesized images. Compared with semi-supervised learning methods, their proposed method achieves higher performance on three datasets. 
main_review: Reducing the annotation cost is a very important thing. In this work, the authors provided a potential and promising way based on DatasetGAN. 

Majors. (1) Contributions. In the last paragraph in Section Introduction, the authors might not very clearly summarize their contributions. I suggest they could highlight the contributions one by one. (2) Method. During the generation process in eq. (1), the authors just did the random sample or sample the z according to the labeled data? Like the examples in Fig. 1, the generated data has similar attributes with the labeled data, such as the view. (3) Experiment. (a) Datasets. The authors use three datasets in the experimental part. Maybe, it might not enough to evaluate the effectiveness. DatasetGAN, the most related work, used more than 3 datasets. (b) Dataset setting. The authors need to explain the reason why they followed (Li et al. (2021), not DatasetGAN (Zhang et al., 2021). In table 1, they just copied the numbers from DatasetGAN (Zhang et al., 2021). I suggest the authors could add a summary of all test datasets as Table 1 in (Zhang et al., 2021). (c) Table 1, the authors also need to provide more details about how to get the results of DatasetGAN, which are not the same ones in DatasetGAN. 

Minors. (1) There exists some typos, such as “our method achieve same performance” -> ‘achieves’. (2) The citations of TL and SSL in Table 1 might be modified to the original work, and the authors might add some footnotes to indicate they just copied the numbers from (Zhang et al., 2021). 

summary_of_the_review: The contributions might need to be clearly summarised and I vote 'the marginally below the acceptance threshold', based on the limited experiment and the unclear experimental settings.
review_title: null

Meta-review: This work deals with training generators of aligned pairs of images and segmentation maps. It is based on the recent DatasetGAN approach, which generates images and maps, but requires human annotations on a handful of generated images. This paper is addressing this problem by learning the annotation model over annotated real images instead of generated ones. To this end, the paper proposes a meta-learning that uses the Gradient Matching Loss.

Overall, the rebuttal provides valuable insight and many issues raised by reviewers have been convincingly answered by the authors.
On the whole, the reviewers converged positively, the novelty and the interest of the proposal stand out clearly, and this despite the lack of very convincing experiments, at least before the rebuttal. Authors are strongly encouraged to take all comments into account for their final version.
Decision: Accept (Poster)