Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
This paper proposes Group-Supervised Learning (GSL) that can learn disentangled representations by swapping components in latent space, and enable one-shot novel view synthesis.  They created large dataset to evaluate their method, and demonstrate its effectiveness in controllable image synthesis, and disentanglement, outperforming existing baselines.

Pros:
1. This paper is good-written and clear to follow. The authors can demonstrate their idea well in Section 3 and 4.
2. Swapping attributes is an interesting and novel idea to encourage disentangled representation learning, which are easy to implement and could have wide applications in many fields.
3. The experiments are extensive, and the results are able to support their claims.

Cons:
1. In Equation (4), can you explain the defination of  separate losses, L_r, L_sr and L_csr along with the equation? They are not mentioned above, but simply in the algorithm 1. 

2. If the cycle loss is removed, will the performance degrade dramatically? Can you please offer ablations for each loss?
review_title: A nice contribution on disentangled representation learning and novel view synthesis

final_score: rating: 7: Good paper, accept
review_content: I. Summary of paper.

The authors propose GZS-Net, a model based on an auto-encoder that synthesizes realistic images by exploring the semantic relationships between group images. The model is trained with a family of objective functions expressed on groups of examples. 

It first maps training images into disentangled latent representations, which can be decomposed into multiple components, which in the specific implementation correspond to semantic attributes. Components are then recombined to form novel images that can correspond to previously unseen combinations of attributes. 

The main experimental setup considered is zero-shot image synthesis: having seen different combinations of attributes, the model relies on disentangling attribute representations to synthesize novel combinations of previously seen attributes. Good performance is achieved on the Fonts, ilab-20M and RaFD datasets. In addition, the authors perform a more detailed analysis of their proposed approach and show that it results in good downstream performance on an object recognition task.

II. Strong and weak points.

Positive:
- The proposed task, zero-shot synthesis, is interesting.
- Overall, the paper is well-written and clear
- Good results against the selected baselines
- The novelty of the approach itself.

Negative:
- The following justifies some of the comments below. The authors claim GAN-related training is unstable with significant mode collapse. This is simply not the case currently, and in particular in the relatively small datasets considered. A comparison with e.g. cycle-GAN is fully warranted.
- The qualitative comparisons are limited. Yes, the model can generate good-looking images on what are mostly toy tasks. But these generations are commonly compared only to a relatively weak baseline (AE + DS) in fig 6, 5. Cycle GANs would work in this setting, and I supposed might be very competitive.
- The paper suffers from a general lack of (serious) quantitative comparisons. The authors provide generated images but the single results table on disentanglement does not contain models that would fare well in this setting.

III. Recommendation and justification.

I feel the paper is borderline, with a slight inclination to reject. On one hand, an interesting task is presented and the writing/clarity/experiments make this submission an interesting submission from a scientific standpoint. On the other hand, a lot of experimental work is needed to show this approach offers a benefit compared to serious baselines.

IV. Advice for author response.

I like this paper conceptually and would be prepared to raise my score. I'm interested in the following, if possible:
- Is there a way for the authors to provide an indication of sample quality versus adversarial baselines (cycleGAN, starGAN or some related method e.g.) to see if results hold?
- On the disentanglement side, is there a way to extend Table 1 to show whether the model performs well when compared to models that are good at disentangling, e.g. beta-VAE family.

If not feasible to include new experiments, please if possible at least discuss the issue wrt. baselines.

V. Improving the paper.

On the link between zero-shot learning, disentanglement, and generative models, I would suggest including [1, 2]. To be clear, these are just suggestions, choosing to include them or not will have no impact on my rating.

References

[1] Sylvain, Tristan, Linda Petrini, and Devon Hjelm. "Locality and Compositionality in Zero-Shot Learning." International Conference on Learning Representations. 2019.

[2] Higgins, Irina, et al. "SCAN: Learning Hierarchical Compositional Visual Concepts." International Conference on Learning Representations. 2018.

Edit: I have read the author's response. They have responded to the questions I had, and they address many of the concerns I had. I am now raising 


Edit: updating score, and recommending acceptance as per my response to the author's rebuttal.
review_title: Review of "ZERO-SHOT SYNTHESIS WITH GROUP-SUPERVISED LEARNING"

final_score: rating: 7: Good paper, accept
review_content: Summary:

The paper proposed a new training framework, namely GSL, for novel content synthesis. And GSL enables learning of disentangled representations of tangible attributes and achieve novel image synthesis by recombining those swappable components under a zero-shot setting. The framework leverages the underlying semantic links across samples which could be instantiated as a multigraph. Cycle-consistent reconstruction loss as well as reconstruction loss are computed on synthetic samples from swapped latent representations.

Reasons for score:

I vote for accepting. Overall, I think the paper is well written and the idea of GSL leveraging the underlying semantic relationships between samples to learn disentangled representations of tangible attributes is pretty novel and interesting. Experimental result on several real and synthetic datasets are also promising.

Concerns:

1) How are hyperparameters {d_j}^m_{j=1} selected? It would be great if the authors could elaborate more about this.

2) Concerning the cycle attribute swap, how to guarantee the intermediate generated image to be reasonable instead of a trivial solution. Seems like no losses (possibly like adversarial loss) is directly added on the intermediate result.
review_title: Recommendation to Accept

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a Group-Supervised learning method for zero-shot synthesis. This method can generate multi-attributes images through the GZS-Net.
However, the authors may pay attention to the following questions:
1. The authors should check for grammar mistakes, e.g.,”Fonts is a...” ->”Fonts are...”.
2. From the description, the Group-Supervised learning method is just a special data sampling manner used in training. Besides, the total framework is commonly used in GANs for attribute control generation. 
3. The contrast methods are too simple. The reviewer thinks the authors should compare their method with some new works, e.g., “ELEGANT: Exchanging Latent Encodings with GAN for Transferring Multiple Face”.
4. In Section 5.3, the reasons for the results in Table 1 are not analyzed. Why is the accuracy of contrast method AE-DS higher than GZS-Net?
5. The author should add more quantitative analysis. Such as comparing the FID/IS values of generated results using different methods.
review_title: More analysis and experiments are needed

Meta-review: This paper presents a zero-shot generation approach by disentangling representations into swappable components (each component corresponding to an attribute) and then conditioning on any desired combination of attributes to do zero-shot synthesis of samples containing those attributes.

There were some concerns raised in the original reviews which the authors have addressed in the rebuttal and the revised submission. Post the discussion phase, all reviewers see merit in the proposed ideas and unanimously recommend acceptance. Based on my own reading of the paper and the reviews/author responses, I agree with the assessment. 
Decision: Accept (Poster)