Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper considers the task of learning GANs that decompose the image formation process into foreground, background and mask generation and composition. Compared to previous methods, the proposed ComGAN aims to avoid trivial solutions (where masks do not correspond to foreground objects) mainly through the network architecture instead of regularizations, which often require extensive hyperparameter searches for suitable regularization strengths.
strengths_and_weaknesses: - Strengths
    - The proposed approach simplifies the design of compositional GANs compared to previous methods and demonstrates improved performance in terms of synthesis quality as well as unsupervised segmentation performance.
    - Compared to similar compositional generative models like FineGAN [25], the supervision requirements regarding weak background supervision is further reduced.
    - Finding a compositional generator architecture that is stable to train has many applications beyond GANs. Thus, the work is potentially interesting for a larger audience.
    - Although there are still a few hyperparameters (mask consistency loss weight, binary regularization weight, dimensionality of latents, relative sizes of subnetworks), experiments demonstrate some robustness to these parameters.
- Weaknesses
    - The core idea and differences to previous approaches are not clearly stated.
    - The requirements for Proposition 1 regarding what it means for an architecture to be "similar to ComGAN" are not stated clearly. I assume the key point is that M consists of only an sigmoid layer. However, the formulation in l. 149 seems to be the only place where this is stated and even there it remains vague and could be interpreted as containing the same layers as F and B and in addition a sigmoid layer. The latter interpretation is also what Fig. 3 suggests (albeit with one residual block less). If this (a shared decoder with a minimal mask decoder) is the key idea of the paper, it should be communicated more clearly and probably also on a higher level already in the introduction. Without clear restrictions on the architecture, one could also think that FineGAN satisfies the requirements with G set to the identity.
    - The lemmas, propositions and proofs seem a bit vague as they do not clearly state assumptions or define all involved quantities. In l. 160-161 it is not clear what is meant by $\bar{x}_m = M^{-1}(G(z))$ - why would the input of M equal its output? Intuitively, I also don't see how "it is clear that any change of foreground or background affects the mask, [...]". Since F and B do contain additional layers, foreground and background could be affected by changes in those layers even though G(z) and hence the mask would remain unaffected, no? The other way, that any change of the mask affects both foreground and background, seems to be true.
    - Motivation for DS-ComGAN architecture is unclear: Why is the $\bar{x}_z$ output needed in addition to the output composited from foreground, background and mask?
questions: - l. 47-60 could be moved to the related work section
- l. 294 refers to the wrong figure
- l. 298: how does fine-tuning mitigate masks that are inconsistent with foreground images? Also, what is the effect of the mask distribution alignment (l. 206)? Why not train the segmentation model S with Eq. (13) only?
- It would be helpful to use the same evaluation protocol as in FineGAN (based on 30k samples) for Tab. 3. The reported Inception Scores seem very low.
- See also weaknesses for questions
limitations: Limitations and potential negative societal impact have been addressed adequately.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors point out two factors that lead a scene decomposition model to fall into trivial solutions in a mathematical analysis.
Those are related to the vanishing gradient phenomena that goes into a mask generator.
To avoid these, they propose a novel network architecture, where features for generating decomposed scene elements are composed of the ones that used for generating the entire scene at once. 
With this architecture, they achieved the SOTA scores on both the mask prediction and the image quality evaluation metrics.
strengths_and_weaknesses: Strengths
- The authors tackle the important problem in the scene component generation models.
- They propose a novel architecture for robust mask generation based on the theoretical analysis on the problem.
- The authors did a thorough ablation study to show that each element proposed are all effective.
- The comparison of both quantitative and qualitative results with previous works imply that the proposed method is effective in boosting the scene decomposition performance.

Weaknesses
- The connection between the theoretical analysis and the proposed architecture design is not well established. More details are needed to understand why the proposed architecture is helpful for model to not fall into the vanishing gradient phenomena.

questions: Questions
- Why the proposed network is helpful for avoiding the vanishing gradient issue?
- Why the performance of PerturbGAN was not compared?

Suggestions
- Maybe simplegan —> finegan at the line 221?
limitations: Authors did not address the limitations and potential negative societal impact of their work.

Suggestions
- I want to know the authors’ opinion on how much would it be difficult to apply this method on coarse grained datasets like ImageNet?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work analyses the reason for trivial solutions during mask learning in image composition GANs, and introduce a new model architecture ComGAN to solve the trivial solution issue.  Furthermore, an unsupervised object segmentation module is also involved to construct the DS-ComGAN model. DS-ComGAN can perform both disentangled image generation and object segmentation, and outperforms semi-supervised and weakly supervised baselines.
strengths_and_weaknesses: Strengths

- It is claimed that this work is the first to solve the trivial solution in disentangled image generation by changing the network architecture. The change is simple to apply and obtained significant improvement. I believe this technic can also be useful for other models and tasks.

- Both disentangled image generation and object segmentation tasks can be performed in a single framework. More importantly, the learning can be achieved in an unsupervised way by a carefully designed adversarial learning strategy. 

- Sufficient experiments and analyses have been done to demonstrate the effectiveness of the proposed method.

Weaknesses

My main concerns are about the description.

- The description in sub-section 3.1 is not that readable. I suggest improving the description with a more intuitive illustration to point out the key contribution: how to avoid vanishing gradient from the network architecture perspective. Besides, some symbols seem not consistent. Do the variable f in equation 9 and the variable F in Figure 3 represent the same thing? What does the variable mean?

- The description of the mask distribution alignment could also be improved. If I understand correctly, the proposed Segmentation Networks S does not need any paired/unpaired segmentation data. For $D_m$, $\bar{x}_m$ is regarded as the real and $\hat{x}_m$ and $x_m$ are regarded as the fake.

- The learning process is not clear. Does DS-ComGAN need to be trained in two stages?  Besides, the overall objective is missing and the loss term βLbinary is introduced in the experiment section instead of the method section.

questions: The presentation of the method part is not clear enough. Please check the questions in the weaknesses part.
limitations: The limitations and potential negative societal impact have been well described.
review_title: null

Meta-review: metareview: The paper proposes a compositional GAN model with a novel network architecture that solves the vanishing gradient problem underlying trivial solutions. The proposed model achieves strong results on image disentanglement and unsupervised segmentation tasks. 

The rebuttals by the authors have successfully addressed most of the concerns of the reviewers. All the reviewers are positive about this paper. Reviewer Tkuw's main concerns regarding the evaluation and the clarity of the method were addressed. The reviewer raised the rating. Reviewer 19KW felt positive about section 3.1 in the revised version and the additional empirical results regarding the gradient values observed during a training stage as given in Figure 5. The reviewer also updated the initial rating. Reviewer rWMN's concerns have also been addressed. The reviewer appreciates the additional detailed theoretical analysis on the problem.
confidence: Certain
Decision: Accept