ComGAN: Unsupervised Disentanglement and ﻿Segmentation via Image Composition
Authors: Rui Ding
Kehua Guo
Xiangyuan Zhu
Zheng Wu
Liwei Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Generative Adversarial Networks
Trivial solutions
Image Disentanglement
Unsupervised Segmentation

Abstract: We propose ComGAN, a simple unsupervised generative model, which simultaneously generates realistic images and high semantic masks under an adversarial loss and a binary regularization. In this paper, we first investigate two kinds of trivial solutions in the compositional generation process, and demonstrate their source is vanishing gradients on the mask. Then, we solve trivial solutions from the perspective of architecture. Furthermore, we redesign two fully unsupervised modules based on ComGAN (DS-ComGAN), where the disentanglement module associates the foreground, background and mask with three independent variables, and the segmentation module learns object segmentation. Experimental results show that (i) ComGAN's network architecture effectively avoids trivial solutions without any supervised information and regularization; (ii) DS-ComGAN achieves remarkable results and outperforms existing semi-supervised and weakly supervised methods by a large margin in both the image disentanglement and unsupervised segmentation tasks. It implies that the redesign of ComGAN is a possible direction for future unsupervised work.

ComGAN: Unsupervised Disentanglement and Segmentation via Image Composition
Rui Ding Central South University ruiding@csu. edu. cn
Kehua Guo" Central South University guokehua@csu. edu. en
Xiangyuan Zhu Central South University
zhuxiangyuan@csu. edu, cn
Zheng Wu
Liwei Wang Central South University wang. www.i@csu..u..... edu. cn
Central South University wuzhhngghuse@gmaill. com
Abstract
We propose ComGAN, a simple unsupervised generative model, which simultane- ously generates realistic images and high semantic masks under an adversarial loss and a binary regularization. In this paper, we first investigate two kinds of trivial solutions in the compositional generation process, and demonstrate their source is vanishing gradients on the mask. Then, we solve trivial solutions from the per- spective of architecture. Furthermore, redesign two fully unsupervised modules based on ComGAN DOS-ComGAN), where the disentanglement module associates the foreground, background and mask with three independent variables, and the segmentation module learns object segmentation. Experimental results show that (i) ComGAN's network architecture effectively avoids trivial solutions without any supervised information and regularization; (ii) DS-ComGAN achieves remarkable results and outperforms existing semi-supervised and weakly supervised methods by a large margin in both the image disentanglement and unsupervised segmen- tation tasks. It implies that the redesign of ComGAN possible direction for future unsupervised work.'
Introduction
Generative adversarial networks (GANs) [1] have been successful in realistic image generation [2, 3], and recent work on GAN-based image composition [4, 5, 6, 7, 8, 9] has shown that GANs identify and disentangle class of objects from the background. To direct that GANs learn the distinction between foreground and background, this kind work commonly requires the following assumption:
Assumption (Image composition [10]) image x taken from the world typically composed of foreground y and background Eb, which can be decomposed by the following equation:
x == xz @ Fm + 3. 3 (1 Km))
where I'm is the mask, and the @ denotes element wise multiplication operator.
notable that the Assumption is mild and achieves significant outcomes in many areas. For example, the models [11, 12] capture meaningful foreground segmentation masks via exactly coupling different foregrounds and backgrounds. C3-GAN [13] proposes scene decomposition based method to enforce the model to learn the features of foreground objects. Yang et al. [14, 15] detect moving
"Corresponding author
'Code and data available http:p:://gglllb..c////uuaa///// http:///iiiiin///nn//namm
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

objects in video by minimizing the mutual information of foreground-buckgroundd and training in an adversarial manner. class of methods [16, 17] explores and perturbs the latent space of pre- pre-trained GANs to find foreground masks. However, it is observed that under the Assumption 1. the generation process is always accompanied by trivial solutions [10, 18, 11, 19, 20, 21, 22, 12]. Trivial solutions [11] can be considered as meaningless masks generated by models. To our knowledge, existing works do not indicate that the source of the trivial solutions and they alleviate this issue in two ways. One way is to add supervised information [23, 21, 19, 18], such as CGN [19] avoids trivial solutions by adding pre-trained U2-Net [24]. Another is to design clever regularization and fine-tune the parameters [10, 11, 20, 22, 12]. For example, PerturbGAN [11] proposes regularization based on image recombination with arbitrarily small relative shifts. Yang et al. [20] propose a regularization based mutual information maximization. These methods are sensitive to object scale, object category and datasets, and therefore consume computational resources and time costs to perform hyperparameters fine-tuning. For our solution, we propose an unsupervised generative model called ComGAN, which simultaneously generates realistic images and high semantic masks. In addition, our model effectively avoids trivial solutions from the perspective of architecture, which implies that the model does not require any supervised information and explicit regularization. More specifically, ComGAN is generic way to generalize two typical image compositional generation methods and alleviate the shortcomings coming from the above two methods. Furthermore, we place unique restrictions on each module, which help the model avoid trivial solutions.
Trivial solutions also have a negative role in related tasks. For example, in the image disentanglement, low semantic masks cause foreground and background conflicts [25, 26], which reduce the synthesis quality. In unsupervised segmentation, the regularization to alleviate trivial solutions may induce over- fitting of the segmentation masks [14, 15, 12]. To highlight the flexibility and robustness of ComGAN, we extend it to the above two tasks. The ComGAN -baed variant is called DS DS-ComGAN, which contains both disentanglement and segmentation modules. For multi-factor disentanglement, existing methods [26, 21, 27] rely on additional supervised information to learn the distinction of image regions. Notice that ComGAN achieves bengrouund-bacgrroung disentanglement in an unsupervised way. As a result, the disentanglement module is designed as follows: we add more global information to the shared features in ComGAN and maximize the mutual information between variables and images. This module simplifies the previous hierarchical generative network and outperforms the state-of-the-art (SOTA) semi-supervised and weakly supervised image disentanglement methods. For unsupervised segmentation, existing methods [ [14, 15, 11, 12] rely on strong assumptions, such as that the foreground and background are largely independent, which limits their applicability. Different from these methods, we train a segmentation network using the images and semantic masks synthesized by the disentanglement module. By adversarial training strategy, the image distribution and the mask distribution are aligned. Furthermore, a consistency regularization is introduced to ensure that the predicted masks are consistent with the inputs. This segmentation module relies only on mild Assumption and outperforms the SOTA unsupervised segmentation methods.
The main contributions are highlighted as follows:
1. To the best of our knowledge, we are the first to find that the source of trivial solutions is vanishing gradients on the mask. Furthermore, we propose ComGAN, simple unsupervised generative model, which is the first to solve trivial solutions from the perspective of architecture and achieve doreground-backgrown
disentanglement with only an adversarial loss and binary regularization.
2. We propose DS-ComGAN, a variant network based on ComGAN, which achieves image dis- entanglement and object segmentation in fully unsupervised manner. Experiments show that DS- DS-ComGAN is robust to various datasets and outperforms SOTAs in both tasks.
2 Related Work
GAN-based Image Composition. Image composition can be regarded as combining multiple visual areas to construct realistic image [28]. A series of typical methods built on GANs [1] perform image composition by utilizing Assumption I to achieve various functions. LR-GAN [10] composites realistic images by learning to generate the backgrounds and foregrounds separately and recursively. FineGAN [25] hierarchically composites images, which disentangles the background, object shape, and object appearance. PerturbGAN [11] learns segmentation masks by training a generative model of layered scene and composing foregrounds and backgrounds. Similarly, SEIGAN [18] performs
2

---

the cut, paste and inpaint operations consistent with semantic information to obtain segmentation masks. Although the compositional image generation performs excellently in several tasks under Assumption I, these methods require additional regularization and hyperparameters tuning to avoid trivial solutions.
Controllable Image Disentanglement. The goal of controllable image disentanglement is to disen- tangle factors of variation (e.g., object shape and object appearance) towards controllable generation. InfoGAN [29] proposes semantic image generation control by imposing regularization of mutual information. series of works [25, 30, 31, 26] hierarchically generates images and disentangles the background, the object's s shape and its appearance by bounding box annotation. MixNMatch [31] extends the FineGAN [25]. which disentangles and encodes four variable factors (background. object pose, shape, and texture) from real images. Along this direction, PartGAN [30] achieves part-level decomposition by learning a part generator. Benny and Wolf [21] further propose a complex GAN-based generative model, which requires set of clean background images to simultaneously solve several tasks including image disentanglement and unsupervised segmentation. These methods [32, 33] learn the independent latent characteristics of an object, especially its appearance and pose. CGN [19] decomposes the image generation process into independent causal mechanisms, and allows for generating counterfactual images. SSC-GAN [27] is semi-supervised single-stage generative model considers three factors of variation via class labels, namely independent variables, cross-class variables, and class variables. Although these methods exhibit strong performance, they all rely additional supervised information to achieve multi-factor disentanglement.
Unsupervised Object Segmentation. Un- supervised object segmentation aims to extract a useful interpretation from an image in an unsupervised way. W-net [34] extracts foreground objects through an encoder-decoder framework and min- imizes reconstruction errors. ReDO [35] captures the mask of the object, relying on the assumption that changing the fore- ground objects does not affect the overall data distribution. Assuming that the fore- ground and background are largely inde- pendent, these methods [14, 15, 11, 12] obtain segmentation masks by reconstruct- ing the realistic images, that is, precisely combining the different foregrounds and backgrounds. Labels4Free [12] performs unsupervised object segmentation based on the idea that exact coupling between foreground and background is highly non- trivial. A similar assumption is imple- mented In method [36], where IEM learns image segmentation from an information informationnthhoretii by maximizing in- painting error. Similarly, Yang et al. [20] synthesize paired realistic images and seg- mentation masks by maximizing mutual information between generated images and latent variables. These methods frequently require stringent assumptions and are not necessarily applicable to complex real- world data.
Rodl tonoge
Fore- Submnet
Feuture Generator
Discsriminator
Z-p
Mask- Submet
Real/Fake
STTL Smutoe
Back
Subnet
Figure 1: Overview ComGAN, The generator of ComGAN is composed of the feature generator & and three independent subnetworks. Three subnetworks synthesize the foreground, background and mask with features é shared input. The "compose" indicates the utilization Assumption to com- pose the image. The discriminator D distinguishes real images from fake ones.
(a) Model Th (b) Model TTa
(c) ComGAN
Figure 2: Comparison results of the synthesized mask. ComGAN synthesizes high semantic masks, while the masks synthesized by models πΙ and both converge to trivial so- lutions. Note that IT, has the same structure as [25, 30, 31, 26], and Il, has a similar structure to [11, 19]. All models include identical adversarial loss and binary regularization / L from [19]. For each model: (left) 8 0.1, (right) $= 1.
3

---

3 Proposed Approach
3.1 Trivial Solutions and ComGAN
Source of t solutions. Two kinds of trivial solutions were introduced in [[1]. The first trivial solution degrades the whole scene into foreground background, and the mask is always full or empty. The second trivial solution makes the foreground and background identical, and the mask is meaningless for the composite scene. Then, state the source of two kinds of trivial solutions.
Lemma (Vanishing gradients on the mask). Let Lall be the overall loss and F.m be the synthesized mask. Consider model that composes images utilizing Assumption 1. There exist vanishing gradients on the mask, i.e. oT faat/Oū,,. if and only if the model converges to two kinds of trivial solutions.
Proof: See Appendix B.1.
Network architecture of ComGAN. An overview of our proposed ComGAN is illustrated in Fig. 1. The generator of ComGAN consists of two parts. The feature generator denoted as synthesizes features conditioned on the variable z, i.e. Φ Φ (z) where z is sampled from prior distribution po- The three subnetworks are defined as F, B and M. They synthesize the foreground, background and
mask, that is, f F(Ø). Io image 7 is defined as:
B()) and Zm .MØ). Driven by Assumption I, the composite
(L- I = M(( + (1-
(2)
Remark This form generalizes two rypical image compositional generation methods:
If only (-) in B is an identity map, i.e. ([((=)) = B(z), then this form is equivalent to the model H1, that is, two independent generators synthesize a composite image where shared features exist in the foreground and mask generation. For example, FineGAN [25], C3-GAN [13] and Labels4Free [12], etc. can be formulated as the model II1
(-) is an identity map, iee. ((2)) = 2, then this form is equivalent to the model II, that is, three independent generators synthesize composite image where foreground, background and mask are generated by three generators respectively. For example, PerturbGAN [11] and CGN [191, etc. can
be formulated the model T2.
From the above observations, the two typical methods have a common shortcoming, that is, models Th and IT2 both contain an independent background generation process. The term ignores the association of foreground, background and mask on the features. Moreover, if the model converges to trivial solutions, then the model degrades to an original GAN that suffers from known shortcomings, such as mode collapse. As a result, our proposed method is a generic way and mitigates the above shortcoming. Then, the adversarial training loss Lode is defined as follows:
min max pade = Ez-pas [log (D(x))) + Ez-po [1 log D(z) &.F.B.M
(3)
Obviously, Lemma points us the limitation of the compositional generation under Assumption 1. Then, we illustrate how to address the issue from the perspective of architecture.
Theorem Given a generation Gg composed of a decoder Do.: Z + é and three subnets Fop, B
and Me.: o + X. Let D be a discriminator and D' D* (G' (-)) be Nash equilibrium. the composite images satisfy max (T ((())) [| [[ ( T (x)))(())) [1 B B(
(() (1-M))))))
[ID(G()) D (-)Il < E.
< 8² and [VoT.gd'11
a, then
8222
V (0,, 8m..) E Ex)p(s) [lo (1 D DFF(
o²
(1/2 ()2
Proof: See Appendix B.1.
We denote that p = 0² (1/2-c)a and trivial masks as F", If p > O, then the updated masks are J. = MM (ooz (=)), which means the model escape from the first trivial solution, i.e. St & n.m.
8222
Corollary The following modules restrictions help the model avoid trivial solutions: the F and B are lightweight and differential, the M is a shallow network and the capacity of & is enough.
Proof: See Appendix B.I.

---

Discsriminator
"Smuimage
Real image
Snd Conr
Real/Rake
Tous
gradiout flow
Aruttioc
Sigwoid
Compuse
SynLimage
Syramark
Upsample
Fore: Subner
Discariminator
ef
Reol/Fake
Dr
Shared Feature
Mask Stabnet
Dwr
Imugle
Seumentation
Networks
Figure 3: Overview of DS-ComGAN. DS-ComGAN consists of two modules: (i) the disentanglement module perform adversarial
(blue): (ii) the segmentation model (yellow). The feature generator G synthesizes training with the discriminator D2 contain more global features. Shared features are extracted from G and fed to three subnetworks F, B and M. Driven by Assumption I, foreground, background, and mask are synthesized and composited into an image ă. The module implements image disentanglement by the adversarial loss L and the mutual information regularization fon Since the model synthesize high semantic masks, the segmentation module performs unsupervised segmentation with image distribution alignment and mask distribution alignment adversarial training.
3.2 The Disentanglement Module of DS-ComGAN
Image generation is controlled by the mask variable z, foreground variable f and background variable b, where ʃ and are both dimensional random one-hot vectors. The generator consists of two parts; namely feature generator denoted G and three subnetworks defined as F, B and M.
Global information for shared features. The features should contain global information so that the model captures a wide range of distinctions between foreground and background, rather than focusing on abrupt regions. To add global information to the shared features of G. we synthesize realistic images for adversarial training, that is, H. is synthesized by G (2) and fed to the discriminator D Then, we extract the activation map from the last three layers inside G, denoted as 01 02 and ag. The shared features with global information can be written follows:
Φ U (a1) U (a2) @ U (a3),
(5)
where U spatially upsample to maintain the same size for activation maps and ( concatenation along the channel dimension. In order to control image disentanglement, F, B and M synthesize foreground, background, and mask conditioned on f. b and z, that is, F F f). Tg B( b) and Z.. = M(Ø). Driven by Assumption the composite image ã are defined as:
I F(o,f) C M(d) B(o, b) @ M())).
Then, the discriminator D2 is trained to classify images as real or fake and the adversarial training loss Lade is defined as follows:
min max Cade G.F.B.M D. D. =E, [lgg D=(=)) + + 1og(1 D2(2=)) B.,s& [log(1 (z()))
(7)
subnetworks. A mutual information regularization incorporated into the J and associate the variables f and b with the synthesized f and Tg. Similar to InfoGAN [29], we obtain similar results by maximizing the lower bound on the variance of the mutual information.
Controllable regularization for wyrerouuddbbackgroueg
max Linfo [Tr (( Ea. log Dy( ky) + log Do(b) [a)], Dr.D,
(8)
where I(-) is the mutual information, two discriminators D, and D,, approximate posterior distribu-
tion of p(f]ă and p(b(Ta). As implemented in [37, 20, 25], the I(-) is regarded as a regularization of a class of unsupervised clusters. other words, D, guides to focus on generating foreground regions and learning the features of objects. The similar analysis applies to D,, According to [10, 38), is difficult for GANs to learn set of disconnected manifolds. Intuitively, F and learn simpler and continuous distributions, and thereby Linfo improves the quality of image generation.
5

---

3.3 The Segmentation Module of DS-ComGAN
The following two networks are defined: a segmentation network predicts segmentation masks conditioned on the input images; discriminator D., classifies masks as real or fake. Our key idea to exploit the high semantic masks synthesized by disentanglement module to align the image distribution and the mask distribution via adversarial training.
Image distribution alignment. the disentanglement module, D2 trained identify a sampled from Pdata and ε composited by our model. When the model achieve Nash equilibrium, F has the identical distribution as x, i.e. Pdata (z) = p(z). Hence, image distribution alignment is accomplished.
Mask distribution alignment. the first step, we input the real image x and the synthetic image F into the segmentation network to obtain the corresponding prediction masks, that is, 2m S()) and F.m = S( Then, a mask triplet (Tm, im. Fm) is fed into the discriminator Dm where the synthesized mask 3im regarded real and both the predicted masks è and Vmn are regarded fake. Mask distributions of Im, &m and F.m are aligned via adversarial training,
min max Lody @mm [lg ( + log(1 Dm + log(1 - Dm (9) Pm
The mask distribution alignment makes the predicted masks clearer, but this adversarial training strategy causes the issue that predicted masks are inconsistent with the input images. Hence, the corresponding consistency loss Leons is introduced to ensure that the predicted masks are consistent with the inputs, rather than merely learning the mask distribution.
min Leons Eg,z.. [| (())l]. S
(10)
We transform the unsupervised segmentation task into two adversarial tasks and acquire segmentation masks by enhancing image and mask distribution consistency.
3.4
Model Training
The whole training process of ComGAN is simple, with only an adversarial loss and a binary regularization. The learning objective is as follows:
Laul
min max Lodv + min 3 3Lainary F. F.B.B
(11)
#.M
where Coinary is the pixel-wise binary entropy of mask from [19]. An explicit formulation of Coinary shown in Appendix B.2. The sharpness of the semantic mask is modified by fine-tuning B. The results of the visualization on fine-tuning 3 are available in Appendix C.I. All the constituent networks of DS-ComGAN perform two kinds of unsupervised tasks: image disentanglement and object segmentation. Overall, we optimize the following loss:
objective for image disentanglement task
Lall max Linfo Dy.Do
min max Lodu + min max Cade + min X.Cons G.F.B.M D.
(12)
Dm
objective for object segmentation task
For the image disentanglement task, the model performed in one stage. Then, the synthesized images and semantic masks are fed into the segmentation network. Hence, the objective segmentation process is composed of two stages.
Experiments
4.1 Experimental Setup
Datasets and implementation details. The experiments are conducted on five fine- fine-graines grained image datasets and multi-object dataset: CUB [39], FS-100 [40], Stanford-Cars [41]. Stanford- Dogs [41], Flowers [42], CLEVR6 [43]. The ground truth segmentation masks of Stanford-Cars and Stanford-Dogs are approximated by following the practice [21, 44, 20J. More details about datasets and the implementation available in the supplementary material.

---

380
250
-4 x((90))
Grafeet
Gradica
250
160
150
2000
250
Epoch
Epoch
Epooch
(a) Model 1I1
(b) Model 1l2
(c) ComGAN
Figure Comparison results of gradient 2-norm during the training.
(a) CUB
(h) FS100* (c) (crsfrd--aar(d)) Stanford-Dogs
(e) Flowers
CLEVR6
Figure 5: Qualitative generation results each dataset. Zoom in better visibility. From top to bottom: (i) final images; (ii) masks; (iii) foregrounds; (iv) backgrounds. indicates that we incorporate a binary regularization to obtain a high semantic mask, since the foreground almost covers the entire image.
Baseline and evaluation protocol. In order to analyze the effectiveness of the adopted components, we established baseline model, i.e. SimpleGAN [25]. SimpleGAN aims to generate realistic image, and its network contains a generator and discriminator, For fair comparison, we use SimpleGAN the feature generator of ComGAN and DS-ComGAN. We assess synthesis quality in terms of Inception Score (IS) [45] and Fréchet Inception Distance (FID) [46], which are computed on 20K randomly synthesized images. To quantitatively evaluate the quality of the predicted masks, both the Intersecion of Union (IoU) and Dice score (Dice) [47] are used as the evaluation metrics.
4.2 Trivial Solutions Avoidance based on ComGAN
Avoidance of trivial solutions. To show that ComGAN solves trivial solutions, we select two typical networks represent model II, and II, for comparison. All models only include identical adversarial loss and binary regularization. As typical network for model II, FineGAN has two generator where a generates foreground and mask and another generates background. PerturbGAN is selected as the typical network for model 2T,, which consists of three generators that generates foreground, background and mask, respectively. For fair comparison, we make the following two things: (i) we use the code provided by the authors FineGAN and retrain it with the identical training parameters; (ii) we simplify StyleGAN [48] with SimpleGAN in PerturbGAN and retrain it. As shown in Fig. 2, the mask synthesized by ComGAN avoids convergence to trivial solutions and has non-trivial semantic information. In addition, we track the gradient 2-norm of the three models. It is clear from Fig. 4 that the gradient norms of mask networks the model II, and Ilg converge to zero and both models degrade to original GAN, while our method effectively avoids vanishing gradients.
4.3 Image Disentanglement based on DS-ComGAN
Quantitative evaluation of image generation. We compare DS-ComGAN with the competing GAN-based generative models without any advanced GAN training strategy. As illustrated in Table 1, the results show that DS-ComGAN significantly outperforms all the SOTA weakly supervised and semi- sumi-supervieed methods. FineGAN and MixNMatch may limited by the grreground-backgree conflict, such as trivial masks. SSC-GAN might have trouble in learning a set of disconnected manifolds. emphasize that DS-ComGAN is fully unsupervised and adapts to complex multi-object scenarios, we further perform our model on diverse datasets. Stanford-Dogs and Flowers typically include single foreground object, whereas CLEVR6 contains variety of occluded and truncated objects. Table 3 shows that model is robust various datasets and outperforms the SOTA image
7

---

Varying variable
Varying variables &
CUB
Stanford Cars
Stanford Dogs
Flowers
CLEVR6
Figure 6: The controllable image generation via varying variables f, and Zoom in for better visibility.
CUB
FS-100
Stanford-Cars
Methods
Sup.
FID 4
ISt
FID
1St
FID 4
IS1
Triple-GAN [49]
Semi.
140.94
3.94
91.05
1.45+0003
114.12
2.45++0.06
EnhancedTGAN [40]
Semi.
133.57
4.17+:0.66
57.58
1.57:0..02
105.20
2.43
Triangle-GAN [50]
Semi.
96.42
4.36+0.05
35.49
1.71+0.04
61.44
2.77+0.10
R²-CGAN [51]
Semi.
88.62
4.43
25.28
1.73+0.02
44.57
3.05+:0.04
SSC-GAN [27]
Semi.
20.03
4.68
20.65
1.82++0.03
39.02
3.00
FineGAN [25]
Weak.
46.68
4.62:10.3
24.63
1.76:0.00
45.72
2.8510.04
MixNMatch [3]
Weak.
45.59
4.78
25.63
1.71
45.94
2.60
SN-GAN [52]
Unsup
160.09
4..05
41.26
1.66
53.20
2.80
DS-ComGAN
Unsup.
16.26
4.7910.47
20.15
1.83
34.17
2.84
Table Image synthesis results for each dataset measured in FTD and IS. DS-ComGAN compared with the state-of-the- un(semi-)sunp sepiiissuerriieed GAN-based models. 8 indicates that models have the ability to achieve image disentanglement.
disentanglement methods, Fig 5 shows the qualitative generation results for each dataset, from which is evident that our model generates not only realistic images but also clear semantic masks.
Controllable image disentanglement. Fig. 6 illustrates that by varying three indepen- dent variables, OS-ConGAN synthesizes re- alistic and diverse images. This indicates that the model achieves controlled image dis- entanglement. We that variable sociated with the masks. The pose the object is altered with the varying variable 2. The colors and textures of the foreground and background are associated with variables if and b. To clarify the role of f and b in image disentanglement, Fig further ana- lyzes the disentanglement of foreground and background. In summary, DS-ComGAN as- sociates three independent variables (f. b and z) with the foreground, background and mask, respectively.
(a) Varying b, fixed and (b) Varying f. fixed and
Figure 7: Disentanglement of foregroun-bbacggrondd
Method
FID
1S 1
Baseline
89.87
4.45++0.46
+ Independent subnetworks.
21.35
4.75++0.08
Lin -based Reg.
17.32
4.73+10.08
+ Extract global features.
16.26
4.79-10.47
Improvement
73.61
0.34
Table 2: The results of the baseline and variants on CUB,

---

Single Object
Multi-Object
Dogs
Flowers
CLEVR6
Methods
FID 4
ISt
FID +
IS
FID 4
IST
SSC-GANS [27]
64.26
8.97=00112
29.09
3.41++0.33
FineGAN' [25]
69.52
8.27=+0.17
MixNMatch R [3]]
68.31
8.32:10.06
DS-ComGAN
60.84
9.17=10233
27.19
3.02
77.08
2.05=1005
Table 3: Performance of DS-ComGAN datasets with various attributes. Noting that the Flowers dataset lacks bounding box annotation, FineGAN and MixNMatch both unsuitahle for this dataset (marked as CLEVV6 lacks hounding box annotation and labels. Consequently, only our model is suitable for CLEVR6.
CUB
(h) Stanford-Dogs
(c) Stanford-Cars
Flowers
(c) CLEVR6
Figure 8: Qualitative segmentation results for each dataset. From left to right: (i) observed images, (ii)
ground-truth masks, (iii) predicted masks. More samples can be found in the supplementary material.
Effectiveness of model components To evaluate the contribution of each proposed component, we analyze quantitatively the performance margin between the baseline model and ComGAN. As demonstrated Table 2, the incorporation of F. and M significantly improves the performance of the baseline model. It illustrates that ComGAN has excellent image generation performance. As analyzed in section 3.2, the Linfo causes a decrease in FID by about 4 points.
4.4 Unsupervised Segmentation based on DS-ComGAN
To demonstrate that our method applies to complex real-world domains and multi-object scenar- ios with challenging spatial configurations, along the lines of [44], DS-ComGAN performs the unsupervised segmentation task on the various datasets. Then, we perform a comparison between DS-ComGAN and several state-of-the-art un (owweelly)superveeee
segmentation methods.
Qualitative analysis. In Fig. 8, qualitative segmentation results DS-ComGAN are presented. We observe that the predicted masks segment visual details more precisely than the ground-truth masks, such as the legs of birds and the rearview mirrors of cars. Although the predicted masks properly detect and segment foreground objects the Flowers and Stanford-CA, -aaford--Car datasets, the masks are inconsistent with the foreground images, which is mitigated by increasing X.
Quantitative comparisons to prior work. Table 4 demonstrates that DS-ComGAN exhibit robust- ness to diverse datasets and achieve superior performance across metrics and datasets. This implies that the model pays more attention to the discrepancy between the objects and background rather than the number of objects. addition, the result benefits from the images and high semantic masks synthesized by DS- OS-ComGAN, which ensure two high-quality distribution alignments.

---

Single Object
Multi-Object
CUB
Stanford Stans
Stanford-Can
Flowers
CLEVR6
Methods
loUt
Dicet
IoU
Dicet
loUt
Dicet
loUt
Dice
loUt
Dicef
W-Net [34]
24.8
38.9
47.7
62.1
52.8
67.6
GrabCut [53]
30.2
42.7
58.3
70.9
61.3
73.1
69.2
79.1
19.0
30.5
ReDOf [35]
46.5
60.2
55.7
70.3
52.5
68.6
76.4
18.6
31.0
OneGANo * [21]
55.5
69.2
71.0
81.7
71.2
82.6
IODINEt [54]
30.9
44.6
54.4
67.0
51.7
67.3
19.9
32.4
PerturbGAN [1]]
38.0
Slot-Attn. [55]
35.6
51.5
38.6
55.3
41.3
58.3
83.6
90.7
IEM+SegNet [36]
55.1
68.7
76.8
84.6
DRC [44]
56.4
70.9
71.7
83.2
72.4
83.7
84.7
91.5
DS-ComGAN
60.7
71.3
74.5
84.6
76.7
86.6
76.9
83.1
90.0
94.6
Table 4: Segmentation results training data measured in IoU and Dice. DS-ComGAN is compared -yuueri--ssperriieed segmentation methods. Following the [44], i indicates unfair baseline results obtained using extra ground-truth information. & represents a GAN-based model. OneGANo weakly supervised baseline, which requires clean backgrounds as additional inputs.
with the state-of-the-art
Conclusion
As the limitation method, DS-ComGAN has struggled achieve the desired performance when the foreground object features are highly diverse (e.g., HKU-IS [56]). We conjecture that the limitation might relate to the capacity of model. DS- DS-ComGAN performs excellently in controlled image synthesis tasks, which may cause the incidence of image falsification.
We first analyze that the source of trivial solutions is vanishing gradients the mask. Subsequently, we propose ComGAN, a simple unsupervised generative model, which simultaneously generates realistic images and high semantic masks, and effectively avoids trivial solutions. Then, design DS- ComGAN based on ComGAN, which exhibits excellent performance both image disentanglement and unsupervised segmentation tasks. Notably, DS-ComGAN obtains these excellent performances by relying on its architecture rather than complex and clever regularization. It implies that ComGAN has the potential shine in many tasks.
Acknowledgments
This work was supported by the Natural Science Foundation of China under Grant 62076255 and 62177047; Open Research Projects of Zhejiang Lab (NO. 2022RC0AB07): Hunan Provincial Science and Technology Plan Project 2020SK2059; Key projects of Hunan Education Department 20A88; National Science Foundation of Hunan Province 2021JJ30082, 20191120025 and 2019JJ40406; National Social Science Fund of China (No. 20&7D120).
References
[1] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Proceedings of Advances in Neural Information Processing Systems (NeurIPS). 2014.
[2] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
[3] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegant+: How to edit the embedded images? Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[4] Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2016.
10

---

[5] SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2016.
[6] Bor-Chun Chen and Andrew Kae. Toward realistic image compositing with adversarial learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
[7] Vladimir V Kniaz, Vladimir Knyaz, and Fabio Remondino. The point where reality meets fantasy: Mixed adversarial generators for image splice detection. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2019.
[8] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[9] Chen-Hsuan Lin, Ersin Yumer, Oliver Wang, Eli Shechtman, and Simon Lucey, St-gan: Spatial transformer generative adversarial networks for image compositing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[10] Jianwei Yang, Anitha Kannan, Dhruv Batra, and Devi Parikh. Lr-gan: Layered recursive gener- ative adversarial networks for image generation. In Proceedings of International Conference Learning Representations (ICLR), 2017.
[11] Adam Bielski and Paolo Favaro. Emergence of object segmentation in perturbed generative models. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2019.
[12] Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. Labels4free: Unsupervised segmentation using stylegan. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2021.
[13] Yunji Kim and Jung-Woo Ha. Contrastive fine-grained class clustering via generative adversarial networks. In Proceedings of International Conference on Learning Representations (ICLR), 2022.
[14] Yanchao Yang. Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto. Unsupervised moving object detection via contextual information separation. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[15] Yanchao Yang, Brian Lai, and Stefano Soatto. Dystab: Unsupervised object segmentation via dynamic-static bootstrapping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[16] Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In Proceedings of International Conference on Machine Learning (ICML), 2020.
[17] Luke Melas- Kariazi,,,azi,, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Finding an unsuper- vised image segmenter in each of your deep generative models. arXiv preprint arXiv:2105.08127, 2021.
[18] Pavel Ostyakov, Roman Suvorov, Elizaveta Logacheva, Oleg Khomenko, and Sergey I Nikolenko. Seigan: Towards compositional image generation by simultaneously learning to segment, enhance, and inpaint. arXiv preprint arXiv:/811.07630 2018.
[19] Axel Sauer and Andreas Geiger. Counterfactual generative networks. In Proceedings of International Conference on Learning Representations (ICLR), 2021.
[20] Learning foreground--ackgroundd segmentation from improved layered gans. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), 2022.
[21] Yaniv Benny and Lior Wolf. Onegan: Simultaneous unsupervised learning of conditional image generation, foreground segmentation, and fine-grained clustering. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
[22] Sébastien Ehrhardt, Oliver Groth, Aron Monszpart, Martin Engeleke, Ingmar Posner, Niloy Mitra, and Andrea Vedaldi. Relate: Physically plausible multi-object scene synthesis using structured latent spaces. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.
11

---

[23] Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas Geiger. Towards unsupervised learning of generative models for 3d controllable image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[24] Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar R Zaiane, and Martin Jagersand. U2-net: Going deeper with nested u-structure for salient object detection. Pattern Recognition, 107404, 2020.
[25] Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. Finegan: Unsupervised hierarchical disentanglement for fine-grained object generation and discovery. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[26] Utkarsh Ojha, Krishna Kumar Singh, and Yong Jae Lee. Generating furry cars: Disentangling object shape & appearance across multiple domains. In Proceedings of International Conference on Learning Representations (ICLR), 2021.
[27] Tianyi Chen, Yi Liu, Yunfei Zhang. Si Wu, Yong Xu, Feng Liangbing, and Hau San Wong. Semi-supervised single-stage controllable gans for conditional fine-grained image generation. In Proceedings of the IEFE International Conference on Computer Vision (ICCV), 2021.
[28] Li Niu, Wenyan Cong, Liu Liu, Yan Hong. Bo Zhang, Jing Liang, and Liqing Zhang. Making images real again: A comprehensive survey on deep image composition. arXiv preprint arXiv:2106. 14490, 2021.
[29] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of Neural Information Processing Systems (NeurIPS), 2016.
[30] Yuheng Li, Krishna Kumar Singh, Yang Xue, and Yong Jae Lee. Partgan: Weakly-supervised part decomposition for image generation and segmentation. In Proceedings of British Machine Vision Conference (BMVC), 2021.
[31] Yuheng Li, Krishna Kumnr Singh, Utkarsh Ojha, and Yong Jae Lee. Mixnmatch: Multifactor disentanglement and encoding for conditional image generation. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[32] Zhixin Shu, Mihir Sahasrabudhe, Riza Alp Guler, Dimitris Samaras, Nikos Paragios, and lasonas Kokkinos. Deforming autoencoders: Unsupervised disentangling of shape and appearance. In Proceedings of European Conference on Computer Vision (ECCV), 2018.
[33] Patrick Esser, Johannes Haux, and Bjorn Ommer. Unsupervised robust disentangling of latent characteristics for image synthesis. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
[34] Xide Xia and Brian Kulis. W-net: A deep model for fully unsupervised image segmentation. arXiv preprini (tt::/7/1.080000 2017.
[35] Mickaël Chen, Thierry Artières, and Ludovic Denoyer. Unsupervised object segmentation by redrawing. In Proceedings of Advances Neural Information Processing Systems (NeurIPS). 2019.
[36] Pedro Savarese, Sunnie SY Kim, Michael Maire, Greg Shakhnarovich, and David McAllester. Informattin-ttheortiic segmentation by inpainting error maximization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR),
[37] Zinan Lin, Kiran Thekumparampil. Giulia Fanti, and Sewoong Oh. Infogan-cr and modelcen- trality: Self-supervised model training and selection for disentangling gans. In Proceedings of International Conference on Machine Learning (ICML), 2020.
[38] Mohammadreza Armandpour, Ali Sadeghian, Chunyuan Li, and Mingyuan Zhou Partition- guided gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2021.
[39] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD Birds-200-2011 Dataset, 2011.
[40] Si Wu, Guangchang Deng, Jichang Li, Rui Li, Zhiwen Yu, and Hau-San Wong. Enhancing triplegan for supi-supe conditional instance synthesis and classification. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
12

---

[41] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshops, 2013.
[42] Maria-Elem Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Proceedings of Indian Conference on Computer Vision, Graphics & Image Processing (ICVGIP), 2008.
[43] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[44] Peiyu Yu, Sirui Xie, Xiaojian Ma, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Unsupervised foreground extraction via deep region competition. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2021.
[45] Hong-Wei Ng and Stefan Winkler. A data-driven approach to cleaning large face datasets. In Proceedings of international conference on image processing (ICIP), 2014.
[46] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. In Proceedings of Neural Information Processing Systems (NeurIPS), 2017.
[47] Lee Raymond Dice. Mcasures of the amount of ecologic association between species. Ecology, 26:297-302, 1945.
[48] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[49] Chongxuan Li, Taufik Xu, Jun Zhu, and Bo Zhang. Triple generative adversarial nets. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2017.
[50] Zhe Gan, Liqun Chen, Weiyao Wang, Yuchen Pu, Yizhe Zhang, Hao Liu, Chunyuan Li, and Lawrence Carin. Triangle generative adversarial networks. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2017.
[51] Yi Liu, Guangchang Deng, Xiangping Zeng, Si Wu, Zhiwen Yu, and Hau-San Wong. Regulariz- ing discriminative capability of egans for semi-supervised generative learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[52] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In Proceedings of International Conference on Learning Representations (ICLR), 2018.
[53] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. "grabcut" interactive foreground extraction using iterated graph cuts. ACM transactions on graphics (TOG), 23:309-314 14, 2004.
[54] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object represen- tation learning with iterative variational inference. In Proceedings of International Conference on Machine Learning (ICML), 2019.
[55] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Proceedings of Neural Information Processing Systems (NeurIPS), 2020.
[56] Guanbin Li and Yizhou Yu. Visual saliency detection based on multiscale deep enn features.
IEEE Transactions on Image Processing (TIP), 25:5012-551 -0244, 2016.
[57] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
[58] Tsung- Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. Proceedings of European Conference on Computer Vision (ECCV), 2014.
[59] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition (CVPR), 2016.
13

---

[60] Francisco Massa Wan- Yen Lo Yuxin Wu, Alexander Kirillov and Ross Girshick. Detectron2.
https://ge gi thub. www.aaaccoorreeeeaeeeeeettttteee
[61] Maria-Elena Nilsback and Andrew Zisserman. Delving into the whorl of flower segmentation. In Proceedings of British Machine Vision Conference (BMVC), 2007.
[62] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[63] Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversar- ial networks. Proceedings of International Conference on Learning Representations (ICLR), 2017.
[64] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings of International Conference Machine Learning (ICML), 2017.
14


Reviews:
[{"review_id": "gOR6FOs7E-s", "reviewer": "Reviewer_Tkuw", "summary": "The paper considers the task of learning GANs that decompose the image formation process into foreground, background and mask generation and composition. Compared to previous methods, the proposed ComGAN aims to avoid trivial solutions (where masks do not correspond to foreground objects) mainly through the network architecture instead of regularizations, which often require extensive hyperparameter searches for suitable regularization strengths.", "questions": "- l. 47-60 could be moved to the related work section\n- l. 294 refers to the wrong figure\n- l. 298: how does fine-tuning mitigate masks that are inconsistent with foreground images? Also, what is the effect of the mask distribution alignment (l. 206)? Why not train the segmentation model S with Eq. (13) only?\n- It would be helpful to use the same evaluation protocol as in FineGAN (based on 30k samples) for Tab. 3. The reported Inception Scores seem very low.\n- See also weaknesses for questions", "limitations": "Limitations and potential negative societal impact have been addressed adequately.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n    - The proposed approach simplifies the design of compositional GANs compared to previous methods and demonstrates improved performance in terms of synthesis quality as well as unsupervised segmentation performance.\n    - Compared to similar compositional generative models like FineGAN [25], the supervision requirements regarding weak background supervision is further reduced.\n    - Finding a compositional generator architecture that is stable to train has many applications beyond GANs. Thus, the work is potentially interesting for a larger audience.\n    - Although there are still a few hyperparameters (mask consistency loss weight, binary regularization weight, dimensionality of latents, relative sizes of subnetworks), experiments demonstrate some robustness to these parameters.\n- Weaknesses\n    - The core idea and differences to previous approaches are not clearly stated.\n    - The requirements for Proposition 1 regarding what it means for an architecture to be \"similar to ComGAN\" are not stated clearly. I assume the key point is that M consists of only an sigmoid layer. However, the formulation in l. 149 seems to be the only place where this is stated and even there it remains vague and could be interpreted as containing the same layers as F and B and in addition a sigmoid layer. The latter interpretation is also what Fig. 3 suggests (albeit with one residual block less). If this (a shared decoder with a minimal mask decoder) is the key idea of the paper, it should be communicated more clearly and probably also on a higher level already in the introduction. Without clear restrictions on the architecture, one could also think that FineGAN satisfies the requirements with G set to the identity.\n    - The lemmas, propositions and proofs seem a bit vague as they do not clearly state assumptions or define all involved quantities. In l. 160-161 it is not clear what is meant by $\\bar{x}_m = M^{-1}(G(z))$ - why would the input of M equal its output? Intuitively, I also don't see how \"it is clear that any change of foreground or background affects the mask, [...]\". Since F and B do contain additional layers, foreground and background could be affected by changes in those layers even though G(z) and hence the mask would remain unaffected, no? The other way, that any change of the mask affects both foreground and background, seems to be true.\n    - Motivation for DS-ComGAN architecture is unclear: Why is the $\\bar{x}_z$ output needed in addition to the output composited from foreground, background and mask?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O0FJYU9yOI8", "reviewer": "Reviewer_19KW", "summary": "The authors point out two factors that lead a scene decomposition model to fall into trivial solutions in a mathematical analysis.\nThose are related to the vanishing gradient phenomena that goes into a mask generator.\nTo avoid these, they propose a novel network architecture, where features for generating decomposed scene elements are composed of the ones that used for generating the entire scene at once. \nWith this architecture, they achieved the SOTA scores on both the mask prediction and the image quality evaluation metrics.", "questions": "Questions\n- Why the proposed network is helpful for avoiding the vanishing gradient issue?\n- Why the performance of PerturbGAN was not compared?\n\nSuggestions\n- Maybe simplegan —> finegan at the line 221?", "limitations": "Authors did not address the limitations and potential negative societal impact of their work.\n\nSuggestions\n- I want to know the authors’ opinion on how much would it be difficult to apply this method on coarse grained datasets like ImageNet?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The authors tackle the important problem in the scene component generation models.\n- They propose a novel architecture for robust mask generation based on the theoretical analysis on the problem.\n- The authors did a thorough ablation study to show that each element proposed are all effective.\n- The comparison of both quantitative and qualitative results with previous works imply that the proposed method is effective in boosting the scene decomposition performance.\n\nWeaknesses\n- The connection between the theoretical analysis and the proposed architecture design is not well established. More details are needed to understand why the proposed architecture is helpful for model to not fall into the vanishing gradient phenomena.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MT0a2UoUuy9", "reviewer": "Reviewer_rWMN", "summary": "This work analyses the reason for trivial solutions during mask learning in image composition GANs, and introduce a new model architecture ComGAN to solve the trivial solution issue.  Furthermore, an unsupervised object segmentation module is also involved to construct the DS-ComGAN model. DS-ComGAN can perform both disentangled image generation and object segmentation, and outperforms semi-supervised and weakly supervised baselines.", "questions": "The presentation of the method part is not clear enough. Please check the questions in the weaknesses part.", "limitations": "The limitations and potential negative societal impact have been well described.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- It is claimed that this work is the first to solve the trivial solution in disentangled image generation by changing the network architecture. The change is simple to apply and obtained significant improvement. I believe this technic can also be useful for other models and tasks.\n\n- Both disentangled image generation and object segmentation tasks can be performed in a single framework. More importantly, the learning can be achieved in an unsupervised way by a carefully designed adversarial learning strategy. \n\n- Sufficient experiments and analyses have been done to demonstrate the effectiveness of the proposed method.\n\nWeaknesses\n\nMy main concerns are about the description.\n\n- The description in sub-section 3.1 is not that readable. I suggest improving the description with a more intuitive illustration to point out the key contribution: how to avoid vanishing gradient from the network architecture perspective. Besides, some symbols seem not consistent. Do the variable f in equation 9 and the variable F in Figure 3 represent the same thing? What does the variable mean?\n\n- The description of the mask distribution alignment could also be improved. If I understand correctly, the proposed Segmentation Networks S does not need any paired/unpaired segmentation data. For $D_m$, $\\bar{x}_m$ is regarded as the real and $\\hat{x}_m$ and $x_m$ are regarded as the fake.\n\n- The learning process is not clear. Does DS-ComGAN need to be trained in two stages?  Besides, the overall objective is missing and the loss term βLbinary is introduced in the experiment section instead of the method section.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept