Factorized Diffusion Architectures for Unsupervised Image Generation and Segmentation
Authors: Xin Yuan
Michael Maire
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion models
unsupervised learning
image segmentation
neural network architecture
Primary area: diffusion_based_models

Abstract: We develop a neural network architecture which, trained in an unsupervised manner as a denoising diffusion model, simultaneously learns to both generate and segment images.  Learning is driven entirely by the denoising diffusion objective, without any annotation or prior knowledge about regions during training.  A computational bottleneck, built into the neural architecture, encourages the denoising network to partition an input into regions, denoise them in parallel, and combine the results.  Our trained model generates both synthetic images and, by simple examination of its internal predicted partitions, semantic segmentations of those images.  Without fine-tuning, we directly apply our unsupervised model to the downstream task of segmenting real images via noising and subsequently denoising them.  Experiments demonstrate that our model achieves accurate unsupervised image segmentation and high-quality synthetic image generation across multiple datasets.

Factorized Diffusion Architectures for
Unsupervised Image Generation and Segmentation
Xin Yuan
Michael Maire Univeerstyy of Chicago mmaire@uchicago. edu
Google yuanxzz28oogle..co
Abstract
We develop a neural network architecture which, trained in an unsupervised manner as a denoising diffusion model, simultaneously learns to both generate and segment images. Learning is driven entirely by the denoising diffusion objective, without any annotation or prior knowledge about regions during training. A computational bottleneck, built into the neural architecture, encourages the denoising network partition an input into regions, denoise them in parallel, and combine the results. Our trained model generates both synthetic images and, by simple examination of its internal predicted partitions, semantic segmentations of those images. Without fine-tuning, we directly apply unsupervised model to the downstream task of segmenting real images via noising and subsequently denoising them. Experiments demonstrate that our model achieves accurate unsupervised image segmentation and high-quality synthetic image generation across multiple datasets.
Introduction
Supervised deep learning yields powerful discriminative representations, and has fundamentally advanced many computer vision tasks, including image classification [13, 58, 21, 28), object detec- tion [18, 49.411 41], and semantic and instance segmentation [42, 22, 33]. Yet. annotation efforts [13], especially those involving fine-gra grained labeling for tasks such segmentation [39], can become prohibitively expensive to scale with increasing dataset size. This motivates an ongoing revolution in self-supervised methods for visual representation learning, which do not require any annotated data during a large-scale pre-training phase [7, 15, 35, 23, 10, 12). However, many of these approaches, including those in the particularly successful contrastive learning paradigm [23, 10, 12], still require supervised fine-tuning (e.g., linear probing) on labeled data to adapt networks to downstream tasks
such classification [23, 10] segmentation 69].
In parallel with the development of self-supervised deep learning, rapid progress on a variety of frameworks for deep generative models [32, 19, 65, 66, 60, 37, 27, 59, 50] has lead to new systems for high-quality image synthesis. This progress inspires efforts to explore representation learning within generative models, with recent results suggesting that image generation can serve as a good proxy task for capturing high-level semantic information, while also enabling realistic image synthesis.
Building upon generative adversarial networks (GANs) [19] or variational autoencoders (VAEs) [32]. InfoGAN [11] and Deep InfoMax [26] demonstrate that generative models can perform image classification without any supervision. PerturbGAN [4] focuses on more complex task, unsupervised image segmentation, by forcing an encoder to map an image to the input of a pre-trained generator so that it synthesizes composite image that matches the original input image. However, here training is conducted in two stages and mask generation relies on knowledge predefined object classes.
*This work was completed while Xin Yuan was a PhD student at the University Chicago.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Revese
Revere
Forwert (Malfa)
Nolyy
(a) Simultaneous Image and Region Generation
(b) Segmentation of Novel Input Image
(c) Generated Images
(d) Generated Regions
(e) Real Images
(f) Segmentations
Figure Unifying image generation and segmentation. (a) We design a denoising diffusion model with a specific architecture that couples region prediction with spatially-masker diffusion over predicted regions, thereby generating both simultaneously. (b) An additional byproduct of running our trained denoising model on an arbitrary input image is a segmentation of that image. Using achieve both high quality synthesis of images and corresponding
model trained on FFHQ [31], semantic segmentations (c-d), as well as the ability to accurately segment images of real faces (e-f).
Segmenting real image is fast, requiring only one forward pass (one denoising step).
Denoising diffusion probabilistic models (DDPMs) [27] also achieve impressive performance in generating realistic images. DatasetDDPM [2] investigates the intermediate activations from the pre-trained U-Net [51] network that approximates the Markov step of the reverse diffusion process in DDPM, and proposes a simple semantic segmentation pipeline fine-tuned on few labeled images. In spite of this usage of labels, DatasetDDPM demonstrates that high-level semantic information, which is valuable for downstream vision tasks, can be extracted from pre-trained DDPM U-Net. Diff- AE [48] and PADE [70] are recently proposed methods for representation learning by reconstructing images in the DDPM framework. However, their learned representations are in the form of a latent vector containing information applicable for image classification.
In contrast to all of these methods, we demonstrate a fundamentally new paradigm for unsupervised visual representation learning with generative models: constrain the architecture of the model with structured bottleneck that provides an interpretable view of the generation process, and from which simply read off desired latent information. This structured bottleneck does exist in isolation, but rather is co-designed alongside the network architecture preceding and following it. The computational layout of these pieces must work together in manner that forces the network. when trained from scratch for generation alone, to populate the bottleneck data structure with an interpretable visual representation.
We demonstrate this concept in the scenario of DDPM for image generation and semantic segmenta- tion the interpretable representation be read from the bottleneck. Thus, frame unsupervised image segmentation and generation in a unified system. Moreover, experiments demonstrate that domain-specific bottleneck design not only allows free, but also boosts the quality of generated samples. This challenges the assumption that generic architectures (e.g., Transformers [61]) alone suffice; find synergy by organizing such generic building blocks into factorized architecture which generates different image regions in parallel.
accomplish end task (segmentation) for
Figure 1 provides an overview of our setting alongside example results, while Figure 2 illustrates details of our DDPM architecture which are fully presented in Section 3. This architecture constrains the computational resources available for denoising in manner that encourages learning of a factorized model the data. Specifically, each step of the DDPM has the ability to utilize additional inference passes through multiple copies of subnetwork if it is willing decompose the denoising task into parallel subproblems. The specific decomposition strategy itself must be learned, but, by design, is structured in manner that reveals the solution to target task of image segmentation. We summarize our contributions as three-fold:

---

m
X
Benc
mg
Decodet
h1
Encoder
Itenc
Middle
Sample
Block
Real Image
Noisy Image
Commection
Aimid
Rguid
Share Weights
MSE
Mask
m
Eg Predicted
Loss
Component Factorization
Generator
Decoder
Riter
Softmax
Skip
Region
Element-wise Multiplication
Masks
Parallel Decoding
Element-wise Addition
Figure 2: Factorized diffusion architecture. Our framework restructures the architecture of the neural network within DDPM [27] so as decompose the image denoising task into parallel subtasks. All modules are end-to-end trainable and optimized according to the same denoising objective DDPM. Left: Component factorization. An Encoder, equivalent to the first half of standard DDPM U-Net architecture, extracts features hene- A common Middle Block processes Encoder output
into shared latent features hmidd Note that Middle Block and hmnd exist the standard denoising DDPM U-Net by default. We draw it as standalone module for a better illustration of the detailed architectural design. A Mask Generator, structured as the second half of a standard U-Net receives hmidd as input, alongside all encoder features hene injected via skip connections to layers of corresponding resolution. This later network produces a soft classification of every pixel into one of K region masks, ma. m...... mk. Right: Parallel decoding. A Decoder, also structured as the second half of a standard U-Net, runs separately for each region. Each instance of the Decoder receives shared features hidd and masked view of encoder features hene my injected via skip connections to corresponding layers. Decoder outputs are masked prior to combination. Though not pictured, inject timestep embedding into the Encoder, Mask Generator, and Decoder.
Unified learning of generation and segmentation. We train our new DDPM architecture once, obtaining model directly applicable two different tasks with zero modification or fine-tuning: image generation and image segmentation. Segmenting novel input image is fast, comparuble in speed any system using a single forward pass of U-Net [51] like architecture.
Unsupervised segmentation for free. Our method automatically learns meaningful regions (e.g.. foreground and background), guided only by the DDPM denoising objective; no extra regularization terms, no use of labels.
Higher quality image synthesis. Our model generates higher-quality images than the baseline DDPM, as well as their corresponding segmentations simultaneously. We achieve excellent quantitative and qualitative results under common evaluation protocols (Section 4).
Beyond improvements image generation and segmentation, our work is case study of new paradigm for using generation as a learning objective, combination with model architecture as a constraint. Rather than viewing a pre-trained generative model source from which to extract and repurpose features for downstream tasks, design the model architecture the first place so that, as a byproduct of training from scratch to generate, it also learns to perform the desired task.
Related Work
Image Segmentation. Generic segmentation, which seeks to partition an image into meaningful regions without prior knowledge about object categories present in the scene, is longstanding challenge for computer vision. Early methods rely on combinations of hand-crafted features based on intensity, color, and texture cues [6, 44], clustering algorithms [57]. and duality between closed

---

contours and the regions they bound [1]. Deep learning modernized the feature representations used in these pipelines. yielding systems which, trained with supervision from annotated regions [43], reach near human-level accuracy on predicting and localizing region boundaries [3, 56, 64, 34].
Semantic segmentation, which assigns a category label each pixel location in image, has been similarly revolutionized by deep learning. Here, the development of specific architectures [42, 51, 20] enabled porting of approaches for image classification the task of semantic segmentation.
Recent research has refocused on the challenge of learning to segment without reliance on detailed annotation for training. Hwang et al. [29] combine two sequential clustering modules for both pixel-level and segment-level to perform this task. Ji et al. [30] and Ouali et al. [46] follow the concept of mutual information maximization to partition pixels into two segments. Savarese et al. [54] further propose a learning-free adversarial method from the information theoretic perspective, with the goal of minimizing predictability among different pixel subsets. Note that even completely unsupervised poreground/backgronn
segmentation is a non-trivial task. Liu et al. [40], a recent advance in this regime, produces similar region mask output, yet depends entirely upon motion cues from video for training. We achieve such unsupervised learning from static images alone.
Learning Segmentation in Generative Models. Previous generative model-based approaches learn semantic segmentation by perturbing [4] or redrawing [9] generated foreground and background masks. Despite good performance, these methods apply only two-class partitions and require extra loss terms based upon object priors in training datasets.
Denoising diffusion probabilistic models (DDPMs) [27] achieve state-of-the-art performance in generating realistic images. Their noise schedule in training may offer advantages for scaling up models in a stable manner. Recent works [2, 48, 70] explore representation learning capability in DDPMs. DaMM [2] examines few-shot segmentation with pre-trained diffusion models, but requires human labels to train linear classifier. With the default U-Net architecture [51], it loses the efficiency and flexibility of generating image and masks a single-stage manner. Diff-AE [48] and PADE [70] perform representation learning driven by reconstruction objective in the DDPM frame- work. Unfortunately, their learned latent vectors are not applicable to more challenging segmentation tasks and they require a pre-trained interpreter to perform downstream image classification.
DiffuMask [63] takes a pre-trained Stable Diffusion model [50], which is built using large-scale text-to-image datasets (and thus solves a far less challenging problem), and conducts post-hoc investigation on how to extract segmentation from its attention maps. Neither our system, nor the baseline DDPM to which we compare, makes use of such additional information. Furthermore, DiffuMask does not directly output segmentation; it is basically dataset generator, which produces generated images and pseudo labels, which are subsequently used to train a separate segmentation model. Our method, in contrast, is both completely unsupervised and provides an end-to-end solution by specifying an architectural design in which training to generate reveals segmentations as a bonus.
MAGE [38] shares with us a similar motivation of framing generation and representation learning in unified framework. However, our approach is distinct in terms of both (I) task: we tackle a more complex unsupervised segmentation task (without fine-tuning) instead of image classification (with downstream fine-tuning), and (2) design: 'masks' play fundamentally different role in our system. MAGE adopts an MAE [24]-like masking scheme on input data, in order to provide a proxy reconstruction objective for self-supervised representation learning. Our use of region masks serves a different purpose, as they are integral components of the model being learned and facilitate factorization of the image generation process into parallel synthesis of different segments.
BlotGAN [16] is generative model for creating images with fine-grained control over the spatial arrangement of content. It leverages blob-like components instead of accurate region masks as basic building blocks for the synthesis process, allowing for intuitive content manipulation. In the generative modeling space, BlobGAN serves different purpose than our method: BlobGAN excels in scenarios requiring explicit spatial control and interactive editing, while our factorized diffusion approach provides framework for learning high-quality image generation and segmentation.
Factorized Diffusion Models
Figure 2 illustrates the overall architecture of our system, which partitions the denoising network within a diffusion model into an unsupervised region mask generator and parallel per-region decoders.

---

3.1 Unsupervised Region Factorization
To simultaneously learn representations for both image generation and unsupervised segmentation, first design the region mask generator based the first half (encoder) of standard DDPM U-Net. We obtain input 2t, a noised version zi., via forward diffusion:
(<1:)0)) == Ma v a0 a,)
3t v vā + vI āGC, ((0,
(I)
where 0₁ 1- Br, aj II Cty.
In addition to the encoder half of the U-Net, we instantiate a middle block consisting of layers operating on lower spatial resolution features. Parameterizing these subnetworks Oene and 0mids we extract latent representations:
hene 6 (2,, himid = miad (hene: t)
where hene encapsulates features at all internal layers fenc. for subsequent use as inputs, via skip connections, to corresponding layers of decoder-style networks (second half of standard U-Net),
We instantiate mask generator, maakk as one such decoder-style subnetwork. A softmax layer produces an output tensor with K channels, representing K different regions in image To.
m, = mmakk (hmids henes!! 1)
Following a U-Net architecture, hene feeds into Omask through skip-connections.
3.2 Parallel Decoding Through Weight Sharing
We aim to extend a standard DDPM U-Net decoder Ode to consider region structure during generation. One simple design is to condition hene along the channel dimension:
{mo, m1, ...) by concatenating it with input hyaid and
dee (concat(hmidd ml), concat[hene: mT),
(5)
where himid and hene are generated from Eq. 2 and Eq. 3. We downsample m accordingly to the same resolution as hmid and hene at different stages. However, such a design significantly modifies (e.g., channel sizes) the original U-Net decoder architecture. Moreover, conditioning with the whole mask representation may also result a trivial solution that simply ignores region masks.
To address these issues, we separate the decoding scheme into multiple parallel branches of weight- shared U-Net decoders, each masked by single segment. Noise prediction for k-th branch is:
č₁ = (hmid hene ① m,g
(6)
and the output is sum region-masked predictions:
é Ek O mm
(7)
k=0
3.3 Optimization with Denoising Objective
We train our model an end-to-end manner, driven by the simple DDPM denoising objective. Model
weights 0 = {Beney Gmid. O dens G mask } are optimized by minimizing the noise prediction loss:
L = E E
(8)
Unlike previous work, our method does not require mask regularization loss term [54, 4, 9], which predefines mask priors (e.g., object size). Algorithm summarizes training.
5

---

3.4 Segmentation via Reverse Diffusion
Once trained, our model can both segment novel input images and synthesize images from noise.
Real Image Segmentation. Given clean input image To, we first sample a noisy version 2, through forward diffusion in Eq. 1. We then perform one-step denoising by passing æ, the model. We collect the predicted region masks as the segmentation for Do using Eq. 4.
Image and Mask Generation. Using reverse diffusion, our model can generate realistic images and their corresponding segmentation masks, starting from a pure noise input æT ~ (0, 1). Reverse diffusion predicts - from Zt..
1- 0(z,, t)) + 012, vI at x (00, if
It-1 1/ Vadaa
(9)
t>1 else = O.
(10)
where is empirically set according to the DDPM noise scheduler. We perform 7 steps of reverse diffusion generate an image. We also collect its corresponding masks using Eq. 4 when 1. Algorithm 2 summarizes this process.
Algorithm 1
Algorithm 2
Training Masked Diffusion
Image and Mask Generation
Input: Data æto
Input: Noise ær, trained model 0 Output:
Output: Trained model 0 Initialize:
Image d and segmentation mip
Model weights Timesteps for iter = 1 to Iter do
Initialize: ær (O, 1)
1 to do
Sample E [1, 7
Sample using Eq. 10
Sample using Eq. 1
Perform reverse diffusion using Eq. 9
Calculate è using Eq.
ift= then
Backprop with Eq.
Collect rigo using Eq. 4
Update end for
Return Zo and rio
end if
return 0
end for
4 Experiments
We evaluate on: (1) real image segmentation, (2) image and region mask generation, using Flower [45], CUB [62], FFHQ [31], CelebAMask HQ [36], and ImageNet [53]. addition the design of flat set of K regions, we also conduct a preliminary investigation into reorganizing our architectural design to support hierarchical segmentations; see Section A.I.
Evaluation Metrics. For unsupervised segmentation on Flower and CUB, we follow the data splitting in IEM [54] and evaluate predicted mask quality under three commonly used metrics, denoted as Acc., IOU and DICE score [54, 9]. Acc. the (per-pixel) mean accuracy of the foreground prediction. IOU the predicted foreground region's 's intersection over union with the ground-truth foreground region. DICE score is defined as fruu" [14]. On ImageNet, we evaluate our method on Pixel-ImageNet [68], which provides human-labeled segmentation masks for 0.485M images covering 946 object classes. We report Acc., IOU and DICE score on randomly sampled subset, each class containing at most 20 images. For face datasets, we train our model on FFHQ and only report per-pixel accuracy on the CelcbAMask test set, using provided ground-truth.
For image and mask generation, we use Fréchet Inception Distance (FID) [25] for generation quality assessment. Since we can not obtain the ground-trute -oound-rruth for generated masks, we apply a supervised respective datasets, the generated images and measure
U-Net segmentation model, pre- trained the consistency between masks in terms of per-pixel accuracy. In addition to quantitative comparisons, show extensive qualitative results.
Implementation Details. We train Flower, CUB and Face models both 64 64 and 128 x 128 resolution. We also train class-conditioned ImageNet models with 64 x 64 resolution. For all

---

Methods
Acc.
IOU
DICE
GrabCut [52]
82.0
69.2
79.1
ReDO [9]
87.9
76.4
IEM [54]
88.3
76.8
84.6
IEM+SegNer [54]
89.6
78.9
86.0
Ours
90.1
79.7
87.2
(a) Real Images
(b) Segmentation
Table 1: Comparisons Flower.
Figure 3: Segmentation on Flower.
Methods
Acc.
IOU
DICE
GrabCut [52]
72.3
36.0
48.7
PerturbGAN [4]
38.0
ReDO [9]
84.5
42.6
IEM [54]
88.6
52.2
66.0
IEM+SegNet [54]
89.3
55.1
68.7
Ours
89.6
56.1
69.4
Images
(h) Segmentation
Table 2: Comparisons on CUB.
Figure 4: Segmentation on CUB.
experiments, we use the U-Net [51] secooder-iddle-ddecooee architecture similar [27]. We use the decoder architecture as our mask generator and set the number of factorized masks K as 3. We note that K the maximum number of regions the model may use. It could learn fewer components during training. For binary segmentation, we found setting K = 3 rather than K = 2 to assist training, with learned regions emerging as foreground, background, and contour or transition between the two. For segmentation evaluation, we simply select the mask channel that emerges foreground and apply standard benchmarks. For 64 x 64 the architecture is follows: The downsampling stack performs four steps of downsampling, each with 3 residual blocks. The upsampling stack setup as a mirror image of the downsampling stack. From highest to lowest resolution, U-Net stages use [C, 2C, 3C, 4C| channels, respectively. For 128 128 architecture, the down/up sampling 20,3C, 3C, 4C]] channels, each with two residual blocks, respectively. We set C' = 128 for all models.
block is 5-step with [C, C,
We use Adam to train all the models with learning rate of 10 and an exponential moving average (EMA) over model parameters with rate 0.9999. For all datasets except ImageNet, we train 64 x 64 and 128 128 models on and 32 Nvidia V100 32GB GPUs, respectively. For Flower, CUB and FFHQ. we train the models for 50K, 50K, 500K iterations with batch size of 128, respectively. For ImageNet, we train 500K iterations on 32 Nvidia V100 GPUs with batch size 512. We adopt the linear noise scheduler as in Ho et al. [27] with T 1000 timesteps.
4.1 Image Segmentation
To evaluate our method on real image segmentation, we as 30 for the forward diffusion process. We also investigate the segmentation results with different noise levels in Figure 18. For Flower and CUB, Figures 3 and 4 show test images and predicted segmentations. Tables and provide quantitative comparison with representative unsupervised image segmentation methods: GrabCut [52], ReDO [9] and IEM [54]. As shown in Table and Table 2, our method outperforms competitors.
We also visualize the predicted face parsing results on FFHQ and CelebAMask datasets in Fig- ure (c))dd) and Figure 5, Our model learns accurately predict three segments corresponding to semantic components: skin, hair, and background. This particular semantic partitioning emerges from our unsupervised learning objective, without any additional prior. With ground-truth provided on SclebAMask-HHO we also compare the pixel accuracy and mean IOU with a supervised U-Net and DatasetDDPM [2]. For the former, we train supervised segmentation model with 3-class cross-entropy loss. For the unsupervised setting, we perform K-means (K=3) on the pre-trained DDPM, denoted as DatasetDDPM-unm -unuuuu Table shows that we outperform DatasetDDPM by large margin and achieve relatively small performance gap with a supervised U-Net.
7

---

Methods
Acc.
mIOU
Supervised UNet
95.7
90.2
DataserDDDM-unsupp [2]
78.5
69.3
Ours
87.9
80.3
Table 3: Seg. comparisons on CelebA
(a) Real Images
(b) Segmentation
Figure 5: Segmentation on CelebA.
Methods
Acc.
mlOU
Supervised UNet
85.7
74.1
DatasecDDPM-unsup [2]
74.1
60.4
Ours
80.7
67.7
Table 4: Seg. comparisons ImageNet.
Images
(h) Segmentation
Figure 6: Segmentation on ImageNet.
Figure shows the accurate segmentation results for ImageNet classes: ostrich, pekinese, papillon, and tabby. We compare with supervised U-Net and DataselDDPM-unsue in Table 4. We show more visualizations in the Appendix.
4.2 Image and Mask Generation
We evaluate our method on image and mask generation. As shown in Figure 7, 8. (cc)d) and 9, our method able generate realistic images. In the upper row of Table 5, we see consistent quality improvement over the original DDPM. This suggests that our method is a better generation architecture than the standard U-Net; separate computational paths for denoising individual image regions is beneficial prior to impose when learning to model the image distribution. Additionally, our method produces accurate corresponding masks, closely aligned with the semantic partitions in the generated image.
We also evaluate the quality of these segmentations. Since there is no ground-truie mask provided for generated images, we apply the U-Net segmentation models (pre-trained respective labeled training sets) to the generated images to produce reference masks. We measure the consistency between the reference and the predicted parsing results in terms of pixel-wise accuracy. We compare our method with a pre-trained DDPM baseline, which we first perform image generation, then pass them to DatasetDDPM-unn achieves better segmentation on generated images than the DDPM baseline. Note that, different from the two-stage baseline, our method performs the computation in a single stage, generating image and mask simultaneously. The Appendix shows more visualizations.
to get masks. As shown in Table 5 (bottom), our method consistently
(a) Generated Images
(b) Generated Masks
(a) Generated Images
(b) Generated Masks
Figure 7: Generation on Flower.
Figure 8: Generation on CUB.

---

Table 5: Image and mask generation comparison on all datasets (top: FID(J) bottom: Acc. (1)).
Models Flower-64
Flower- 128
CUB-64
CUB-128
FFHQ-64
FFHQ-128
ImageNet-64
DDPM
15.81
14.62
14.45
14.01
13.72
13.35
7.02
Ours 13.33
11.50
10.91
10.28
12.02
10.79
6.54
DDPM
80.5
82.9
84.2
83.7
84.2
84.2
71.2
Ours
92.3
92.7
91.4
91.2
90.3
90.7
84.1
Methods
1OU.(1)
FID (4)
Concat
20.7
14.21
Masking Amid
20.2
14.33
w/o weight sharing
50.5
17.21
Ours
56.1
10.28
Table 6: Ablations of decoding scheme on CUB.
(a) Generated Images
(b) Generated Masks
Figure 9: Conditional generation ImageNet.
4.3 Ablation Study and Analysis
Multi-branch Decoders with Weight Sharing. Separating computation in multi-branch decoders with weight sharing is an essential design in our method. We show the effectiveness of this design by varying how to apply factorized masks in our decoding scheme: (1) concat: we use single branch to take concatenation of h and m. (2) masking hmidd we use m to mask hwid instead of hene- (3) w/o weight sharing: we train decoders separately in our design. Table shows separate design consistently yields better visual features than other designs for CUB. This suggests that our design benefits from fully utilizing mask information in the end-to-end denoising task and avoids a trivial solution where masks are simply ignored.
Investigation Mask Factorization. each representing a particular segment of the input image. We show this by visualizing the individual channels from softmax layer output in our mask generator. As shown in Figure 10, skin, hair, and background are separated in different channels.
Our architecture is able generate factorized representations,
Mask Refinement along Diffusion Process. In the DDPM Markov process, the model implicitly formulates a mapping between noise and data distributions. We validate that this occurs for both images and latent region masks by visualizing image and mask generation along the sequential reverse diffusion process Figure 11. We observe gradual refinement as denoising steps approach f 0.
Zero-shot Object Segmentation. We evaluate zero-shot object segmentation on both PASCAL VOC 2012 [17] and DAVIS-2011 videos [47]. Baseline DDPM generation not solved for these datasets when training from scratch without external large-scale datasets (e.g.. LAION [55] used in Stable Diffusion [50]). We directly adopt zero-shot transfer from our pre-trained ImageNet model by applying the conditional label mapping to VOC. Table 7 details the mapping rule. Figure 13 shows the accurate segmentation results for images of classes: aeroplane, monitor, person, and sofa from VOC. Since our method does not require any pixel labels, evaluate the performance of
Real Image
mg
= 600
500
400
300
200
100
t=
Timestep
Figure 10: Mask factorization (3 parts) on FFHQ.
Figure 11: Gen. refinement along diffusion.

---

Source R A 0.1 == 0.3 == 0.4 0
0.6 == 2 0.8 2 0.9 Rec.
Source B
Source
Rec. A
A 0.1 0.3 A= 0.4 = 0.5 A 0.6 A 0.7 = 0.8 == 0.9 Rec.
Source B
Figure 12: Interpolations on FFHQ with 250 timesteps diffusion.
Frames of 'Bear'
(a) Real Images
(b) Segmentation
(b) Frames 'Dog'
Figure 13: Segmentation on VOC-2012.
Figure 14: Segmentation on DAVIS-I1 17.
each object class individually. We report pixel accuracy and mIOU of each class in VOC in Table 7, which demonstrates that our method can achieve reasonably high performance. Our method achieves an accuracy of 0.78 and mIOU of 0.54 when averaging over all 20 classes. We also show video segmentation on DAVIS-2017 in Figure 14 and the Appendix, without any labeled video pre-training.
Face Interpolation. We also investigate face interpolation on FFHQ. Similar to standard DDPM [27], perform the interpolation in the denoising latent space with 250 timesteps of diffusion. Figure 12 shows good reconstruction in both pixels and region masks, yielding smoothly varying interpolations across face attributes such as pose, skin, hair, expression, and background.
Conclusion
We propose a factorized architecture for diffusion models that is able to perform unsupervised image segmentation and generation simultancously, while being trained once, from scratch, for image generation via denoising alone. Using model architecture component factorization and parallel decoding schemes, our method effectively and efficiently bridges these two challenging tasks in a unified framework, without the need for fine-tuning altering the original DDPM training objective. Our work the first example of engineering an architectural bottleneck so that learning desired end task becomes necessary byproduct of training generate.
constraint, via carefully designed
Our work at the stage of a new architectural design for diffusion-based segmentation and generation, with 2- or 3-class segmentation results demonstrating improvements across multiple datasets, scaling up to ImageNet. Our initial investigation into hierarchical extensions suggests a promising future path towards handling complex scenes.
10

---

References
[1] Pablo Arbeláez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. PAMI, 2011.
[2] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. ICLR, 2022.
[3] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. DeepEdge: A multi-scale bifurcated deep network for top-down contour detection. In CVPR, 2015.
[4] Adam Bielski and Paolo Favaro. Emergence of object segmentation in perturbed generative models. In NeurIPS, 2019.
Chris Burgess and Hyunjik Kim. 3D shapes dataset. http://iiiibbbboreeeeeeeeeeeeiis 2018.
[6] John Canny. A computational approach to edge detection. PAMI, 1986.
[7] Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre- pre-training of image features on non-curated data. In ICCV, 2019.
[8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, In ICCV, 2021.
[9] Mickaël Chen, Thierry Artières, and Ludovic Denoyer. Unsupervised object segmentation by redrawing. In NeurlPS, 2019.
[10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In /CML, 2020.
[11] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In NeurlPS, 2016.
[12] Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. 2xXx::2033.4229,, 2020.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. ImageNet: A large-scale hicrarchical image database. CVPR, 2009.
[14] Lee Raymond Dice. Measures of the amount of ecologic association between species. Ecology, 1945.
[15] Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
[16] Dave Epstein, Taesung Park, Richard Zhang, Eli Shechtman, and Alexei A. Efros. BlobGAN: Spatially disentangled scene representations. In ECCV, 2022.
[17] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012). http://hhss..robots.oou o...ottsooo.......o wwt.pa/aaaa//////oo2222..2
[18] Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
[19] Tan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
[20] Bharath Hariharan, Pablo Andrés Arbeláez, Ross B. Girshick, and Jitendra Malik. Hypercolumns for object segmentation and fine-grained localization. In CVPR, 2015.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[22] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV, 2017.
[23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsuper- vised visual representation learning. In CVPR, 2020.
[24] Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross B, Girshick. Masked autoencoders scalable vision learners. In CVPR, 2022.

---

[25] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a time-scale update rule converge to local Nash equilibrium. In NeurlPS, 2017.
[26] R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Manni Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. ICLR, 2019.
[27] Jonathan Ho, Ajay Jain, and Picter Abbecl. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[28] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[29] Jyh-Jing Hwang, Stella X. Yu, Jianho Shi, Maxwell D. Collins, Tien-Ju Yang, Xiao Zhang, and Liang-Chich Chen. SegSort: Segmentation by discriminative sorting of segments. In ICCV, 2019.
[30] Xu Ji, Andrea Vedaldi, and João F. Henriques. Invariant information clustering for unsupervised image classification and segmentation. In ICCV, 2019.
[31] Tero Karras, Samuli Laine, and Timo Aila. A style-hased generator architecture for generative adversarial networks. In CVPR, 2019.
[32] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
[33] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao. Spencer Whitehead, Alexander C. Berg. Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In ICCV, 2023.
[34] lasonas Kokkinos. Pushing the boundaries of boundary detection using deep learning. In JCLR, 2016.
[35] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as a proxy task for visual understanding. In CVPR, 2017.
[36] Cheng-Han Lce, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In CVPR, 2020.
[37] Ke Li and Jitendra Malik. Implicit maximum likelihood estimation. arXiv: 1809.0098909 2018
[38] Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. MAGE: Masked generative encoder unify representation learning and image synthesis. In CVPR, 2023.
[39] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
[40] Runtao Liu, Zhirong Wu, Stella X. Yu, and Stephen Lin. The emergence of objectness: Learning zero-shot segmentation from videos. In NewrIPS, 2021.
[41] Wei Liu, Dragomir Anguelov. Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yan Fu, and Alexander C. Berg. SSD: Single shot multibox detector. In ECCV, 2016.
[42] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR, 2015.
[43] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ccological statistics. In ICCV, 2001.
[44] David Martin, Charless Fowlkes, and Jitendra Malik. Learning to detect natural image boundaries using local brightness, color and texture cues. PAMI, 2004.
[45] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008.
[46] Yassine Ouali, Céline Hudelot, and Myriam Tami, Autoregressive unsupervised image segmentation. In ECCV, 2020.
[47] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine- Hornung, and Luc Van Gool, The 2017 DAVIS challenge video object segmentation. arXi::/744000675, 2017.
[48] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion autoencoders: Toward a meaningful and decodable representation. In CVPR, 2022.
12

---

[49] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.
[50] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent difffusion models. In CVPR, 2022.
[51] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[52] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. "GrahCut": Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph., 2004.
[53] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bersttein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. IJCV, 2015.
[54] Pedro Savarese, Sunnie S. Y. Kim, Michael Maire, Greg Shakhnarovich, and David McAllester. Information- theoretic segmentation by inpainting emorr maximization. In CVPR, 2021.
[55] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kun- durthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. In NewrlPS, 2022.
[56] Wei Shen, Xinggang Wang. Yan Wang, Xiang Bai, and Zhijiang Zhang. DeepContour: A deep convolutional feature learned by positive-sharee sharing loss for contour detection. In CVPR, 2015.
[57] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. PAMI, 2000.
[58] Karen Simonyan and Andrew Zisserman, Very deep convolutional networks for large-scale image recogni- tion. In ICI.R, 2015.
[59] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[60] Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukeuoglu. Conditional image generation with PixeICNN decoders, In NeurlPS, 2016.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NewrlPS, 2017.
[62] C. Wah, S. Branson, P. Welinder, Perona, and Belongie. The Caltech- UCSD Birds-200-2011 Dataset. Technical Report MONS-TR-000 2011-001, Californa Institute of Technology, 2011.
[63] Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. DiffuMask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. ICCV, 2023.
[64] Saining Xie and Zhuowen Tu. Holistically-nester edge detection. In ICCV, 2015.
[65] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang. Zhe Gan, Xiaolei Huang. and Xiaodong He. AttnGAN: Fine-grained text to image generation with attentional generative adversarial networks. In CVPR, 2018.
[66] Han Zhang, Tao Xu, Hongsheng Li. Shaoting Zhang, Xiaogang Wang. Xiaolei Huang, and Dimitris Metaxas, StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In TCCV,2017
[67] Richard Zhang, Phillip Isola, and Alexci A. Efros. Colorful image colorization. In ECCV, 2016.
[68] Shiyin Zhang, Jun Hao Liew, Yunchao Wei, Shikui Wei, and Yao Zhao. Interactive object segmentation with inside-outside guidance. In CVPR, 2020.
[69]
Xiao Zhang and Michael Maire. Self-supervised visual representation leamning from hierarchical grouping. In NeurlPS, 2020.
[70] Zijian Zhang, Zhou Zhao, and Zhijie Lin. Unsupervised representation learning from pre-trained diffusion probabilistic models. In NeurlPS, 2022.
13

---

A Appendix
A.1 Hierarchical Factorized Diffusion
We conduct a further investigation to reorganize our architectural design to support hierarchical mask factorization in place of flat set of K regions. We formulate hierarchical factorized diffusion architecture to progressively refine segmentation results from a coarse initial prediction to fine, detailed final segmentation. This approach helps in capturing both global context and fine details in the segmentation task. As shown in Figure 15, the first level replicates the factorized diffusion architecture depicted in Figure 2 to generate initial region masks mß: ml. .... each applied on the noisy input for the next level factorized diffusion process, Each branch of the second level architecture generates finer representations of region masks mi, m constructing the final denoising output
as E. m. (R.+ + m)m)) This nested architectural design can be instantiated as repeated levels of factorized diffusion, which is promising way to handle multiscale scenes. As proof of concept, we experiment on the shape 3D dataset [5] with a 2-level hierarchy. We first visualize each level's region mask in Figure 16. We observe that the first level generates coarse segmentation, based on which, second-level factorized diffusion generates fine segmentations of 3D shapes. Figure provides more direct visualization of partitions at each level through a 3-class mapping.
1st Level
Ag
Parallel
Noisy Input
Decoding
Regions
Regions
112
Final Recon.
Recon.
Recon.
2nd Level
hg
(A Σ, kmm3)
Σ mg
2
h]
170
m
Figure 15: Hierarchical factorized diffusion architecture.
A.2 Additional Segmentation Results
We show more segmentation results for Flower, CUB, FFHQ, CelebA and ImageNet. As shown in Figures 19, 20, 21, 22, and 23, our method consistently predicts accurate segmentations for real image inputs.
A.3 Additional Generation Results
We show more generation results for Flower, CUB, FFHQ, and ImageNet (classes: flamingo, water buffalo, garbage truck, and sports car). As shown Figures 24, 25, 26, and 27, our method consistently produces images and masks with high quality.
14

---

Level
Level 2-1
Level 2-2
Level 2-3
Figure 16: Mask factorization for each level. Level 1: visualization of each mask channel at the first level. Level 2-1, 2-2, 2-3: visualization of each mask channel per branch the second level.
A.4 Additional Zero-shot Results on VOC
We provide more segmentation results of 'bicycle', 'chair', 'potted plant" and 'train' in Figure 28.
A.5 Additional Zero-shot Results DAVIS
We provide more DAVIS-201 video segmentation results of 'elassic-car', 'dance-jump' in Figure 29.
15

---

Level 2-1
Level 2-2
Level 2
Level
Combined Segmentations
Level 2-3
Figure 17: Segmentations for each level. Level I: 3-color-coded region assignments at the first. level. Level 2-1, 2-2, 2-3: 3-color-coded region assignments per branch the second level. Level combined segmentations: 9-color-coded region assignments the second level.
(a) Acc.
(b) IOU.
DICE
Figure 18: Segmentation results on CUB with E {0. 10, 20, 30, 40, 50, 60).
16

---

Table 7: We perform class label mapping from ImageNet VOC, and report zero-shot transfer Accuracy and mIOU per class on VOC validation dataset.
VOC Class.
ImageNet Class,
Num. VOC-val Image
Accuracy
mIOU
:areollaee
warpaane
136
0.82
0.57
2:bicycle
67 1.L.munttinn-bie
108
0.79
0.47
3:bird
94:hummingbird
168
0.83
0.58
4:boat
81::speedboat
115
0.81
0.51
5:hottle
907:wine-botll
133
0.76
0.47
6:bus
779:school-bus
114
0.73
0.54
7:car
817:sports-car
191
0.74
0,48
8:cat
281:abbby
206
0.82
0.66
9:chair
765:rocking-chee
175
0.75
0.64
10:cow
346:water buloe-
102
0.82
0.45
1 diiniggtabe
532:dining-tabe
89
0.69
0.62
12:dog
153 153:maltese-dog
204
0.82
0.67
13:horse
603: horsecart
104
0.84
0.53
14:motorbike
117
0.76
0.52
15:person
981 ballplaaer
584
0.77
0.46
16:potted plant
883:vase
116
0.74
0.46
17:sheep
348:ram
89
0.84
0.64
18:sofa
:studi--couu
109
0.73
0.51
19:train
466:bullet-trail
126
0.76
0.56
20:tv/monitor
664:monitor
106
0.73
0.47
Average
0.78
0.54
(a) Real Images
(b) Segmentation
Figure 19: Segmentation on Flower.
17

---

(a) Real Images
(b)
Segmentation
Figure 20: Segmentation on CUB.
(a) Real Images
(b)
Segmentation
Figure 21: Segmentation on FFHQ
18

---

(a) Real Images
(b) Segmentation
Figure 22: Segmentation on CelebA.
(a) Real Images
(b) Segmentation
Figure 23: Segmentation on ImageNet.
(a) Generated Images
(b) Generated Masks
Figure 24: Generation on Flower.
19

---

(a) Generated Images
(b) Generated Masks
Figure 25: Generation CUB.
(a) Generated Images
(b) Generated Masks
Figure 26: Generation on FFHQ.
(a) Generated Images
(b) Generated Masks
Figure 27: Conditional ImageNet generation.
20

---

(a) Real Images
Segmentation
Figure 28: Segmentation on VOC-2012.
Frames of 'Classic-car'
(b) Frames of 'Dance-jump'
Figure 29; Segmentation on DAVIS-2017
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer:
[No]
Justification:
We will release the code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https: /ttsss////ppppeeee cc/
publ / deud Codeeuumm ss ssion sssooolll Piolllcy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss. cc/ cctppcccppccccccccc pubppcplco.ooo wwwp//ggissseeeeeeeeeeeeeeeeesee Code Submi ssissiionnn ceiionnoliinn ooeliyyyyy......l for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /neurips..... hc/ppubllcc/tthhcccclleeee.ee
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
25

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper ww hhoode com/ dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
26

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "K2JaK9oR77", "reviewer": "Reviewer_9WST", "summary": "The paper proposed a structural modification of a DDPM which causes it to learn a decomposition of images into regions. This factorization enables unsupervised segmentation and simultaneously improves the quality of the generated images. The method is evaluated on various datasets, demonstrating its effectiveness in both tasks.", "questions": "- In addition to DiffuMask, please consider discussing DiffSeg (https://arxiv.org/pdf/2308.12469), which like DiffuMask, relies on a Stable Diffusion model, but unlike it, does output masks explicitly.\n- Is the proposed scheme specific to diffusion? Could it be applied in other autoencoder settings?\n- What happens if you set K to higher values, despite not needing the additional classes? e.g. K = 4, 5 for binary segmentation?\n- You mentioned that K = 3 was helpful for binary segmentation. How much worse were the results with K=2?", "limitations": "Limitations are discussed explicitly in the paper. No concerns about negative societal impact.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Evaluations on 5 datasets.\n- Consistently good performance relative to other unsupervised methods.\n- Conceptually simple but powerful design.\n- Ablation on encoder design details.\n- Zero-shot segmentation shown on 2 additional datasets.", "weaknesses": "- Experiments only cover 2-3 class scenarios.\n- ImageNet results only on downsampled 64x64 images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer 9WST\n\n**Q: Experiments only cover 2-3 class scenarios.**\n\nA: Our work is at the stage of a new architectural design for diffusion-based segmentation and generation, with 2 or 3 class segmentation results demonstrating improvements across multiple datasets, scaling up to ImageNet.\n\nTraining unconditioned DDPM from scratch for complex domains, e.g., COCO or PASCAL is challenging, even if limited to the goal of high-quality image generation. We are addressing an additional challenge of simultaneously generating a latent representation of segmentation.\n\nOur work proposes and validates new ideas on object-centric datasets. We believe there is a path toward scaling our method to more complex data; e.g., see the Appendix for our effort towards hierarchical segmentation. Fully exploring the new technique we have introduced will require follow-up papers.\n\n**Q: ImageNet results only on downsampled 64x64 images.**\n\nA: This choice is due to computational constraints and the need to validate our method's feasibility at a lower resolution before potentially scaling to higher resolutions. Training models on full-resolution (e.g., 256 $\\times$ 256) ImageNet images would require significantly more computational resources and time. Resolution is an aspect of the system orthogonal to our architectural innovation.\n\n**Q: In addition to DiffuMask, please consider discussing DiffSeg, which like DiffuMask, relies on a Stable Diffusion model, but unlike it, does output masks explicitly.** \n\nA: Thanks for the suggestion. We will add the discussion about DiffuMask in the revised draft.\n\n**Q: Is the proposed scheme specific to diffusion? Could it be applied in other autoencoder settings?**\n\nA:  While the details of our proposed scheme are tailored for diffusion models, our core principles could potentially be adapted to other autoencoder architectures, e.g., variational autoencoders (VAEs) or masked autoencoders (MAEs):\n- Bottleneck Design: The use of a structured bottleneck could be implemented in autoencoders to encourage learning of meaningful latent representations that aid in segmentation.\n- Parallel Decoding: The idea of parallel decoding for different segments could be utilized in the decoder portion of any autoencoder to encourage factorizing the reconstruction task into independent subtasks.\n\n**Q: What happens if you set K to higher values, despite not needing the additional classes? e.g., K = 4, 5 for binary segmentation?**\n\nA: $K$ is the maximum number of regions the model may use; it could learn fewer. Setting K to higher values may allow for some additional flexibility during training, as the model learns how to partition images into regions. Setting K much larger than necessary simply wastes compute as the model learns to leave some region maps empty (unused). During the rebuttal phase, we experiment with enlarging the flat set of $K$ regions to 5 for the CLEVR dataset, as shown in Figure 2 (rebuttal PDF). The model still performs foreground-background segmentation adequately by ignoring extra classes if not present.\n\n**Q: You mentioned that K = 3 was helpful for binary segmentation. How much worse were the results with K=2?**\n\nA: For binary segmentation, we found setting $K=3$ rather than $K=2$ to assist training, with learned regions emerging as foreground, background, and a contour or transition between the two, as shown in Figure 1 of the rebuttal PDF. Also, as shown in Table 1 in the rebuttal material, the results with K=2 were less satisfactory for both segmentation and generation. With the goal of handling more regions and multiscale structure, we believe a more promising future investigation is to reorganize our architectural design to support hierarchical mask factorization in place of a flat set of $K$ regions, as shown in Appendix Section A.3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YLsijWry66", "reviewer": "Reviewer_FkB4", "summary": "The paper introduces a novel neural network architecture capable of simultaneous image generation and segmentation in an unsupervised manner, eliminating the need for pre-labeled data. The core concept involves training the network to dissect an image, clean individual sections, and reassemble them. Notably, the system can decipher the image content solely through the analysis of these sections. The authors demonstrate the effectiveness of their unsupervised model for real-image segmentation. This work presents a promising approach for generating realistic images and understanding their content with minimal reliance on labeled data.", "questions": "- How does the proposed method differ from the mentioned related works? What are the key advantages it offers?\n- Are the results quantitatively compared with existing segmentation methods (e.g., DatasetGAN, etc.)?", "limitations": "The ethical implications of using generative models to create harmful content should be addressed.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clear and concise presentation\n- Experimental results validate the proposed method's effectiveness", "weaknesses": "- Limited discussion of related work: The paper lacks a comprehensive review of relevant research, hindering the assessment of genuine novelty. It's crucial to mention existing works like DatasetGAN (utilizing inner features of trained GANs for segmentation), Diffuse, Attend, and Segment (high-quality zero-shot segmentation with attention maps), DINO, and FeatUP (representation utilization in generative models for downstream tasks).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer FkB4\n**Q: Limited discussion of related work; How does the proposed method differ from the mentioned related works? What are the key advantages it offers?**\n\nA: Thanks for the suggestion. We will incorporate this discussion into our related work section to provide a clear picture of how our approach diverges from existing methods:\n\n- DatasetGAN has utilized intermediate GAN features to perform segmentation, demonstrating the potential of leveraging generative models for downstream tasks. However, our approach differs by directly integrating segmentation within the diffusion process, eliminating the need for separate feature extraction and potentially offering more coherent results.\n\n- Similarly, Diffuse, Attend, and Segment employs attention mechanisms for zero-shot segmentation, providing high-quality segmentation through attention-driven methods. Our factorized diffusion architecture offers an alternative by producing segmentation masks as an intrinsic part of the denoising process, potentially yielding more integrated and coherent segmentation results.\n\n- In the realm of self-supervised learning, DINO showcases the power of representation learning through knowledge distillation, applicable to tasks like segmentation. Our approach aligns with this goal of versatile representation learning but achieves segmentation directly within the generative model, eliminating the need for additional supervision or distillation processes.\n\n- FeatUP highlights the adaptability of generative model representations for downstream tasks. While FeatUP focuses on broad applicability, our method targets segmentation with a specifically designed architecture, achieving simultaneous image generation and segmentation in a unified framework.\n\nIn comparison with prior work, we are offering an entirely new approach to solving an end task (segmentation) using unsupervised generative learning: (1) Specify an architecture whose bottleneck representation encodes the solution to the end task, and whose decoder's computational structure constrains synthesis of data from that representation. (2) Train the system end-to-end, from scratch, for generation alone. (3) Read off the bottleneck features as the solution.\n\n**Q: Are the results quantitatively compared with existing segmentation methods (e.g., DatasetGAN, etc.)?**\n\nA: Both DatasetGAN and DatasetDDPM are not unsupervised approaches like ours. They require annotations and train in a few shot manner. In the paper, we compared with DatasetDDPM, which is a more suitable baseline. As shown in Table 3 and 4, our method outperforms DatasetDDPM by a large margin under the same unsupervised setting.\n\n**Q: The ethical implications of using generative models to create harmful content should be addressed.**\n\nA: We are committed to ensuring that the development and application of generative models are guided by ethical principles. We will incorporate the discussion in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "axckCPViR2", "reviewer": "Reviewer_vzUs", "summary": "The authors propose a model for simultaneous unsupervised semantic segmentation and image generation based on denoising diffusion probabilistic models. The methodology alters the architecture of a typical DDPM by conditioning the decoder on the outputs of a mask generator.", "questions": "see above the comments about the concatenation of ℎ𝑚𝑖d and ℎenc.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "In general, the paper is well-written and technically sound.\n\nThe experimental results surpass the state-of-the-art on the datasets they were shown and the results seem promising.", "weaknesses": "The abstract lacks context/motivation. The conclusions and experiments lack discussions of the limitations of the proposed approach.\n\nThe authors propose a simple solution to a complex problem (unsupervised segmentation) through altering the architecture of a DDPM by adding an additional decoder to generate masks in an unsupervised manner and conditioning the DDPM’s original decoder on the\noutputs of the mask generator. However, the explanation of the model is unclear:\nInitially, 𝑚𝑘 is introduced as the output of the mask generator after applying the softmax activation function, for each of the k channels. As such, 𝑚𝑘 should have dimensions width x height x 1, where width and height are the dimensions of the original images. Then, it is stated\nthat 𝑚𝑘 is concatenated with ℎenc (constituted by several outputs of the encoder provided to the decoder as skip connections) and ℎ𝑚𝑖d (latent vector) and provided as input. It is unclear how these variables of different dimensions are concatenated. In the manuscript, it says “We downsample 𝑚 accordingly to the same resolution as ℎ𝑚𝑖d and ℎenc at different stages” (line 157), however it is still unclear how exactly this downsampling step is achieved. Does 𝑚 represent the outputs of each layer of the mask generator and is it provided as input to the decoder as skip connections? If so, how is 𝑚𝑘 obtained from 𝑚? Do the outputs of all layers of the mask generator possess exactly 𝑘 channels? An example highlighting the dimensions of 𝑚𝑘, ℎ𝑚𝑖d and ℎenc and how these are concatenated would be helpful.\n\nThe experiments are a limited when it comes to the datasets used to evaluate the model for semantic segmentation. The only two datasets used for over 2 classes (distinguishing more than background and foreground) were face datasets, where all images are focused on the face and present relatively low variability. It is unclear how the method would work with over 3 classes and on datasets with higher variability such that not all objects are present in all images. Furthermore, even when only two classes are used, to distinguish background from foreground, the datasets seem to be limited, with low variability in backgrounds, making it difficult to assess the model’s results. The results on the CUB dataset, whose backgrounds present more variability and the regions with objects of interest (birds) are smaller, show difficulties segmenting the birds from the backgrounds, especially in the presence of branches (Fig 8). In the ImageNet experiment, it would also be interesting to see what would happen if 5 classes (4 objects + background) were used (k=5 in the softmax layer) rather than just two.\nWould the network be able to learn to distinguish certain objects or would it still only be able to separate background from foreground? As it stands, it becomes difficult to assess the proposed network’s strengths and whether it is truly capable of capturing semantic information or whether it just separates background from foreground without requiring more in-depth semantic information about the nature of the objects.\n\nThe paper lacks a discussion of the limitation of the methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer vzUs\n\n**Q: The abstract lacks context/motivation. The conclusions and experiments lack discussions of the limitations of the proposed approach.**\n\nA: In the current abstract, we provide a clear introduction to the challenges and motivations driving our research. The abstract begins by discussing the limitations of supervised deep learning, specifically the reliance on large amounts of annotated data, which is a well-known barrier in computer vision tasks. We explicitly mention the goal of achieving unsupervised image generation and segmentation, addressing the significant problem of high annotation costs in supervised learning. The abstract succinctly outlines our approach—a denoising diffusion model with a computational bottleneck—which is a novel strategy for tackling the dual tasks of generation and segmentation simultaneously. It concludes by highlighting the success of our model in achieving high-quality image synthesis and segmentation across multiple datasets, reinforcing the effectiveness of our approach.\n\nThe conclusion and experiment sections of the paper do acknowledge and discuss limitations, which are crucial for a balanced understanding of the proposed method. The conclusion acknowledges that the work is at an early stage of architectural design, focusing on simpler class scenarios to establish a proof of concept before tackling more complex tasks. We discuss that the experiments are performed primarily on two to three class scenarios, recognizing these as initial limitations with plans to extend the approach to higher resolutions and more complex segmentation tasks in future work.\nThe experiments highlight the need for further exploration of scalability and generalization to more diverse datasets and tasks. This is particularly important for understanding how the model performs in real-world applications beyond the current scope. The conclusion outlines future research directions aimed at addressing these limitations, such as investigating hierarchical extensions and handling more complex scenes.\n\n**Q: The explanation of the model is unclear .**\n\nA: This question is about Eq.(5), which is not our proposed design. In line 158-160, \"However, such a design significantly modifies (e.g., channel sizes) the original U-Net decoder architecture. Moreover, conditioning with the whole mask representation may also result in a trivial solution that simply ignores region masks.\" As such, there's no concatenation of mask and $h_enc$ in our proposed parallel decoding scheme. Refer to Eq.(6) and Eq. (7) for the description of our design.\n\nWe again clarify the design as depicted in Figure 2: $h_{mid}$ (latent features) is directly passed to the decoders following conventional DDPM without concatenation.\n$m_k$ denotes the $k-th$ channel of the mask generator **final** output $m$. The mask $m_k$ is used at each decoding layer as part of skip connections. It provides region-specific information that helps the decoder refine outputs by distinguishing between different semantic regions.\nAs shown in Figure 2, we adopt **channel-wise masking**: Each channel of $m$, denoted as $m_k$, is applied to $h_{enc}$ through **element-wise multiplication**.\nWe downsample $m_k$ using bilinear interpolation to align its spatial dimensions with $h_{enc}$ at different stages. Only element-wise multiplication is used with consistent spatial dimensions, allowing integration across the network's architecture without significantly changing the channel dimension of conventional DDPM.\n\nQ: **The experiments are a limited when it comes to the datasets used to evaluate the model for semantic segmentation.**\n\nA: Training unconditioned DDPM from scratch for complex domains, e.g., COCO or PASCAL, is challenging, even for the single goal of high-quality image generation. Simultaneous generation and segmentation introduces another level of difficulty. Our work proposes and validates new ideas on object-centric datasets. Subsequent efforts, and more computational resources, will be needed to scale up our techniques to larger and more complex datasets.\n\nMoreover, our work is itself on the path of scaling the segmentation capabilities of DDPMs in comparison to prior work. Consider the evolution from: DatasetDDPM (ICLR'2022) introduced the idea of using a **pre-trained** DDPM to extract segmentations in a **supervised** manner, validated on the datasets up to the scale of **CelebA**, to: Ours (submission'2024) trains a factorized diffusion model **from scratch** to accomplish both segmentation and generation **simultaneously** in an **unsupervised** manner, validated on the larger scale ImageNet dataset with **more complex scenes**.\n\nAs for the segmentation results, we provide a supervised result as a reference point against which to compare unsupervised methods using the same architecture. One would not expect any unsupervised method, limited to training on the same data, to be able to match the supervised method's performance. Our method, which is unsupervised, outperforms the unsupervised baseline (DatasetDDPM-unsup) based on the same UNet architecture. \n\nIt is the case that we want to explore much larger K for ImageNet in the future, as doing so will be necessary for segmenting complex scenes. A challenge is implementing K parallel pathways efficiently for large K; note that training is already expensive as everything takes place within a diffusion model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uU1l9QvyIo", "reviewer": "Reviewer_FyWe", "summary": "This paper proposes a novel architecture for denoising diffusion probabilistic models (DDPMs) that enables simultaneous unsupervised image generation and segmentation. The key idea is to constrain the denoising network with a structured bottleneck that factorizes the image into regions, which are then denoised in parallel. This architectural design encourages the model to learn meaningful segmentations as a byproduct of training for image generation. The authors demonstrate that their approach can generate high-quality images along with corresponding segmentations, as well as segment real images, without any supervision or additional loss terms. Experiments on multiple datasets show improvements in both image quality and segmentation accuracy compared to baselines.", "questions": "* The setting of generating both the images and the segmentation maps is interesting. However, Isn't this setup a little trival? Why we need to generate both contents at the same time? How could this feature help us?\n* How sensitive is the method to the choice of number of regions K? Is there a way to automatically determine the optimal K for a given dataset?\n* The paper mentions that the method can be extended to hierarchical segmentations. Could you provide more details on how this would work and what challenges might arise?\n* Have you explored applying this method to more complex segmentation tasks with many object categories? What modifications might be needed?\n* How computationally expensive is the proposed method compared to standard DDPMs, both for training and inference?\n* The paper claims the method can be applied to segment novel images with just one forward pass. How does the runtime compare to other unsupervised segmentation methods?\n* Some literature that share the similar insights of unifying generation and segmentation can be refered to:\n[1] Lai Z, Duan Y, Dai J, et al. Denoising diffusion semantic segmentation with mask prior modeling[J]. arXiv preprint arXiv:2306.01721, 2023.\n[2] Li C, Liu X, Li W, et al. U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation[J]. arXiv preprint arXiv:2406.02918, 2024.", "limitations": "The authors have adequately addressed the limitations of their work in Section 5, acknowledging that the current results are limited to 2-3 class segmentations and discussing the need for further work on handling more complex scenes. They also mention computational costs as a potential limitation. The paper does not directly discuss potential negative societal impacts, which could be briefly addressed, though the risks seem relatively low for this type of fundamental methodological work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Novel architectural design that unifies unsupervised image generation and segmentation in a principled way\n* Achieves strong results on both tasks without requiring additional loss terms or supervision\n* Provides insights into how the model learns to factorize images into regions\n* Explores extensions to hierarchical segmentations", "weaknesses": "* Limited theoretical analysis:\nThe paper lacks a rigorous theoretical foundation for why the proposed architecture leads to meaningful segmentations. While empirical results are strong, a deeper analysis of why factorizing the denoising process encourages semantic segmentation would strengthen the contribution. For instance, the authors could explore connections to information bottleneck principles or analyze the gradients flowing through different parts of the network.\n* Scope of experiments:\nThe experiments primarily focus on datasets with relatively simple segmentations (2-3 regions), such as Flower, CUB, and face datasets. While the ImageNet results are promising, they are limited in scope.\nThe paper lacks experiments on more challenging datasets with complex, multi-object scenes (e.g., COCO, Cityscapes) that would better demonstrate the method's scalability and generalizability.\n* Incomplete comparisons:\nThe paper misses comparisons with some recent, relevant unsupervised segmentation methods, such as PiCIE (CVPR 2021) or STEGO (ICCV 2021). Including these would provide a more comprehensive evaluation of the state-of-the-art.\nFor the image generation task, comparisons with other recent diffusion model variants (e.g., latent diffusion models) would be valuable to contextualize the improvements.\n* Architectural choices and ablations:\nThe paper does not extensively explore variations in the factorized architecture. For instance, how does performance change with different numbers of parallel decoders or alternative ways of combining their outputs?\nMore comprehensive ablation studies would help isolate the impact of different components of the proposed architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer FyWe\n\n**Q: Limited theoretical analysis.**\n\nA: Our system could potentially learn to assign the entire image to a single mask, leaving all other masks empty. In doing so, it would essentially fall back to being equivalent to a standard DDPM UNet architecture. All synthesis would occur as a result of a single run of the decoder. However, if the system learns to split the image into different regions, it gets to use multiple runs of the decoder in order to synthesize the denoised result. There is a computational advantage (using more decoder runs) that the system can leverage to better denoising, if it can learn how to partition the problem into parallel subtasks. This is the fundamental driver behind the denoising loss encouraging learning of region partitions. We agree that a quantitative analysis of these dynamics would be valuable.\n\n**Q: Scope of experiments and Incomplete comparisons; Have you explored applying this method to more complex segmentation tasks with many object categories?**\n\nA:**Regarding dataset selection**, our work is at the stage of a new architectural design for diffusion-based segmentation and generation, with 2 or 3 class segmentation results demonstrating the improvement across multiple datasets, scaling up to ImageNet.\n\nTraining unconditioned DDPM from scratch for complex domains, e.g., COCO or PASCAL, is challenging, even for the single goal of high-quality image generation. Simultaneous generation and segmentation introduces another level of difficulty. Our work proposes and validates new ideas on object-centric datasets. Subsequent efforts, and more computational resources, will be needed to scale up our techniques to larger and more complex datasets.\n\nMoreover, our work is itself on the path of scaling the segmentation capabilities of DDPMs in comparison to prior work. Consider the evolution from: DatasetDDPM (ICLR'2022) introduced the idea of using a **pre-trained** DDPM to extract segmentations in a **supervised** manner, validated on the datasets up to the scale of **CelebA**, to: Ours (submission'2024) trains a factorized diffusion model **from scratch** to accomplish both segmentation and generation **simultaneously** in an **unsupervised** manner, validated on the larger scale ImageNet dataset with **more complex scenes**.\n\n**Regarding baseline selection**, our goal is to perform both unsupervised segmentation and image generation in a unified framework. This is an entirely novel combination of capabilities; our work is an initial proof-of-concept, not an attempt to achieve state-of-the-art unsupervised segmentation results. The relevant competing baselines mainly lie in the scope of generative models for unsupervised segmentation. Our approach is able to achieve better unsupervised segmentation results across multiple datasets. We can also achieve improved generation quality (over standard DDPM) as shown in Experimental Section 4.1. Simply putting the state-of-the-art unsupervised segmentation methods in comparison with our approach ignores the more challenging setting in which we operate (requiring that we are also able to generate images), as well as the fact that our system improves generation quality.\n\n**Q: Architectural choices and ablations.**\n\nA: We do have a more systemic investigation in different architectural choices in Appendix Section 3: reorganizing our architectural design to support hierarchical mask factorization in place of a flat set of $K$ regions. Additionally, during the rebuttal phase, we enlarge the flat set of $K$ regions to 5 for CLEVR dataset, as shown in Figure 2 of the rebuttal PDF. We also show the results with K=2 were found to be less satisfactory than K=3 for both segmentation and generation in Table 1 (rebuttal PDF).\n\n**Q: Why need to generate both contents at the same time?**\n\nA: We only train the model once and achieve both targets. Doing both together actually improves generation quality, while yielding the ability to segment as a bonus. Alternatively, through the lens of a traditional computer vision task, we learn to segment in an unsupervised manner and get the ability to generate as a bonus.\n\n**Q: How sensitive to the choice K?**\n\nA: We are defining the maximum number of regions the model can use for factorization.  There is no requirement that it utilize all of these mask components. For example, it could learn to only use two mask channels out of three. Or it could predict every pixel as belonging to the same mask channel, thereby collapsing back to the vanilla UNet as a base case. The fact that the model prefers to learn something nontrivial is related to the actual structure of the data; nothing forces it to use all K channels.\n\n**Q: More details on hierarchy segmentation.**\n\nA: We have detailed the investigation in hierarchy segmentation in Appendix A.3. \nWe formulate a hierarchical factorized diffusion architecture to progressively refine segmentation results from a coarse initial prediction to a fine, detailed final segmentation. Our initial investigation into hierarchical extensions suggests a promising future path towards handling complex scenes. \n\n**Q: How computationally expensive for training and inference, compared with DDPMs?**\n\nA: Given the nice property of weight sharing scheme in parallel decoding, the only additional weights are from our mask generator. We have an efficient batching implementation for decoding, which makes training and sampling speed comparable to standard DDPMs. As for segmentation, the inference speed is the same as a single forward pass in a standard DDPM.\n\n**Q: Runtime compared to other unsupervised segmentation methods?**\n\nA: If both use the same UNet encoder-decoder architecture, the inference complexity is the same. The parallel decoding scheme and diffusion process does not affect the efficiency of segmentation inference speed.\n\n**Q: Some literature can be referred to.**\n\nA: Thanks for the suggestion. We will include a discussion in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Reply

We thank the reviewers and answer specific questions in individual responses to each review below. We provide some general clarifications here, as well as specific clarifications below.

Our goal is both image segmentation and image generation, learned simultaneously and in an entirely unsupervised manner.  Our system resembles a standard DDPM, but innovates on the neural network architecture inside that DDPM.  Our novel network architecture is designed around heterogeneous subcomponents which have the emergent effect of encouraging the DDPM to factorize the denoising process into parallel subproblems and allow us to easily examine the learned factorization.  For images, it happens that the natural factorization, simply as a consequence of the structure of the data, corresponds to region segmentation.  Thus, as a result of learning to generate (denoise) with this particular network architecture, we also learn to segment.

Once trained, our system can operate in two modes: segment an arbitrary input image, or generate an image from random noise.  Segmenting a novel input image is fast (single forward pass of a UNet, comparable in speed to any other image segmentation network); generation is slow (many steps of a reverse diffusion process, just as in a standard DDPM).

Our work represents the first example of a new strategy for leveraging diffusion models to learn other latent information in an unsupervised manner: constrain the generation (denoising) architecture to have a computational structure and representation bottleneck that reveals that desired latent information.  In our case, the latent information is image segmentation, the bottleneck is feature tensors restricted to region masks, and the computational structure is parallel pathways processing those masked feature tensors.  If this recipe proves to be general, it should inspire a new strategy for the architectural design of future foundation models.

Meta-review: Authors present an unsupervised diffusion model that can not only
generate images but also provide segmentations, provided the number of
masks required as input. The approach and the architecture seems quite
novel as also stated by some reviewers.

The main concern raised during the review was that the extent of
experiments could be larger. Authors could have done experiments with
higher number of classes, full resolution images and more
datasets. During the rebuttal authors note that the main issue is the
computational cost. Given the number of experiments and the ablation
study, I tend to agree with the authors. Some reviewers also agree with
this.

The last reviewer also mentions theoretical insights are not
provided. Authors' response, while not answering it fully, provided
insights. This reviewer gave a BA for this paper.

There are two reviewers who gave BR to this article. Their main
concers were about experiments and comparisons. The second BR
reviewer also mentions clarity and motivation. Authors did their best
to answer these concerns during the rebuttal. Unfortunately, reviewer
did not engage in a discussion. However, reading the rebuttal, I think
authors response were valuable.
Decision: Accept (poster)