Discrete Distribution Networks
Authors: Lei Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Generative Models
Image Generation
Primary area: generative models
TL;DR: A novel generative model with simple principles and unique properties.

Abstract: We introduce a novel generative model, the Discrete Distribution Networks (DDN), that approximates data distribution using hierarchical discrete distributions. We posit that since the features within a network inherently capture distributional information, enabling the network to generate multiple samples simultaneously, rather than a single output, may offer an effective way to represent distributions. Therefore, DDN fits the target distribution, including continuous ones, by generating multiple discrete sample points. To capture finer details of the target data, DDN selects the output that is closest to the Ground Truth (GT) from the coarse results generated in the first layer. This selected output is then fed back into the network as a condition for the second layer, thereby generating new outputs more similar to the GT. As the number of DDN layers increases, the representational space of the outputs expands exponentially, and the generated samples become increasingly similar to the GT. This hierarchical output pattern of discrete distributions endows DDN with unique properties: more general zero-shot conditional generation and 1D latent representation. We demonstrate the efficacy of DDN and its intriguing properties through experiments on CIFAR-10 and FFHQ. The code is available at https://discrete-distribution-networks.github.io/

Published as a conference paper at ICLR 2025
DISCRETE DISTRIBUTION NETWORKS
Lei Yang
StepFun
Megvii Technology
yanglei wttpleeiisteefu.c.cmmmm
ABSTRACT
We introduce novel generative model, the Discrete Distribution Networks (DDN), that approximates data distribution using hicrarchical discrete distribu- tions. We posit that since the features within network inherently capture dis- tributional information, enabling the network to generate multiple samples si- multaneously, rather than single output, may offer an effective way repre- sent distributions. Therefore, DDN fits the target distribution, including con- tinuous ones, by generating multiple discrete sample points. To capture finer details of the target data, DDN selects the output that is closest to the Ground the first layer. This selected
Truth (GT) from the coarse results generated output is then fed back into the network as condition for the second layer, thereby generating new outputs more similar to the GT. As the number of DDN layers increases, the representational space of the outputs expands exponen- tially, and the generated samples become increasingly similar to the GT. This hierarchical output pattern of discrete distributions endows DDN with unique properties: more general zero-shot conditional generation and ID latent repre- sentation. We demonstrate the efficacy of DDN and its intriguing properties through experiments on CIFAR-10 and FFHQ. The code is available at https: 7/di seriiicreteeeee dit.-distrriboee on networks github..hu......g io/
f-e-ee- level
level
level
(Gail)
Rool
Sampler
Sampler
Sampler
lool
DDN
8
Lasent
2
8
8 Recenstracted
8
8
1- Enl
Tanget
Uaicakd
(a) Image Reconstruction through DDN
(b) Tree Structure of DDN's Latent
Figure 1: (a) Illustrates the process of image reconstruction and latent acquisition in DDN, Each layer of DDN outputs K distinct images approximate the distribution P(X). The sampler then selects the image most similar to the target from these and feeds it into the next DDN layer. As the number of layers increases, the generated images become increasingly similar to the target. For generation tasks, the sampler is simply replaced with a random choice operation. tree-structured representation space of DDN's latent variables. Each sample can be mapped a leaf node on this tree.
(b) Depicts the
INTRODUCTION
With the advent of ChatGPT Brown et al. (202 and DDPM Ho et al. (202)), deep generative models have become increasingly popular and significant in everyday life. However, modeling the complex and diverse nigh-dimensional data distributions is challenging. Previous methods Kingma & Welling 22014); Radford et (2016)) Kingma & Dhariwal 2018 Goyal et al. b0 [Song al. (2021); et al. (202)); (2002))/rrvvee Graves et al. (2023) have each demonstrated their unique strengths and characteristics in modeling these distributions. In this work, propose novel approach to model

---

Published as a conference paper at ICLR 2025
SR16
Denoise
Lawbir
Colorial
Iupaiating
CLIP
ST
Generated Condition
S
Generated Conditan
Generated Condition
Figure 2: DDN enables more general zero-shot conditional generation. DDN supports zero-shot conditional generation across non-pixel domains, and notably, without relying on gradient, such as text-to-image generation using black-box CLIP model Radford et al. (2021). Images enclosed in yellow borders serve as the ground truth. The abbreviations in the table header correspond to their respective tasks follows: 'SR' stands for Super- RusoutReso,ution, with the following digit indicating the resolution of the condition. 'ST' denotes Style Transfer, which computes Perceptual Losses with the condition according to Johnson et al. (016)).
the target distribution, where the core idea to generate multiple samples simultaneously, allowing the network to directly output an approximate discrete distribution. Hence, we name our method Discrete Distribution Networks (DDN). DDN embraces core concept as simple as autoregressive models, offering another straightforward and effective form for generative models.
Most generative models applied in real-world scenarios are conditional generative models. Tak- ing image generation as example, these models generate corresponding images based content provided by users, such as images to be edited, reference images (Saaaria et al. (202T), text descrip- tions, hand-drawn editing strokes Voynov et al. (2022). sketches, and generative models Rombach et al. (2021); Ramesh et al. (2022); Zhang et al. (2023) typically re- quire training separate models and parameters for each condition. These models restricted to fixed condition formats and lack the flexibility to adjust the influence of each condition dynamically, thereby limiting users' creative freedom.
Current mainstream
Recent works have attempted to address this issue through zero-shot conditional generation (ZSCG). However, these methods either only support conditions in the same pixel domain the training data Wang et al. [2022), Lugmayr et al. 2202);; Meng et al. (222)); Nair et al. (2023) or depend on dis- criminative models to supply gradients during generation Yu et al. (2023). In contrast, DDN supports a wide range of ZSCG tasks, encompassing both pixel-domain and non-pixel-domate conditions, as shown in fig. 2 To the best of our knowledge, DDN is the first generative model capable of perform- ing shoo conditional generation in non-pixel domains without relying on gradient information. This implies that DDN can achieve ZSCG solely based on black-box discriminative models.
The core concept of Discrete Distribution Networks (DDN) is to approximate the distribution of training data using a multitude of discrete sample points. The secret to generating diverse samples lies in the network's 's ability to concurenttyy generate multiple samples (K). This is perceived as the network outputting a discrete distribution. All generated samples serve as the sample space for this discrete distribution. Typically, each sample in this discrete distribution has equal probability mass of 1/K Our goal is to make this discrete distribution as close as possible to the target dataset.
To accurately fit the target distribution of large datasets, substantial representational space is quired. In the most extreme scenario, this space must be larger than the number of training data samples. However, current neural networks lack the feasibility to generate such vast number of
2

---

Published as a conference paper at ICLR 2025
samples simultaneously. Therefore, adopt strategy from autoregressive models Van Den Oord et al. (2016) and partition this large space into a hierarchical conditional probability model. Each layer of this model needs only small number of outputs. We then select one of these outputs the output for that layer and use it as conditional input to the next layer. As a result, the output of the next layer will be more closely related to the selected conditional sample. If the number of layers is L and the number of outputs per layer is K, then the output space of the network is K Due to its exponential nature, this oulput space will be much larger than the number of samples in the dataset. Figure | shows how DDN generates images.
We posit that the contributions of this paper are follows:
We introduce novel generative model, termed Discrete Distribution Networks (DDN), which exhibit a more straightforward and streamlined principle and form.
For training the DDN, we propose the Appit-and-Prune" optimization algorithm, and a range of practical techniques.
We conduct preliminary experiments and analysis on the DDN, showcasing its intriguing properties and capabilities, such as zero-shot conditional generation and ID latent repre- sentations.
2 RELATED WORK
Deep Generative Model. Generative Adversarial Networks (GANs) Radford et al. 6): Brock al. (2019) and Variational Autoencoders (VAEs) Kingma & Welling (2014); van den Oord et al. (2018) are two early successful generative models. GANs reduce the divergence between the gen- crated sample distribution and the target distribution through a game between the generator and discriminator. However, regular GANs cannot map samples back to the latent space, thus they cannot reconstruct samples. VAEs encode data into a simple distribution's 's latent space through an Encoder, and the Decoder is trained reconstruct the original data from this simple distribution' latent space. Autoregressive models van den Oord et al. (2016). with their simple principles and methods, model the target distribution by decomposing the target data into conditional probability distributions of each component. They can also compute the exact likelihood of target samples. However, the efficiency of these models is reduced when dealing with image data, which is not suitable to be decomposed into sequence of components. Normalizing flow Kingma & Dhariwal (018) is another class of generative models that can compute the likelihood. They use invertible networks to construct mapping from samples to noise space, and during the generation stage, they map back from noise to samples. Energy Based Models (EBM) Goyal et al. (202T)))); [Song et al.) (021) with their high-quality and rich generative results, have led to the rise of diffusion models. However, their multi-step iterative generation process requires substantial computational resources. The recent Idempotent Generative Network Shocher et al. (2024) introduces novel approach by training a neural network to be idempotent, mapping any input to the target distribution effectively.
Connections to VQ-VAE While both VQ-VAE van den Oord et al. (2018) and DDN involve dis- crete representations, they differ significantly in their approach and capabilities. VQ- VAE enhances the traditional VAE by replacing the continuous latent space with discrete codebooks, thus achieving a more compact representation. VQ-VAE-2 Razavi et al. (2019) further improves this by employ- ing multi-scale hierarchical structure, thereby enhancing its representational power. However, the discrete representation in VQ-VAE remains two-dimensional. potentially leading to redundant in- formation. Furthermore, VQ-VAE and its successors still rely on an additional prior network for generative modeling in the latent space. Notably, DDN can also serve as this prior model to effec- tively model the latent space of VQ-VAE, Other distinctions between DDN and VQ-VAE include the absence of an encoder and codebook in DDN, as well as its capacity for Zero-Shot Conditional Gen- cration. VQ- VQ-VAEs are known to encounter codebook collapse, problem that some researchers have addressed by reinitializing unused codes near frequently used ones Williams et al. (2020); Dhariwal jet al. (2020). Our Aplit-and-Prrun algorithm shares a similar core idea, albeit with some differences. While the reinitialization method aims to balance code usage to mitigate codebook collapse, goal is to align the discrete distribution output by our network as closely as possible to the target distribution.
3

---

Published as a conference paper at ICLR 2025
8
8 loage
Taaaetaennedd Ieage
Geuaral sage
Fark
Corcal
Ouger Seatur
(b) Single Shot Generator (L=6)
comnlsl
conntal
GoneeAk Iraig
convist
Traking Yeage
LAws
8 Laage
8
Thenes brapt
8
(a) Overview
(c) Recurrence Iteration
Figure 3: Schematic of Discrete Distribution Networks (DDN), (a) The data flow during the training phase of DDN is shown at the top. As the network depth increases, the generated images become increasingly similar to the training images. Within each Discrete Distribution Layer (DDL), K samples are generated, and the one closest to the training sample selected as the generated im- age for loss computation. These K output nodes are optimized using Adam with the Split-and-Prumt method. The right two figures shown the two model paradigms supported by DDN. (b) Single Shot Generator Paradigm: Each neural network layer and DDL has independent weights. (c) Recurrence Iteration Paradigm: All neural network layers and DDLs share weights. For inference, replacing the Guided Sampler in the DDL with random choice enables the generation of new images.
3 DISCRETE DISTRIBUTION NETWORKS
Network architecture. Figure 3a illustrates the overall structure of the DDN, comprised of Neural Network Blocks and Discrete Distribution Layers (DDL). Each DDL contains K output nodes, each of which a set of 1x1 convolutions responsible for transforming the feature into the corresponding output image. The parameters of these 1x1 convolutions are optimized by Adam with Split-and- Prune. The K images generated by the K output nodes are inputted into the Guided Sampler. The Guided Sampler selects the image with the smallest L2 distance to the training image, which serves as the output of the current layer and used to calculate the L2 loss with the training image. Simultaneously. the selected image is concatenated back into the feature, acting the condition for the next block. The index (depicted in green as "K-1" in fig. 3a) of the selected image represents the latent value of the training sample this layer. Through the guidance of the Guided Sampler layer by layer, the image generated by the network progressively becomes more similar to the training sample until the final layer produces an approximation of the training sample.
For computational efficiency, we adopted a decoder structure similar to the generator in GANs for coarse-to-fine fine image generation, shown in fig. 3b We refer this the Single Shot Generator which is our default choice. As each layer of DDN can naturally input and output RGB domain data, DDN seamlessly support the Recurrence Iteration Paradigm fig. 3c
Objective f'unction. The DDN model consists of L layers of Discrete Distribution Layers (DDL). For given layer 1, denoted as fi, the input is the selected sample from the previous layer, x²_ The layer generates K new samples, Jr (x(xx1)) from which we select the sample x; that is closest the current training sample x, along with its corresponding index ky. The loss Jj for this layer is then computed only on the selected sample x
k = argmin ke kE(I....K) [|](({]-)[[)) x1|
x fi(i-1)(
J1 = ||x] - x||
(2)
Here, x1 = 0 represents the initial input to the first layer. For simplicity, we omit the details of input/output feature, neural network blocks and transformation operations in the equations.
4

---

Published as a conference paper at ICLR 2025
By recursively unfolding the above equations, derive the latent variable ki.,. and the global objective function J.
[[( xxp]
x|?
ii = ki; ki] =
argmin kE(l.....))
(3)
1=1
L F(k - xx12
(4)
Here, FF represents the composite function formed from ft. defined as: F(kia) filfi-((... fi [xx))[ka]... .(.)))))))))))))) Finally, we average the L2 loss across all layers to obtain the final loss for the entire network.
How to generation. When the network performs the generation task, replacing the Guided Sampler with a random choice enables image generation. Given the exponential representational space of KL sample points and the limited number of samples in the training set, the probability of sampling an image with the same latent space as those in the training set is also exponentially low. For image reconstruction tasks, the process almost identical to the training process, only substituting the training image with the target image to be reconstructed and omitting the L2 Loss part from the training process. The Final Generated Image in fig. 3a represents the final reconstruction result. The indices of the selected samples along the way form the target image's latent ki,,, same as eq. (3). Therefore. the latent ki,, sequence of integers with length L, which we regard as the hierarchical discrete representation of the target sample. The latent space exhibits tree structure with layers and K degrees per node, where each leaf node represents sample point, and its latent denotes the indices of all nodes along the path to this leaf node, as shown in fig. [b] In the latent sequence, values placed earlier correspond to higher-level nodes in the tree, controlling the low-frequency information of the output sample, while later values tend to affect high figh-freyuency information.
1E E21
C.
1.5
04
0.5
0.5
0.5
0.
0.4
022
AG2
P-1
P-0
(a) Initial, K=5
(b) Split, K=5
(c) Prune, K=6 +5
(d) Final, K=5
Figure 4: Illustration of the principle behind the Split-and-Prune operation. For example in (a), the light blue bell-shaped curve represents a one-dimensional target distribution. The 5 "f" under the x-axis the initial values from a uniform distribution of 5 output nodes, which divide the entire space into 5 parts using midpoints between adjacent nodes as boundaries (i.c., vertical gray dashed lines). Each part corresponds to the range represented by this output node the continuous space r. Below each node are three values: P stands for the relative frequency of the ground truth falling within this node's range during training; Q refers the probability mass of this sample (node) in the discrete distribution output by the model during the generation phase, which is generally equal for each sample, i.e., 1/K. The bottom- most value denotes the difference between P and Q. Colorful horizontal line segments represent the average probability density of P, Q within corresponding intervals. In (b), the Split operation selects the node with the highest P (circled in red). In (c), the Prune operation selects the node with the smallest P (circled in red). In (d), through the combined effects of loss and Split-and- towards final optimization. From the observed results, the KL divergence (KL(P) [Q) consistently decreases as the operation progresses, and the yellow line increasingly approximates the light blue target distribution.
operations, the distribution of output nodes moves
5

---

Published as a conference paper at ICLR 2025
3.1 OPTIMIZATION WITH SPLIT-AND-PRES PRUNE
We have observed two primary issues resulting from each layer calculating loss only with the closest output samples to the ground truth (GT). Firstly, a problem similar to the"dead codebooks" VQ- VAE, wherein oulput nodes that are not selected for a long time receive no gradient updates. During generation, these "dead nodes" are selected with equal probability, leading to poor output. The second issue is the probability density shift. For instance, distribution, the target distribution is mixture of two thin and tall Gaussian distribution functions, with one larger and one smaller peak. The means of these two peaks are and 1, respectively. Therefore, half of the output samples with initial values less than 0 will be matched with samples from the larger peak and optimized towards the larger Gaussian distribution. Meanwhile, the other half of the output samples with values greater than 0 will be optimized towards the smaller Gaussian. A problem arises as the large and small peaks carry different probability masses but occupy equal portions of the output samples, resulting in the same sampling probability during generation.
one-dimensional asymmetric bimodal
Inspired by the theory of evolution and ge- netic algorithms Katoch et al. (202I). propose the Split-and-Prune algorithm to ad- dress the above issues, outlined in algo- rithm I The Split operation targets nodes fre- quently matched by training samples, while the Prune operation addresses the issue of "dead nodes", similar to those in Williams eet al.) (2020); Dhariwal et al. (2020). These nodes are akin to species evolution, sub- ject to diversification and extinction. During training. we track the frequency with which each node is matched by training samples. For nodes with excessive frequency, we exe- cute the Split operation, cloning the node into two new nodes, each inheriting half of the old node's frequency. Although these two new nodes have identical parameters and outputs, the next matched training sample will only be associated with one node. Therefore, the loss and gradient only affect one node's parame- ters. Consequently, their parameters and out- puts exhibit slight differences, dividing the old node's 's match space into two. In subse-
Algorithm Split-and- Panuiieend-r of one layer
Require:
Output nodes number K, model S, non-outpuc
parameters O, target distribution (xx)
1: Initialize output node parameters (b(k) for kt
{1...., K} with random values
2: Initialize counter e(k) 0 for &s (1....,K)
3: Set split/prune threshold Paplit +- 2/K, Pprune 0.5/K
4: n 4 O, knew +- K +1
5: repeat
6:
g(x)
7:
Choose k² argmin II5 ((k)) x||² kE
8:
Gradient descent Ve (0, <bk))) x11 c(k*) c(k") +1
9:
10:
n m+++1
11:
kmaz =
argmax c(k) and kmin
argmin c(k)
12:
if c(kmar)/n > P Paptie or c(kmms) n < Perune
then
Split:
13:
(bineew)) = clone( (bb( (kmaz))
quent training, the outputs of the two new nodes will move towards the centers of their 15:
14:
c(knew = c(kmas 1/2
c(kmaz) = c(kimax) 1/2
respective spaces under the influence of the loss, diverging to produce more diverse out- puts. For nodes with low matching frequency
16:
knew f- kinww +1
Prune:
17:
n f- n c(kims))
(dead nodes), we implement the Prune opera-
18:
remove (bkmm)))) and c(kmin)
19:
end if
tion, removing them outright. Figure 4 illus--
20:
until converged
trates the process of the Split-and- ation and how it reduces the distance between
oper-
the discrete distribution represented by the Output Nodes and the target distribution. The efficacy of the Split-and- Prune optimization algorithm is validated through examples of fitting 2D density maps fig.
3.2 APPLICATIONS
Zero-Shot Conditional Generation (ZSCG). The DDN sampling process occurs directly within the sample space, which inherently facilitates Zero-Shot Conditional Generation. Each layer of the DDN generates multiple target samples, with selected sample being forwarded condition to the next layer. This enables the generation of new samples in the desired direction, ultimately producing sample that meets the given condition. In fact, the reconstruction process shown in

---

Published as a conference paper at ICLR 2025
fig- la is ZSCG process guided by target image. It is important note that the target image is never directly input into the network. Instead, it shapes the generation outcomes by steering the sampling process.
To implement ZSCG, we replace the Guided Sampler in fig. 3a with a Conditional Guided Sampler. For instance, when generating image of class Yi guided by classifier Icts: replace the "L.2 Distance to Training Image" in the Guided Sampler of fig. Ba with the probability of each output image belonging to class y/ according the classifier. We then replace "argmin" with "argmax" to construct the Guided Sampler for this classifier. Similar to eq. T). the sampling method is:
Kt argmax RE(L...,KK
(5)
After performing L steps guided sampling and L x K steps of classification, the ZSCG result can be obtained without any gradient.
For super-resolution and colorization tasks, we construct a transform that converts the generated images into the target domain (low-resolution or grayscale). This approach significantly reduces the impact of the missing signal from the condition on the generated images, enabling DDN to successfully perform super-resolution tasks even when the source image has a resolution as low as 4 x 4.
The use of "argmax" in the Guided Sampler causes each layer to select a fixed sample, resulting in the same image output under identical conditions, similar to greedy sampling in GPT. To increase diversity, we use Top-k sampling with 2 for most ZSCG tasks, which balances diversity and condition appropriateness within the large generation space (2%).
The versatility of ZSCG can be further enhanced by combining different Guided Samplers. For example, an image can guide the primary structure while text guides the attributes. The influence of each condition can adjusted by setting their respective weights. Experiments involving the combination of different Guided Samplers are presented in appendix B
Efficient Data Compression Capability. The latent of DDN a highly compressed discrete rep- resentation, where the information content of a DDN latent is L x log2 K bits. Taking our default experimental values of K=512 and L=128 as an example, a sample can be compressed to 1152 bits, demonstrating the efficient lossy compression capability of DDN. We hypothesize this ability originates from two aspects: 1) the compact hierarchical discrete representation, and 2) the Split- and-Prune operation makes the probabilities of each node equal possible, thereby increasing the information entropy Shannoo (1948) of the entire latent distribution and more effectively utilising each bit within the latent.
In our experiments, we set K=512 as the default, considering the balance between generation per- formance and training efficiency. However, from the perspective of data compression, setting K to 2 and increasing L provvdes better balance between representation space and compression effi- ciency. We refer to DDN with K as Taiji-DDN because of its similarity to the concept of Taiji in ancient Chinese philosophy, as described in fig. [8] To our knowledge, Taiji-DDN is the first generative model capable of directly transforming data into a semantically meaningful binary string which represents leaf node balanced binary tree.
3.3 TECHNIQUES
In this subsection, we present several techniques for training Discrete Distribution Networks.
Chain Dropout. In scenarios where the number of training samples is limited, each data sample undergoes multiple training iterations within the network. During these iterations, similar selections are often made by the Guided Sampler at each layer. However, the representational space of DDN far exceeds the number of training samples in the dataset. This disparity leads to a situation where the network is only trained on a very limited set of pathways, resulting in what can be perceived overfitting these pathways. To mitigate this, we introduce strategy during training where each Discrete Distribution Layer substitutes the Guided Sampler with a "random choice" at a fixed probability rate. We refer to this method "Chain Dropout".

---

Published as a conference paper at ICLR 2025
(a) ColebA-Ho HH 64x64
(b) FFHQ 64x64
/
/
2 2 Z 2 2 2
3 3 3 3 3 3
+
4 4 H 4 4
5
5
S 5 5 5 5 5
6
6 6 6 6 6 6 6
6
6
7
7
7
8
8 8 8
8
9 9 (e) Conditional MNIST
(c) CIFAR
(d) Conditional CIFAR
Figure 5: Random samples from DDN. Figures (d) and (c) showcase images that are conditionally
generated by conditional DDN, with each row of images representing distinct category.
Learning Residual. In the context of utilizing the Single Shot Generator structure, Discrete Dis- tribution Layer introduced every two convolution layers. Given such small amount of computation between adjacent layers, directly regressing the images themselves with these convolutions becomes challenging. Drawing inspiration from ResNet He et al. (016)), we propose scheme for the net- work to learn the residual between the output images from the preceding layer and the ground truth. The output of the current layer is then computed as the sum of the output from the previous layer and the current layer's residual. This approach alleviates the pressure of the network represent complex data and enhances the flexibility of the network.
Leak Choice. In each DDN layer, the output is conditioned on the image selected from the previous layer. This condition serves as a signal in the image domain, requiring the current layer to expend computational resources extract features and interpret the choices made by the sampler. However, the computation between two adjacent layers in Single Shot Generator Paradigm is very small, involving only two convolutional layers. To facilitate faster understanding of the choices made by the Sampler in the subsequent layer, we have added extra convolutional layers to each output node. The features produced by these extra convolutions also serve as conditional inputs to the next layer. But these features don't 't participate in the calculation of distance or loss in DDL.
EXPERIMENTS
We trained our models on a server equipped with 8 RTX2080Ti GPUs, setting the Chain Dropout probability to 0.05 by default. For the 64x64 resolution experiments, we utilized DDN with 93M parameters, setting K = 512 and L = 128. In the CIFAR experiments, we employed a DDN with 74M parameters, setting K 64 and L 64. The MNIST experiments were conducted using Recurrence Iteration Paradigm UNet model with 407K parameters, where K = 64 and L = 10. the foundation of the EDM Karras et al. (2022) codebase, with training
DDN is implemented parameters nearly identical to EDM. More extended experiments exploring the properties of DDN can be found in the appendix.
8

---

Published as a conference paper at ICLR 2025
4.1 QUALITATIVE AND QUANTITATIVE RESULTS
Generation Quality. Figures Sa and 5b depict random generation results of DDN CelebA-HQ- 64x64 Karras et al. (0117) and FFHQ-64x66 Karras et al. (2019), verifying the model's effectiveness in modeling facial data. The generation quality CelebA-HQ appears superior that FFHQ, which is also reflected in the lower FID score (35.4 VS 13.1). We surmise this disparity arises from CelebA-HQ's relatively cleaner backgrounds and less diverse facial data compared to FFHQ
Figure 5c showcases the random generation results of DDN the CIFAR dataset. We also present the FID score of DDN on unconditional CIFAR in table [ com- paring it with classical generative models. is worth noting that modeling CIFAR remains challenging task, especially for new and under-explored generative mod- els like DDN. For instance, the recent work IGN [Shocher ctt al. 2024) did not conduct experiments on CIFAR.
Table I: Quantitative comparison CIFAR-10. The data for VQ-VAB
comes from Vuong al. (2023). Data for other baselines comes from Bond-
Taylor al. (2021).
Method
Type
FID
Conditional Training. Training a conditional DDN is quite straightforward, only requires the input of the condition or features of the condition into the network, and the network will automatically learn PYX Fig- ures 5d and 5e show the generation results of the class- conditional DDN on CIFAR and MNIST, respectively, Figure X further demonstrates the combination of condi- tional generation and ZSCG on image-to-image tasks.
DCGAN
GAN
37.1
IGEBM
EBM
38.0
VAE
VAE
106.7
VQ-VAE
VAE
117.4
Gated PixelCNN
AR
65.9
GLOW
Flow
46.0
DDN(ours)
DDN
52.0
Zero-Shot Conditional Generation. We trained DDN model on the FFHQ-64x64 dataset and then evaluated its zero-shot conditional image generation capability using the CelebA-HQQ64xx6 dataset, as shown in fig. In appendix [B] we presented the experiments of text-guided ZSCG using CLIP Radford et al. 20 In particular, our generation process does not require gradient derivation, numerical optimization, or iterative steps. To the best of our knowledge, DDN is the first generative model to support the use of purely discriminative model guide for zero-shot conditional generation.
In addition, we employed an off-the-sheif CIFAR classifier Phan (202T) to guide the generation of specific category images by a DDN model trained unconditionally on CIFAR. We want to emphasize that the classifier is an open-source, pre-trained ResNet18 model, with additional mod- ifications or retraining. Figure displays images of various CIFAR categories gen- erated by the model under the guidance of the classifier.
Table Ablation study FFHQ-64x64. We use the reconstruction Fréchet Inception Distance (rFID) to reflect the reconstructive performance of the network. All models are trained on the FFHQ-64x64 dataset. The rFID-FFHQ represents the reconstructive perfor- mance of the model on the training set, while rFID- CelebA can be seen as an indication of the model's gen- eralization performance the test set. "w/o" stands for "without".
Latent analysis. We trained a DDN on the MNIST dataset with K 8 and L visualize both the hierarchical generative
Model
FIDI
FFHQL
CelebA.
K=512 (default)
43.1
26.0
33.2
K=64
47.0
32.3
38.7
behavior of the DDN and the distribution
K=8
52.6
40.9
49.8
of samples in the entire latent representa-
K=2 (Taiji-DDN)
66.5
38.4
70.6
tion space, as shown in fig.
w/o Split-and- Prune
55.3
31.2
34.7
w/o Chain Dropout
182.3
26.5
37.4
w/o Learning Res.
56.2
40.2
40.2
4.2 ABLATION STUDY
w/o Leak Choice
56.0
34.3
32.2
In Table we demonstrate the impact different numbers of output nodes (K) and various techniques on the network. Interestingly, despite the substantial difference between having and not having the Split-and-PP Pnn-rne technique in the toy example, shown in fig. 17 the perfor- mance in the ablation study without Split-and- -runne not as poor as one might expect. We hy- pothesize that this is due the Chain Dropout forcing all dead nodes receive gradient guidance,

---

Published as a conference paper at ICLR 2025
preventing the network from generating poor results that unoptimized. A particular case is when K = 2. where the representational space of TajjiDDDN is already sufficiently compact. The use of Chain Dropout in this case tends to result in more blurred generated images. Therefore, did not employ Chain Dropout when K = 2.
5 CONCLUSION
In this paper, have introduced Discrete Distribution Networks, generative model that approx- imates the distribution of training data using a multitude of discrete sample points. DDN exhibits unique property: more general zero-shot conditional generation. We also proposed the Split-and- Prune optimization algorithm and several effective techniques for training DDN. Additionally, we showcase the efficacy DDN and its intriguing properties through experiments.
6
C
6
3
5
2
1
PSC
2.
2
5
228
2
2
4
7
7 7
9
9
7 7
4
9
4
9 9
9
4
2 2
4
6 6
4
9
7
299
4
9
7
7
666
4
333
3
9
4 9
3
8
0
333 8
3
7
9
3
3 333 533333
4
5
328
55
4
4
P 8
5
8
5
9
9
4
5
2 2
2
2
5
0
S
6
6
4
4 9
8
2
7
0
0 6
3
5
q
9
9
2
5
Figure 6: Hierarchical Generation Visualization DDN. We trained DDN with output level L = 3 and output nodes K = per level on MNIST dataset, its latent hierarchical structure is visualized in the middle is the average of all the generated images. Each sample with colored border represents interme- diate generation image: blue border indicates generation by the first layer, and a green border by the second layer. The samples within the surrounding grid of each colored- bordered sample are refined versions generated condition- ally based on it (enclosed by the same color frontier). The small samples without colored borders are the final gener- ated images. More detailed visualization of L 4 is pre- sented in the appendix fig. 19
Figure 7: Zero-Shot Conditional Generation CIFAR-10 Guided by pretrained classification model with- out gradient. Each row corresponds to a class in CIFAR-10. Specifi- cally, the first planes, the third row displays birds flying against the blue sky, and the last row presents trucks. Our model suc- cessfully generates reasonable images for each class without any conditional training, demonstrating the powerful zero-shot generation capability of our DDN.
recursive grids. The large image
consists of air-
ACKNOWLEDGEMENTS
Special thanks to Haikun Zhang for her comprehensive support, including but not limited to paper formatting, CIFAR classification experiments. Thanks to Yihan Xie for her significant assistance in illustrations and CLIP experiments. Gratitude is extended to Jiajun Liang and Haoqiang Fan for helpful discussions. Thanks Xiangyu Zhang for hosting the Generative Model Special Interest.
10

---

Published as a conference paper at ICLR 2025
Appreciation is also given to Hung-yi yi Lee for developing and offering his engaging machine learning course.
Group.
The work was supported by National Science and Technology (2033ZD0121300),
Major Project of China
REFERENCES
Sam nond-r Adam Leach, Yang Long, and Chris G Willcocks. Deep generative modelling:
A comparative review of vacs, gans, normalizing flows, energy-based and autoregressive models. IEEE transactions on pattern analysis and machine intelligence, 2021.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity
Natural Image Synthesis. arXiv:1809.11090 2019.
Tom В Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint wxXv::2055.14165 2020.
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE signal processing magazine, 35 (1):53-65, 2018.
Prafulla Dhariwal, Heewoo Jun, Christine Payne,Jong Wook Kim, Alec Radford, and llya Sutskever. Jukebox: generative model for music. arXiv preprint arXiv:2005.00341, 2020.
Patrick Esser, Robin Rombach, and Björn Ommer: Taming Transformers for Hiigh---Reluttiin Image Synthesis. arXiv:20/2.09844 2021. http:tp://// http//axx......aarr./../. 2012. 09841
Kartik Goyal, Chris Dyer, and Taylor Berg-Kirkpatriin Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis-Hastings. wrXiv:2106.02736, 2021.
Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez. Bayesian flow networks. arXiv preprint arXiv:2308.070037 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. IEEE CVPR, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020.
Justin Johnson, Alexandre Alahi, and Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pp. 694-711. Springer, 2016.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Proc. NeurIPS, 2022.
Sourabh Katoch, Sumit Singh Chauhan, and Vijay Kumar. A review genetic algorithm: past, present, and future. Multimedia tools and applications, 80:8091-8126, 2021.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. ICLR, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible 1x1 Convolutions. NeurlPS 31, 2018.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
11

---

Published as a conference paper at ICLR 2025
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11461-11471, 2022.
2022.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.00 010077 2021.
Nithin Gopalakrishnan Nair, Anoop Cherian, Suhas Lohit, Ye Wang, Toshiaki Koike-Aking Vishal M Patel, and Tim K Marks. Steered diffusion: A generalized framework for plug-and- and- play conditional image synthesis. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20850-20860, 2023.
Huy Phan. github.ccom/huyynnaaa//yyyy CIFARRI,, January 2021. UtL/tttp:///cc.oo
(O.1/28//eee 1e181//zeodo..o www.zeeooo...44434 4431043
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. In ICLR, 2016
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- $748-8766 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. ArXiv, abs/2204.06125, 2022. URL https: 248097655
//api. www.smtticccoollr..rrreoooooo cotp::/ooorrooooo:::::: 24:0099
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Com- puler Vision and Pattern Recognition (CVPR), pp. 10674-10685, 2021. URLhttts:://st
semant esccolas ar. :trr://oorusS::2453333
Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: Image-to-inet -mmage diffusion models. ACM SIG-
GRAPH 2022 Conference Proceedings, 2021. URL pattss://aple sew...e.mmttsscooll...........
borg/CrpussTi TD: 243938678
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379 1948.
Assaf Shocher, Amil Dravid, Yossi Gandelsman, Inbar Mosseri, Michael Rubinstein, and Alexei A Efros. Idempotent generative network. In ICLR, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In ICLR, 2021.
Bisio Tom. Decoding the Dao, Nine Lessons Daoist Meditation. Outskirts Press, 2013.
Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, koray kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional Image Generation with PixeICNN Decoders. NewrlPS 29, 2016.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel Recurrent Neural Net- works. In ICML, 2016. ISBN 978-1 1-5100822990--8-
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing, 2018.
Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or, Sketch-guides text-to-image diffusion models. ACM SIGGRAPH 2023 Conference Proceedings, 2022. URL nattps://aple
semanti www.nntisschllooo www..rro./ceeee orw.cco....r..oor//ooooss//////.
12

---

Published as a conference paper at ICLR 2025
Long Tung Vuong. Trung Le, He Zhao, Chuanxia Zheng, Mehrtash Harandi, Jianfei Cai, and Dinh Phung. Vector quantized Wasserstein auto-encoder. In Andreas Krause, Emma Brun- skill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Pro- ceedings of the 40th International Conference on Machine Learning, volume 202 of Proceed- ings of Machine Learning Research, pp. 35223-35223 352242.. PMLR, 23 Jul 2023. URL https:
//proceedings.. mlr.
wwwp://wwwwoo.aa..ooo html
Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-oos--ree load balancing strategy for mixture-of-experts. arXiv preprint arXiv:2408. I 15664, 2024.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint (..iv:2222000444 2022.
Will Williams, Sam Ringer, Tom Ash, David MacLeod, Jamie Dougherty, and John Hughes. Hier- archical quantized autoencoders. Advances in Neural Information Processing Systems, 33:4524- 4535, 2020.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-free energy-guided conditional diffusion model. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 23174-23184, 2023.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control text-to-image diffusion models. ArXiv, abs/2302.055533, 2023. Dtc/CorpusID::2562771
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features Perceptual Metric. In IEEE CVPR, 2018.
A EXPERIMENTAL RESULTS ON GENERATIVE PERFORMANCE
We present additional experiments to verify the generative and reconstructive capabilities of Discrete Distribution Networks (DDN).
Conditional DDN for Image-to-Image Tasks. the domain of image-to-image tasks, we extend our Discrete Distribution Networks (DDN) to conditional setting, resulting in the conditional DDN. The image condition is fed directly into the network during training at the beginning of each stage. The condition serves informative guide, significantly reducing the task's generative space and, consequently, mitigating the modeling task's ' complexity. Through this conditional design, the DDN leverage the abundant information contained in the conditions to generate detailed images as shown in fig. g
accurate and
Verify whether DDN can generate new images. As depicted in fig. we compare the images that are closest in the training dataset, FFHQ, to those generated by our DDNs. It suggests that our DDNs can synthesize new images that, while not present in the original dataset, still conform to its target distribution.
Demonstration of Generation and Reconstruction Quality. Figures 12 to 14 illustrate the results of generation and reconstruction in various ablation experiments using the DDN model.
Efficacy of Split-and-Prune and Chain Dropout. series of experiments were conducted to sep- arately investigate the effectiveness of the Aplit-and-Prune and Chain Dropout methods. To isolate the impacts of these two algorithms, we simplified the experimental conditions much as possible, base, setting K = 8 and L, = 10, and disabling Learning Residual.
using the MNIST dataset The generated image quality under three different settings is displayed in fig. TO The results demon- strated that the Split-and- Prune method is indispensable. leading to significant improvements in the quality generated images. Meanwhile, the Chain Dropout method was found to alleviate the poor results observed when the Split-and- Prune method was not implemented.
Dolit-and-Pruma In algorithm I the Split and Prune operations are executed simultancously. During implementation. the newly split nodes occupy the positions of the pruned nodes in the tensors, ensuring that the total number of network parameters remains constant and the GPU memory usage is fixed. It is important note that the algorithm must not only operate
Implementation Details
13

---

Published as a conference paper at ICLR 2025
Guided by:
Condition
GT
10
LES
LLS
LES
ELSS
LES
ILES
Figure 8: Conditional DDN performing coloring and edge-to-RGB tasks. Benefiting from the reduction of the generative space by the condition, DDN is capable of generating high-quality im- ages 256 256 resolution. Columns and display combination of conditional generation and ZSCG, the generated results under the guidance of other images, where the produced image strives to adhere to the style of the guided image as closely as possible while ensuring compliance with the condition.
14

---

Published as a conference paper at ICLR 2025
Figure 9: Nearest neighbors of the model trained on FFHQ The leftmost column presents images generated by the DDN. Starting from the second column, we display the images from FFHQ that are most similar to the generated images, as measured by LPIPS ZhiPS/Zang et al. (2018).
15

---

Published as a conference paper at ICLR 2025
O
9
O
5
2 5 9 3
6
3
(
O
O
J 4
7 6
0
6
O
5
7
7 O
3
O
2
6
4 S 6 K 6 2
4
2
8
O
3
C
2 5
2 O
2
/ 8
7
O
7
8
8
D
9
O
2
O 7
O
7
5
4
5
5
6
7
/
I
O 9 4
3
9
9
O
7
6
3
7
7
2 3 2
8
4
8
4
4
9
B 5
8
3
O
6
2
3
(a) With Split-and- -itt-nnd-Prue,, set Chain
(b) Without Split-and- Prune,
set
(c) Without Split-and- -Prune, set
Dropout
Chain Dropout 0
Chain Dropout 0.05
Figure 10: Efficacy of Split-and-Prum aannnnddd---------- and Chain Dropout on MNIST.
(a) DDN (Ours), FID=35.4
(b) Idempotent Generative Network (IGN), FID=39
Figure 11: Comparison of randomly generated images on CelebA-HQ 64x64. DDN produces
images with clearer details and fewer artifacts compared to IGN Shocher et al. (2024).
on the output parameters but also apply the same operations to the corresponding state variables in the optimizer. This ensures consistency between the parameters being optimized and their associated state information throughout the training process.
Comparison with other novel generative models. We conduct qualitative comparison with a recent work, the Idempotent Generative Network (IGN) Shocher al. (2024), accepted by ICLR 2024. Since IGN was only experimented on the CelebA dataset and did not release its code, our comparison is limited the CelebA dataset. As depicted in fig. 11 DDN demonstrate better generation capabilities over IGN.
B FURTHER DEMONSTRATIONS ON ZERO-SHOT CONDITIONAL GENERATION
this section, we present additional experimental results on Zero-Shot Conditional Generation (ZSCG).
Utilizing CL.PP as Conditioning Guidance. As illustrated in fig. 15 we use CLIP Radford et al. (2021) along with the corresponding prompts conditions. We use DDN model, which exclu- sively trained on the FFHQ dataset, to yield Zero-Shot Conditional Generation (ZSCG) results. The results indicate that, DDN generate corresponding images under the guidance of CLIP without the necessity for gradient computations.
ZSCG with Multiple Conditions. In fig. 16 illustrate the operation of ZSCG under the com- bined action of two Guided Samplers: Inpainting and CLIP. Each sampler operates under its own specific condition. The inpainting sampler utilizes mask cover the areas where the CLIP prompt
16

---

Published as a conference paper at ICLR 2025
(defouit) (HI143.1)
K-64 (FID 47.0)
K-8
K=2 (Ta))-DDNN) 66.5)
w/o Split-and-Prune 55.3)
Chain Dropout (FID 182.3)
w/o Learning Residual (HID 56.2)
wlo Leak Choice 56.0)
Figure 12: Illustration of the random sample generation effects as part of the ablation study on our DDNs model.
Nodes/level
Level
Representation space
Validation accuracy!
K
K
128
1024
2
10
1024
65.9
77.3
85.8
86.9
8
512
69.0
81.1
8
5
3.3 x 10*
67.5
79.1
87.5
90.5
8
1.1 10
58.6
75.3
84.9
89.0
64
10 1.1 x 10
52.5
70.4
80.9
86.3
Table 3: Fine-tuning DDN latent as decision tree on MNIST. Constructing a decision tree based on the latent variables from the DDN and fine-tuning on MNIST trainning set. We report the validation set accuracy of the decision tree after majority voting for class prediction with varying number of training samples: 128, 1, 1,024, 10,000, and 50,000 (the full training set).
acts. Specifically, "wearing sunglasses" masks the eyes, "wearing a hat" masks the upper half of the face, and "happy person" masks the lower half.
For each Discrete Distribution Layer (DDL), both samplers assign a rank to every generated image, corresponding the degree of match their respective conditions- the better the match, the higher the rank. We apply a weight to each sampler, which represents their influence on the final assigned ranking. In this case, both samplers have a weight of 0.5. To promote diversity in the generated samples, we randomly select one image from the top two ranked generated images, serving as the output for that DDL layer.
C LATENT ANALYSIS
Semantic Performance of Latents. We explored the semantic capabilities of DDN latents through classification experiment on the MNIST dataset. Given the inherent tree structure of DDN's latents, we employed a decision tree classification method, using fine-tuning data to assign class votes to nodes in the tree. For unassigned nodes, their class is inherited from the closest ancestor node with an assigned class. We fine-tuned DDN's latent decision tree using various numbers of labeled training set data, and the results the test are shown table All experiments in this table were
17

---

Published as a conference paper at ICLR 2025
GT
5-5 (FD 26.0)
K=64
(FID 32.3)
Km8 (FID 40.9)
K-2(taj--DDN) (iFID
Sylit-and-Prune (FFD 31.2)
wio Chuin Dropout (FID 26.5)
w/o Learning Residual (nFID 40.2)
wlo Leak Choice (FIT 34.3)
Figure 13: Demonstration of the reconstruction capability of our ablation study model FFHQ- 64x64, which can be interpreted as the model's fitting ability on the training set.
18

---

Published as a conference paper at ICLR 2025
GT
K=512(default)
33.2)
K-64 (FFID 38.7)
K=8 (FDD 49.8)
K=2(Tajj-DON) (FRD 70.6)
wio Split-anadd Prune (FFID 34.7)
wHo Chain Dropout (FFD 37.4)
wro Learning Residual GFID 40.2)
wlo Leak Choice (FITD 32.2)
Figure 14: Demonstration of the reconstruction capability of our ablation study model on CelebA- 64x64, which can be interpreted as the model's generalization ability on the test set.
American
Bostthd
yeeng
nerg
Bandrume
bapar
bey
Figure 15: Shro-Shot Conditional Generation guided by CLIP. The text at the top is the guide text for that column.
19

---

Published as a conference paper at ICLR 2025
Guided
"Weciing sunghassee"
Fyes masked
"Werring
Upper fize mesked
"Smiling person"
Lower face masked
Figure 16: Zero-Shot Conditional Generation under the Influence of Multiple Conditions. The DDN balances the steering forces of CLIP and Inpainting according to their associated weights.
conducted with Recurrence Iteration Paradigm's 's UNet, which has approximately 407k parameters. These experiments substantiate that DDN's latents encompass meaningful semantic information.
More Comprehensive Latent Visualization. fig. demonstraaes more comprehensive distribu- tion of samples that correspond to the latent variables.
D MORE DETAILED EXPERIMENTAL EXPLANATION
In fig. 20 have expanded the "Illustration of the principle behind the Split-and- operation" by providing a schematic when increases to 15. This demonstrates that a larger sample space yields approximation closer to the target distribution.
In the caption of fig. 17 we have detailed the experimental parameters for "Toy examples for two- dimensional data generation". Additionally, will explain why the KL divergence in model is lower than that found in the Real Samples.
E LIMITATIONS AND FUTURE WORK
There are some key limitations of Discrete Distribution Networks (DDNs):
The KL output space is insufficiently large to represent complex distributions:: We currently developing new approach to enhance the efficiency of high-dimensional data representation by expanding the output space from KL. This involves dividing an image into patches, independently selecting the optimal patch from K candidates for each patch. These selected patches are then merged into complete image, which serves as the output of the current DDL layer and input into the next layer. Consequently, this increases the output space to N-I.
The Split-and- Prune algorithm continuously discards trained parameters.: Discard- ing parameters that have undergone extensive training is not a wise choice, particularly when scaling up the model. The goal of the Split-and aplit-and-Prune algorithm is to balance the fre- quency at which each node is sampled during training, similar to maintaining load balance among experts in Mfxture-of-Fxxere to address this issue the Loss-Free Balancing method Wang et al. (024). After comput- ing the L2 distance between each node's output and the ground truth, -oss-Free Balancing first applies node-wiss bias to these distances. By dynamically updating the bias of each node based on its recent load, this method ensures that nodes with lower sampling frequen- cies receive a lower bias, thereby increasing their chances of being sampled. Consequently, Loss-Free Balancing helps maintain balanced distribution of node sampling without dis- carding parameters.
(MoE) models Liu et al. (2024). A potential solution
Loss of high-frequency signals:
The high level of data compression and the use of pixel 1.2 loss during optimization may result in the loss of fighfrreu signals, causing the
20

---

Published as a conference paper at ICLR 2025
Spirals
Blurs
Code
Strings
GT Density
Discrete Distribution Networks
Gradient Descent
tht uhions
20.19
DKi 15.79
GD With Split Split-and-Ph dit--and-Prune
Discrete Distribution Networks
DEL- 0.11
DKL 0.14
DxL 0.12
DKL 0.05
Real Sumples
Discrete Distribution Networks
0.21
DKL 0.24
DEL 0.29
DKL 0.16
Figure 17: Toy examples for two-dimensional data generation. The numerical values at the bottom of each figure represent the Cullback-Leible divergence, DKL- Due to phenomena such as "dead nodes" and "density shift". applying Gradient Descent alone fails to properly fit the Ground Truth (GT) density. By employing the Split-and- Prune strategy, the density map looks the same as Real Samples. In the experiment, we use K 10, 000 discrete nodes emulate the probability distribution of the GT density. Each node encompasses two parameters, æ and y, initialized from uniform distribution. Each experiment consists of 10 x K 100, 000 iterations, wherein each iteration, L2 loss is calculated based only on the node closest to the GT. To compute the KL divergence, the GT density map is converted into a discrete distribution with bins size 100 x 100, which is then used to caleulate the DKL against the discrete distribution represented by these nodes. Notably, the KL divergence of Split-and- and-Prune is even lower than that of the Real Samples. This because algorithm has been exposed to 10 K samples of the GT distribution, thus it better reflects the GT distribution compared to the Real Samples, which are drawn only K times from the GT distribution.
21

---

Published as a conference paper at ICLR 2025
A MX EREEE AMEUS.
From the Taiji generales the Two Poles: From the Two Poles generate the Four Symbols; From the Four Symbols generate the Eight Trigrams; And from the Eight Trigrams generate the Myriad Things.
I Ching: The Book of Changes
Wu J to Tal Ji the Eight Trigrams and Ultimanaly the 10,000 Things
Famous quote and its translation about Taiji
(b) Tree structure of Taiji
Figure 18: The Taiji-DDN exhibits a surprising similarity to the ancient Chinese philosophy of Taiji. Records of Taiji can be traced back to the Ching (Book of Changes) from late 9th century BC, often described by the quote the left (a) that explains the universe's 's generation and transformation. This description coincidentally also summarizes the generation process and the transformations in the generative space of Taiji-DDN. Morcover, the diagram (b) from the book Tom (2033) bears a closely resemblance to the tree structure DDN's latent fig. 1b Therefore, we have named the DDN with K 2 as Taiji-DDN.
images to appear blurred. A potential improvement could be learning from VQ-GAN Esser [ et al. (202T) and incorporating adversarial loss Creswell al. (2018) to enhance the mod- eling of high-frequency signals.
Computational burden of zero-shot conditional generation: ZSCG requires L x K for- ward passes through the guided model, where L the number layers and K is the number of possible outputs per layer. When the guided model itself is computationally ex- pensive, this results significant computational overhead and prolonged generation time. However, since the discrimination process is parallelizable across the K-dimensional out- put space, batching techniques can be employed to mitigate latency. addition, it is not necessary for every layer to be guided by the condition signal, further research will be conducted to reduce the number guided model calls during the ZSCG process.
22

---

Published as a conference paper at ICLR 2025
7
1373939831
7
4
5
2
5
Figure 19: Hierarchical Generation Visualization of DDN with L 4. We trained DDN with output level L and output nodes K per level on MNIST dataset, its latent hierarchical struc- ture is visualized recursive grids. Each sample with colored border represents intermediate generation product. The samples within the surrounding grid of each solored-bordent sample are refined versions generated conditionally based on it (enclosed by the same color frontier). The small samples without colored borders are the final generated images. The larger the image, the earlier it the generation process, implying a coarse version. The large image the middle is the average of all the generated images. The samples with blue borders represent the outputs of the first level, while those with green borders represent the 82 64 outputs of the second level. It can be observed that images within the same grid display higher similarity, due to their shared "ancestors". Best view color and zoom in.
23

---

Published as a conference paper at ICLR 2025
divegennofPPl 0..0.34
diviengencerP P
1.2
1.2
122
1.0
0.8
0.8
0.8
0.4
0.4
0.4
0.5
0.5
0.0
f1
01
0.20
0.20
0.20
0.20
0.20
0.17
0.17
0.11
0.17
0.20
81 2.200:20
0.20
0.20
[P-0F
0.37
[P-0j
0:13
0.57
0.08
0.07
0.33
[P-0]
5.07
0.23..03
0.04
0.17
-1.0
-0.5
-0.2
0.2
0.5
1.0
-1.0
-0.6
-0.2
02
0.5
1.0
-1.0
-0.6
-0.2
02
0.
10
(a) Initial, K=5
(b) Split, K=5
(c) Prune, K=6-55
KLdtrargancs/ 0.01
KL (Ldderganccs 0.39
dirargenca(" IJC 0.1
1.2
1.2
3.2
1.0
1.0
1.0
0.8
0.6
3.5
3.6
52
02
52
0.20
200 180223
0.29
0.07
0.87
0:30
20h 223-2
D.2
0.000.01..00
|P-0|
|P
Final, K=5
(c) Initial, K=15
(f) Final, K=15
Figure 20: Illustration of the principle behind the Split-and-Prune operation. For example in (a), the light blue bell-shaped curve represents a one-dimensional target distribution. The 5 "1" under the x-axis are the initial values from a uniform distribution of 5 output nodes, which divide the entire space into 5 parts using midpoints between adjacent nodes as boundaries (i.e., vertical gray dashed lines). Each part corresponds to the range represented by this output node on the continuous space x. Below each node are three values: P stands for the relative frequency of the ground truth falling within this node's 's range during training; Q refers to the probability mass of this sample (node) in the discrete distribution output by the model during the generation phase, which is generally equal for each sample, i.c., 1/K. The pattom-most value denotes the difference between P and Q Colorful horizontal line segments represent the average probability density of P, Q within corresponding intervals. In (b), the Split operation selects the node with the highest P (circled in red). In (c), the Prune operation selects the node with the smallest P (circled in red). In (d), through the combined effects of loss and Split-and-Prune operations, the distribution of output nodes moves towards final optimization. From the observed results, the KL divergence (K (()))) consistently decreases as the operation progresses, and the yellow line increasingly approximates the light blue target distribution. Finally, we show the distributions of the initial and final stages when the number of output nodes K = 15 in (e) and (f). Due to the increased representational space, the generated probability distribution Q. represented by the yellow line segment in (f), is closer to the light blue target distribution than in the case of K 5.
24


Reviews:
[{"review_id": "iX2q1x3Lbk", "reviewer": "Reviewer_tvs8", "summary": "The paper proposes to generate samples in a sequential way: pass the output of the previous module into a function randomly selected from a set of K functions in the current module, with the initial input to the first module being a zero vector. To train each module in the space, one first collect a trajectory with these modules except that one does not do random sampling out of K functions but pick the one with the output closet to the data sample $x$. With this trajectory, one computes L2 distances between the output of each module and $x$ and optimize it. The paper shows some empirical results on the CIFAR-10 dataset.", "questions": "See weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed method is simple and seemly intuitive, and has been experimented on small-scale datasets like CIFAR-10.", "weaknesses": "Purely based on the presentation of the paper, I find the proposed method neither theoretically attractive, as it does not explicitly model probability distributions or model some complex distributions, nor empirically useful due to its worse performance compared to simple baselines like DCGAN. Indeed, the paper claims that VQ-VAE is unsatisfactory (e.g., in the abstract) but it does not even compared against VQ-VAE.\n\nIf we take a closer look into the proposed method, it seems to me not too different from a slightly-different VQ variant of diffusion models (e.g., https://arxiv.org/abs/2111.14822) -- running discrete diffusion (as the proposed model is basically trained in a similar way) on a latent space represented by a codebook of features, probably with some additional gradients of hierarchical latent spaces. The paper fails to connect the proposed method to a lot of existing papers and explain 1) what is different and novel given these existing methods and 2) why these design choices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qdUWrK2c55", "reviewer": "Reviewer_1dyp", "summary": "This work proposes Discrete Distribution Networks (DDN) a new class of image generative models. Discrete Distribution Layers within a DDN output multiple images at a time, and one image is sampled and concatenated as a feature to input to the next layer in an autoregressive fashion. During training, the loss is a reconstruction loss of the output sample of each DDL that best matches with the input image. Several tricks are provided to help optimization. During sampling, external signal can be provided without using gradients in order to condition generation. Qualitative and quantitative results are provided for unconditional CIFAR-10, FFHQ, and Celeb-A generation in addition to qualitative results for MNIST.", "questions": "* Can DDN scale to larger and higher resolution datasets, such as FFHQ 256x256, or ImageNet 256x256? \n* What are the parameter counts of the baselines compared (DC-GAN, IGEBM, VAE, etc.)? DDN benefits from more layers since it can refine more, so the baselines should have similar parameter counts. \n* How does a single shot DDN compare against a default DDN allowing for the same number of refinements?\n*  Generating multiple samples in parallel for sampling is expensive. What are the memory requirements compared to the baselines in the paper?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The idea of sampling an image and autoregressively feeding it back in through each layer in a forward pass is novel to the best of my knowledge. \n* The ability to condition generation on external signals without gradients is a useful property. \n* DDN is showcased for a variety of applications such as inpainting, colorization, denoising, super-resolution, CLIP-guided editing. \n* A variety of tricks are provided to help training, such as the Split-and-Prune algorithm, chain dropout, residual learning, leak choice.", "weaknesses": "* I understand that this paper serves as a proof-of-concept for a new type of generative model, but the model is validated on small scale datasets (e.g., CIFAR-10, MNIST, CelebA, FFHQ-64x64). Furthermore, it does not compare against more modern approaches such as diffusion models, and quantitatively lags behind DC-GAN. However, I do not think it is fair to hold this novel method to the same standard as more matured, modern approaches. \n* The method relies on generating multiple samples in parallel and requires multiple layers for further refinement. This seems quite expensive and hard to scale to higher resolutions. \n* The placement of tables and figures reduces the quality of presentation. For instance, Figure 9 has a large white space above it. Table 2 is presented on page 8 even though ablations are not addressed in text until page 10. Figure 5 showcases an experiment which is not addressed in the main text of the paper. So it would be more fitting in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h0B0GaonHv", "reviewer": "Reviewer_r4YK", "summary": "The authors propose a new generative model, based on generating K examples at each layer and choosing the one closest to a ground-truth instance. This formulates a discrete hierarchical distribution. At inference, taking a random node at each layer provides a random sample. Both conditional and unconditional generation are demonstrated.", "questions": "While has some notable weaknesses, I think this is a very good paper that can open a door to new directions in generative modeling. It is important for me to point to the courage of trying new refreshing approaches (as opposed to building upon current trends). Such papers should always be judged under the understanding that for existing methods, a lot of engineering has taken place and should be thought of as the first GAN paper or the first Diffusion models paper (Sohl-Dickstein 2015).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. I find the method novel and elegant.\n2. The ability to produce visualization of the hierarchical generation (figs 8, 18) is a very enlightening feature.\n3. The authors propose some practical techniques to deal with this non-differentiable sampling. The proposed \"split-and-prune\" trick is clever and elegant.\n4. The novelty is very strong, and this should not be overlooked. This is a whole new method, very different from any of the existing generative models.", "weaknesses": "1. Theoretical analysis is missing. No mathematical derivation that shows why the distribution of generated images should converge to the real data distribution.\n2. Eventually, if I understand correctly, for unconditional generation, the model has a finite number of specific examples it can produce K^L. Even if this number is big, this suggests that the model essentially is a compressed way of storing all its possible results in a tree based database. This might make a problem to scale up. Since the demonstrated data relatively has few dimensions, this suggests that actually holding the entire dataset might require more or close storage to the model itself. A quick calculation to demonstrate: MNIST has 70k images (train+test) of 28x28, this is 54880000 integers: ~55MB. EDM smallest model has 62M parameters, even if we take these as 1 byte each it is heavier.\n3. I am adding the weakness of low quantitative results and lack of demonstration of higher scales. To be clear, I think it is legitimate for novel methods but it is still a weakness. (i.e. if this model would have produced SotA generation results it would have been rated higher).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)