Unified Discrete Diffusion for Simultaneous Vision-Language Generation
Authors: Minghui Hu
Chuanxia Zheng
Zuopeng Yang
Tat-Jen Cham
Heliang Zheng
Chaoyue Wang
Dacheng Tao
Ponnuthurai N. Suganthan
Venue: ICLR 2023 poster (2023)
Keywords: Multi-modal
Image generation
Image Caption.

Abstract: The recently developed discrete diffusion model performs extraordinarily well in generation tasks, especially in the text-to-image task, showing great potential for modeling multimodal signals. In this paper, we leverage these properties and present a unified multimodal generation model, which can perform text-based, image-based, and even vision-language simultaneous generation using a single model. Specifically, we unify the discrete diffusion process for multimodal signals by proposing a unified Markov transition matrix and a unified objective. Moreover, we design a multimodal mutual attention module to highlight the inter-modal linkages, which is vital for multimodal generation. Extensive experiments indicate that our proposed method can perform comparably to the state-of-the-art solutions in various generation tasks.

Published as a conference paper at ICLR 2023
UNIFIED DISCRETE DIFFUSION FOR SIMULTANEOUS VISION-LANGUAGE GENERATION
Minghui Hu, Chuanxia Zheng', Tat-Jen Cham & P. N. Suganthan'
Nanyang Technological University. TUniversity of Oxford, +Qatar University (e200008, ASTJCham]@ntaa edu.sg. exzhengerobots. ox ax.accuk, w.t. a.r.sagantthaeeu.eed
Zuopeng Yang Heliang Zheng, Chaoyue Wang & Dacheng Tao
JD Explore Academy, SShangaai Jiao Tong University yzpenges edu. chaoyue. wang@outlook com, dacheng. t gmall. com
edu. en, zhengheliame janiein.gg..oooo com,
ABSTRACT
The recently developed discrete diffusion models perform extraordinarily well in the text-to-image task, showing significant promise for handling the multi- modality signals. In this work, we harness these traits and present unified mul- timodal generation model that can conduct both the "modality translation" and "multi-modality generation" tasks using single model, performing text-based, image-basets and even vision- laiguuge simultaneous generation. Specifically, unify the discrete diffusion process for multimodal signals by proposing a uni- fied transition matrix. Moreover, we design mutual attention module with fused embedding layer and a unified objective function to emphasise the inter-modal linkages, which are vital for multi-modality generation. Extensive experiments indicate that our proposed method can perform comparably to the state-of-the-art solutions in various generation tasks.
INTRODUCTION
Fuily Mask
this grey with long. pointy beal..
black has pointy beak.
glasa
brown
pointy beak:
Parzally Maskk
thos hard to bloe with
this bird bright woth black belly and has very short
ominge vivel lllhh table
orange filled with the yellow
Jhin bird black with beawn and pointy beale.
this buni is black brown and howa leng, pointy beak.
black tou and has
Averyy sboot beale.
(a) Cross Modal
Image Conditioral
(c) Text Conditional
Figure 1: Examples of various tasks supported by UniD3. The dark brown portions of the image and description represent the [MASK].
Diffusion models (Ho et al., 2020; Song al., 2021b) have gameered significant interest various high quality conditional image generation tasks, such as image super-resolution (Rombach et al., 2022), image inpainting (Lugmayr et al., 2022), image editing (Avrahami al., 2022), image trans- lation (Saharia et al., 2022a), among others. Concurrently, the Vector Quantized (VQ) models have also achieved rapid advances in image generations, especially cross-modal tasks, examples in-

---

Published as a conference paper at ICLR 2023
Quantised and Fision Embeddingg
Urtbedding Space 24
Unified Denoising Dilliason
Image Encuder
Firecil Einlscchin
Maya
Dillason Ploeess
Dillnsion Pracess
Visticnary
close up ofa ginifees
Diniiingg Process
Denoising Proness
United Tannfomee with Manak Abenton
Deccuple
Imagee Decoder
XE2
Text Deculer
Compoed Imogee Tosce
Comppied
[MASK]
Figure 2: The pipeline of UniD3. With an offline model (red background part), the given inputs represented by discrete token sequence in separate domain. The fusion embedding concatenate the tokens in different modal and embed them to the same space. The unified diffusion (in blue background) will construct the joint distribution of all modalities based on the fused embedding with fixed unified Markov transition matrix.
clude text-to-image (Kim al., 2022), sketch-to-image (Esser et al., 2021b), image-to-video (Wu et al., 2021).
Despite their success, all of these generation tasks are designed for only a single special modality, i.e. either modality translation or modality generation, with the help of powerful diffusion model. In particular, the former modality translation translates the given conditional signals into the traget domain, while the latter modality generation only works for the unconditional image generation, e.g. image generation on CelebA Karras et al. (2017) or LSUN Yu al. (2015) datasets. However, none of them consider to learn join distribtuion for the mixture modality.
Here, we take the use discrete diffusion model into a new realm mulli-modaliity generation using unified vision-language model. In contrast the modality translation tasks outlined above, our multi- multi-modality generative model does not require any conditional signals given in prior and is capable of simultaneously generating content pairs with the associated multi-modalities. UniD3, our new Unified Discrete Denoising Diffusion model, allows us to construct joint vision-language probability distribution by mixing discrete image tokens with text tokens, leading to capability of simultaneously generating cross-domain results. This is achieved by the two-stages framework, illustrated in Fig. 2, (1) An offline model to generate a compact yet expressive discrete representation for both images and texts (the pink part in Fig. 2). (2) A novel unified discrete diffusion model to estimate the joint distribution of such latent visual and language codes (the cyan part Fig. 2). Once trained, UniD3 can not only inherit the ability to manipulate the provided text or image, but is also able to unify the text and image generation, e.g., unconditional vasion-language pairings generation, cross modal manipulation, text guided image completion, and image conditional text caption (Fig. depicts the tasks that our model is capable of handling).
Based on the empirical exploration, our model can achieve comparable image quality to the text condition in the unconditional case. terms of image caption, our model also comparable to some states of the art.
In summary, our key contributions include the following:
We design a specific Markov transition matrix for our unified discrete denoising diffusion model, which lead to sophisticated control of diffusion process, estimate the joint
2

---

Published as a conference paper at ICLR 2023
Unified Block
Mumal Attention
Iuage Sequencc
Fised Hiddeen
Tapui
Feed forward Layer
Fipedd Hodden State
Sell Atteation
Condition
Decouple
Couple
Condition
Input
Text Sequence
Figure 3: Illustration of transformer blocks with mutual attention. unified transformer composed of several blocks stacked on top of one another.
distribution of language and image. The purposive design of transfer matrices based on task objectives and data properties is also pioneering for discrete diffusion models.
We additionally propose mutual mechanism with fuse embedding to fulfil the objective of multi-moda integration. And we alter unified objective function offer the optimization with more concise restrictions.
To the best of our knowledge, the UniD3 the first work to solve both multi-modality generation and modality translation problems that capable of handling the simultane- ous unconditional vision-language generation and bi-directional vision-language synthesis tasks with single model.
PRELIMINARIES
Vector Quantised Model The Vector-quantiscd Variational Auto Encoder (VQ-VAE) (Van
Den Oord et al., 2017) learns to embed the high-dimensional data, e.g., image or audio, into a discrete representation. In particular, given high dimensional data CC the encoder E first converts to the spatial latent features z = } E I dxxxx and then transfer this continuous features into discrete space by looking up the closest features in the codebook 2 = {zx} E RRxx to obtain the tokens zq:
HXW
Zg = Quantisc(2)
Quantiso == arg ming |2,,, z ll.
(1)
where the dimensions h, w of latent feature = are substantially smaller than the original dimensions H, W. The reconstructions can be obtained through decoder G:
a- =((z) ((Quantise(E(()))
(2)
Recently, there have been significant advances in learning more compact representation, yet higher- quality reconstruction, such as introducing new losses (Esser et al., 202 applying powerful back- bone (Yu al., 2021), using multiple channels representation (Lee et al., 2022) and spatially nor- malization (Zheng et al., 2022). However, in this paper, we focus applying the technology in our novel UniD3, rather than exploring a new codebook learning approach.
Discrete Diffusion The discrete diffusion model was originally mentioned in (Sohl-Dickstein al., 2015), with transitions converging to binomial distribution. Subsequent work extended this to multinomial diffusion (Hoogeboom et al., 2021; Song et al., 2021a), while Austin al. (2021) provided more options for transition matrices. Recent works integrated discrete diffusion models with VQ- VAE, allowing for high-quality image synthesis (Gu et al., 2022; Hu al., 2022).
Here we briefly describe the multinomial diffusion with the absorbing state, as employed in VQ- Diffusion (Gu et al., 2022). Besides K tokens from discrete VAE, an additional [MASK] token is introduced. The forward process is defined as:
( (32,|a0)-1)
Cat (2:; p Q. o. Q:2111,
(2:;;
(3)
3

---

Published as a conference paper at ICLR 2023
where is one-hot vector identifying the token index. Here [Qila g(2; i| R(K+1)((++)) is the Markov transition matrix from
5)
which can be expressed
Ot +
B
3
B
Be
+
B
B
[-1-+
Be
B
+
B
0
(4)
7
7t
where E [0, 1] is the probability of retaining the token, and has probability of n to be replaced by the [MASK]] token, leaving a chance of 3 = -0-ay at - 7r//K K to be diffused.
The posterior of the diffusion process can be formulated as:
q(24-1 mo) g( ((1r/(zz11
zo ( g (a)
Q::: (== x 2,000
(5)
... -1, which can be calculated in closed form, and given Q, = Q....
a xD) = Cat(# Q, = 2 Q,
(6)
In the reverse process, instead of explicitly predicting the posterior using denoising neural network, the Ey parameterisation might increase the stability and permit fast inference (skipping At steps per iteration). The reverse transition with reparameterisation is given
pa pa(2t-1(2;)) x Σ -g(x)-11zz To 2o (zo)
in which the neural network predicts the logits of the target data g(v)).
3 METHOD
Our goal is adapt the discrete diffusion model learn the joint distribution of linguistic and visual features concurrently. First, we propose transition matrix that allows the diffusion model to capture the implicit association between text and images. Second, we present mutual attention transformer architecture with fuse embedding layer as the denoising function and a unified objective function, which fits our unified diffusion process objective and permits for more precise predictions.
Our overall pipeline is illustrated Fig. 2. Specifically, our solution begins by compressing the figures and texts into discrete token sequences using dVVE and BPE, respectively. A robust diffusion model with unified transition matrix is then constructed to fit the joint distribution across different modalities, which is further empowered by a transformer with mutual attention mechanism.
3.1 UNIFIED DIFFUSION PROCESS
Unified Transition Matrix Diserete diffusion models with transition matrices can capture global links. The presence of transition matrix determines the nature of the discrete diffusion model, which also provides with more choices for token evolution. Thus we may wonder if feasible to design transition matrices that capture the global connections between various modalities.
The Markov transition matrix of the discrete diffusion model should satisfy the following require- ments: 1. each column in Q, should sum to one to conserve probability mass; 2. each column in the cumulative-produms Q, should converge to either a known stationary distribution or a learnt prior when becomes large.
On the basis of these criteria, we construct unified transition matrix [Q+]c,; q( = i|.;-1 capable of encapsulating discrete representations among various modalities. The following ma- trix Q E ( +*))(((++XKKK +1) illustrates unified transition process with only text and image
4

---

Published as a conference paper at ICLR 2023
modalities:
B1
B
0
0
Bt.
0t
+
Br
Br
Be
0
Be
B8
+
0
0
Q
0
0
+
31
87
+ 1
0
0
[8]
19
19
+ 1
Yt 7 7 7e T 7/ Th
(8)
where 04 [0, 1] is the probability to keep this token, 33 and B the probabilities of token to be replaced by any other accessible tokens in different modality, and 7o is the absorbing probability.
The dimension of the matrix Q is (K + KM 1) x (K K 1), where K and K are the number of states in different modals respectively, e.g., K is the size from the codebook in discrete VAE and K is the dictionary size of BPE.
The matrix comprises five sections:
The final row and column form section associated with the transition of the absorbing state. Intuitively, if the token is [MASK] at 1 step, then the token must be [MASK] at time Conversely, any other modal token has the equal possibility "/. being diffused to [MASK].
The remainder of the matrix is composed of four quadrants, the first and third of which are fully zeros. Specifically, these two sub-matrices prevent tokens transitioning from one modality to another. The form of the second and fourth quadrants closely resembles that for multinomial diffusion. Here, in addition to some probability of being converted a [MASK] token, each token also has some chance of transiting to different state within the same modality, remaining unaltered.
The dimensions of these four quadrants are not identical, which are K K, K K x K* and K x K" respectively.
is worth noting that a₂ and " are the same all modalities, whereas 3r varies based on the number of states in different modalities. Mathematically, 3r (1 (1-a;- -1) and 8 = (1-a;-7))a
The sum each column in this transition matrix one to preserve probability mass, and also all the mass of the stationary distribution falls on the [MASK] token, which satisfies the prerequisite for discrete diffusion model transition matrix.
The computation of Q, needed for g( (x)) in Eq. 6, can be efficiently obtained in closed form:
Q,4D =0,T0 + N22 ((xo)3, 1 (20033 @ [M] + L(x0)3, + 1'(x))3
if argmax ac E [0, K); where 1 (xxx) otherwise.
" 1x)))) = {
if argmax E [K, K. K*), otherwise.
[M] t argmax K + K and ar Br, T. are the corresponding cumulative product.
The detailed proof is provided Appendix E.
Unified Objective For conditional generation tasks such as text-to-image generation, the goal is
to maximize p(aly) by finding æ. where y the given text condition. our task, can be approx- imated that the model try to maximize the joint distribution p(æ, y) simultancously. In practice, minimize the Kullback-Leibe and text directions, as shown in the following:
divergence between q(x;-1 xt, 2o) and pe (w:-1 (w,)
in both image
Lo -Eg(e (ew) [log po ( (m9 |m,, m (xl) log (10 |x), m (mg))
(10)

---

Published as a conference paper at ICLR 2023
C1-1 (mn)
DKL (g(0)-1|<; xo)ll
po (अ;;-_ " xx)) pol:pp((e po(a) tert (x 1
(11)
where pa( (xxz11) is the integration of the logits po (1( and po (x){ from separate modal, and they can obtained with ao-paraaetterisation as following:
pa 1 xx))
q
smg
tut
po ( m [x))]s
(12)
200
smng
po (at Lin (e.) & Σ
q( [es abr mo
m )oo ( [x:)].
(13)
#00
Due to the po-pparameterisation, our model is also capable of achieving fast sampling by increase the step size. And the last term of the variational lower bound loss Cr is a constant and can be ignored during the training:
LT = DKL (g(2r (n)) ((((r)))
(14)
as the prior p(ær) is fixed with the unified transition matrix:
p(ær)) Br. Br, Fr
Yr
(15)
The full expression of the loss function can be found in Appendix C.
3.2 DENOISING FUNCTION FOR MULTIMODAL
Mutual Attention As indicated by Eqs. & 13, the neural network responsible for prediction of the distribution po (ão (z.). However, the input to the neural network covers all modalities, and a simple self-attention mechanism can scarcely highlight the inter-modal linkages. In addition, the distributions predicted by the neural network need to be decoupled according to the various modalities throughout the reparameterisation. other words, the network predictions should be expressed in terms of different modalities, e.g., (o (mm (x.) and po (#ffrt [xx), with interconnections.
Therefore, propose the mutual attention mechanism and construct the unified transformer as shown in Fig. 3. The unified transformer contains several transformer blocks, each of which consists of self-attention, two parallel mutual attention operations and one cerd-forward module. Each block receives sequence of mixed-modaa tokens as input that traverses layer of self-attention to capture the inherent connection within the modalities. Afterwards, the hidden tokens decoupled according to the location of the different modalities and fed to the corresponding mutual attention layers. Mutual attention modified version of the cross- coossattentioon layer, with the conditional inputs to cross-attention being maintained constant while the inputs to mutual attention are derived from hidden features. Next, the outputs from the various mutual attention are concatenated into one mixed-modal token sequence for transmission to the next block. At the end of the unified transformer, we layer-normalin normalise the sequence of tokens from the blocks, which is then decoupled to different predictive distributions by fully-connected layers.
In our model, each modality may become component what needs to be generated. Therefore, we propose mutual attention enable tokens of different modalities in sequence to be conditional on each other, allowing the capture of the relationships between the various modalities. Mathematically, mutual attention be expressed
Q,KT /2 V
MA(T, (T, softmax
where Q. - WoT,, K Wx & WvT, and T; & T, are the tokens in a different modality, e.g. image and text. The Unified block contains self-attention and several mutual attention in parallel. Given a hidden token vector H' = H HJ], the whole pipeline of Unified block is
1.7" = SA(H') 2. Decouple: T" + +T,
[.. MA(T), T7)
3.
Couple: T +- T..T
T, MA(T).T)

---

Published as a conference paper at ICLR 2023
with black crown and long beak.
clean kitchen
gizafic is standingg in forest
building with a park area.
bellyy spotted breast. short cail, and pointed hill
has a small yellow eyes. and white beak.
with woolen cullicts for
this bind how wings that pray and has white belly.
the small bind moeely brows and has light red head.
this bind has wings that black and has yel- low and black belly.
colorfal flosers in in onange vase the win- dow:
there a banch by the trees the seaside purk.
window on the fromt of beick building.
Figure 4: Generated vision-language Pairs from CUB-200 and MSCOCO. Both the image and cap- tion are generated simultaneously. The quality of the created photos and text is comprehensible, and there is correlation between the descriptions and the visuals.
Fused Embedding In addition, discrete tokens various modalities should be embedded in the same space. Considering that two modalities have K and K states respectively, we firstly create embedding layer with size K + K* + 1, with the additional state for the [MASK] token. We then create learnable spatial position encoding for the image modality and learnable sequence position encoding for the text modality. The final fused embedding is obtained by adding the embedded vectors the positional encoding of their associated modalities.
EXPERIMENTS
The description of the datasets and model experimental details can be found in Appendix B.
4.1 QUANTITATIVE RESULTS
In the unconditional scenario, our model is able to sample from all modalities while maintaining the relationship between them. For conditional generation, we conducted experiments on text-based image generation and image-hased caption generation, respectively.
Metrics We evaluate the multimodal capacity of our model in three distinct areas: Image sampling quality, text sampling quality and the similarity between sampled image and text. Fréchet inception distance (FID) (Heusel et al., 2017) and Inception Score (IS) (Salimans et al., 2016) are used to evaluate the realism and variety of the produced images. The BLEU-4, METEOR and SPICE are used assess the quality of picture caption. And CLIP (Radford et al., 2021) is used to identify the similarity between visuals and text.
Image Quality We quantitatively compare our unconditional and text-conditional results with sev- eral statc-of-ther text-to-image and bidirectional conditional generation methods, including GAN- based methods, diffusion-baces methods and autoregressive methods. The comparison results are given in Table in which Pair means the unconditional generated vision-language pairings and 72/ represents the text-to-image generation.
The first section of the table solely covers modality translation methods. In the second part, these models are claimed to be endowed with cross modal capabilities. However, such methods are not able achieve simultaneous vision-language pair generation and necessitate given signals. For fair

---

Published as a conference paper at ICLR 2023
Rel. Image
Misked Imase
Sample
metiiom gitta vanz
metiom white belly and evering.
aonodio find with wienpheneeseeee till and pepperr back wines and. cream Tell and svering
like timaa Mack beak, and belyy una evering,
belly and back and head, bell and evesing.
Figure 5: Presentation the results of cross modal vision and language manipulation and infilling. The dark brown portions of the image and description represent the [MASK], while the strike- through represent the caption manipulation. Image and text compliment each other simultaneously.
comparison, we exhibit the quantitative generative results based on various modalities individually. We provide more experimental results for text conditional generation in Appendix G.2
CUB
MSCOCO
B-4 (1)
M (1)
(
ETD (1)
15
FTD (1)
(1)
BUTD (Anderson et al., 2018)
27.7
DM GAN (2tu 2019)
16.09
4,75
52.64
27.3
ORT (Herdade et al., 2019)
38.6
28.7
22.6
LDM wio G (Romboch
DE GAN (Ibo et al, 2022)
VQ Diffusion (Go et 2022)
2022)
11.98
14.11
5.1
23.31
21.42
19:75
20.23
X-LAN (Pan 2020)
AoA (Huang al,, 2019)
38.9
39.7
29.5
29.3
23.4
LAFTTELF) (Zhou et 20221
27.53
4.32
15.04
27.2
SimVLM (Wang et al., 2021)
40.6
33.7
25.4
Unitying (Hanng
2021)
25.9
OSCAR
2020)
29.7
OFA (Wung et al, 3022)
L-Vere (Kim 2022)
45.8
10.5
31.1
Unifying (Huang al,, 2021)
37.3
28.2
21.9
L-verse (Kim et al., 2022)
39.9
31.4
23.3
Ous/PPur) Ous/(T21
16.19
17.38
6.11
6.02
25.11
28.63
24.44
21.03
OFA (Wang et., 2022)
30.9
Qurs(1T)
39.6
29.3
23.4
Table I: Evaluation of image synthesis on the CUB-200-2011 and MS-COCO dataset. The first section the text-conditioned re- sults while the model in second part includes multimodal solution.
Table 2: Evaluation of image caption the MSCOCO Karpathy split. The first section is the text-conditioned results while the model in second part includes multimodal solution.
Text Quality In the case of image captions, compared several other text description methods, the details are shown in Table 2, where 12T is the image-based text-generation task. Similarly, the first part of the table has pure image caption solutions and the bi-directional methods demon- strated in the second part. Some results for image caption tasks are provided in Appendix G.1
Similarity In the multimodal generation phase, all the other methods for comparison need the input of at least one specified modality; instead our method generates both visual and verbal features. The generated vision-language pairings are shown in Fig. 4 while more pair samples are provided in Appendix G.4. this experiment, we use CLIP scores to evaluate the similarity between our generated visual and verbal modalities, while generated images with given texts were used for other methods. The comparison results are given Table where a higher correlation between the text/image samples of our model can be found.
4.2 CROSS MODAL MANIPULATION
To demonstrate the modification potential of our vision-language federation. we present the out- comes of picture alteration with description modification in Fig. and more samples can be found in Appendix G.3. In this experiment, we obscured a portion of the image and modified the text description as well, the tokens in the masked region are merely designated as [MASK] tokens. We anticipate that the model will complete the image based on the amended text, in conjunction with description based on the visible image region. Depending on the unmasked images, our model may supplement the caption that derived from a portion of the image. Moreover, based on the amended description, the model may enhance the appearance of the masked component. These findings in- dicate that our model is capable of modifying text and pictures both ways, in addition to jointly generating text and images.
8

---

Published as a conference paper at ICLR 2023
4.3 ABLATION STUDY
In the ablation test, we utilized a rapid sampling scheme, with an interval between each step of 10. First, we evaluated the outcomes without mutual attention, where we replaced all mutual attention in the model structure with causal self-attention. In addition, we evaluated the performance without the unified transition matrix, we employed matrix similar to Equation 4, with the out-of-bound outputs substituted by the minimal ordinal number of the corresponding modal.
(t)
CUB
COCO
(4)
FID
Bval set
0.286
0.306
Ours
17.38
VQ-Diff (Gu et al., 2022)
0.257
w/ Fast Sampling
20.96
iVQ-Diff (Tang et al., 2022)
0.304
w/o Mutual Attention
23,19
Unilying (Huang al., 2021)
0.309
w/o Unified Transition Matrix
32.63
OFA (Wang et al., 2022)
0.344
Ours (T2I)
16.19
Ours(Pair) Ours(T21)
0.280
0.301
w/ Fast Sampling
18.33
0.293
0.306
w/o Mutual Atlention
21.65
Ours(12T)
0.302
0.312
w/o Unified Transition Matrix
28.93
Table 3:
The CLIP similarity between the generated captions and images.
Table 4: Ablations studies. The experiments are conducted on the CUB-200.
5 RELATED WORK
Denoising Diffusion Model Sohl-Dickstein et al. (2015) initially presented the diffusion model, which consists a base Gaussian diffusion and a basic binomial diffusion. Ho et al. (2020) recently reignited interest in the extraordinary generating capacity of diffusion models. This is extended in subsequent research to enhance quality with the classifier guidance (Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021) and speed of sampling (Song al., 2021a; Salimans & Ho, 2021). In addition to continuous diffusion models that have received the significant attention, there is also substantial research based discrete diffusion models (Hoogeboom et al., 2021; Austin et al., 2021) for text generation or image segmentation. With the help of VQ-VAE, Esser et al. (2021a); Hu al. (2022); Gu al. (2022); Tang al. (2022) discrete diffusion models have become capable of generating high quality images.
Modality Translation Text-to-image generation is one of the key aspects of modality translation tasks. On simpler datasets, traditional GAN-based models (Xu al., 2018; Zhu al., 2019) may create high-quality pictures based on text hints, but face difficulties on more complex datasets. As result of the emergence of transformers employing the attention mechanism (Vaswani et al., 2017), conventional autoregressive models (Van den Oord et al., 201 6) have gained more powerful generat- ing capacities as ART (Auto-regressive Transformers) (Ramesh al., 2021; Esser al., 2021 Yu et al., 2022). Other types of modality translation models involve the addition of prospective modali- ties as criteria for improved generation outcomes (Gafni et al., 2022). There are also some works that focus addressing bidirectional generation between different modalities (Kim et al., 2022; Huang et al., 2021; Wang et al., 2022); however, these solutions are task-specific, necessitating different input formats and specific models with distinct modalities.
CONCLUSION
This work presents UniD3, novel unified framework for multi-modality generation tasks. We fisrtly introduced a unified transition matrix for the diffusion process which permits the states of fused tokens to be corrupted in a stable way. We demonstrated that designing transition matrices based on the objective and data characteristics is beneficial for discrete diffusion models. To capture the connection from the various modalities, we suggested mutual attention with fuse embedding for noiseless state recovery. Our method is also generic and competent to perform modality translation tasks for the specified modality. The specific designs of transition matrix are intriguing for further exploring, well the performance of more modalities is also one of the future directions.

---

Published as a conference paper at ICLR 2023
7 ETHICS
In this work, we propose UniD3, a new method for multimodal multi-task generation based on the discrete denoising diffusion model. All datasets evaluated in our experiments are open source, publicly accessible, and used with permission.
Similar to other generation algorithms, our method has both beneficial and negative societal effects, depending on its application and usage.
Positively, UniD3 may explore the limits deep learning generation through a unique pattern of graphic pair generation, while letting the typical user's imagination wild and decreasing the entrance barrier for visual content production.
On the other hand, UniD3 may be used to make modified photographs of great quality that are difficult to discern from the originals. Ways to fool humans and propagate false- hoods that are malicious. UniD3 can be abused by nefarious users, which may have serious repercussions.
In our code distribution. We shall outline the permitted uses of our system and give the corresponding licences.
However, we observe that the present discriminators used distinguish generated images are in- effective at identifying images generated by diffusion models. There is still the necessity for more exploration to discern between authentic and fake images.
REPRODUCIBILITY
In order to achieve reproducibility, we have made the following efforts:
1. The inference codes are released at sectts://gitien
2. Details on dataset and model architectures are provided Appendix B, as well more experimental settings are provided in Appendix D.
3. Some critical proofs are included in Appendix C & E.
4. Additional experimental results are provided Appendix G.
REFERENCES
Peter Anderson, Xiaodong Hc, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 6077-6086, 2018
Jacob Austin, Daniel Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:1 7981-17993, 2021.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended difffusion for text-driven editing of natural images. In Proceedings of the IEEE/VVF Conference on Computer Vision and Pattern Recognition, pp. 1208-18218, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-877 2021.
Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer. Imagebart: Bidirectional context with multinomial diffusion for autoregressive image synthesis. Advances in Neural Infor- mation Processing Systems, 34, 2021a.
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- fion, pp. 12873-12888 2021b.
10

---

Published as a conference paper at ICLR 2023
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based eoxt-to-image generation with human priors. arXiv preprint arX1v:2203.13131, 2022.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quamtized diffusion model for text-to-image synthesis. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10696-10706, 2022.
Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. Image captioning: Transforming objects into words. Advances in Neural Information Processing Systems, 32, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-683 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454-12465 2021.
Minghui Hu, Yujie Wang, Tat-Jen Cham, Jianfei Yang, and Ponnuthurai N Suganthan. Global context with discrete diffusion in vector quantised modelling for image generation. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11502-11511, 2022.
Lun Huang, Wenmin Wang. Jie Chen, and Xiao- Yong Wei. Attention attention for image cap- tioning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4634- 4643, 2019.
Yupan Huang, Hongwei Xue, Bei Liu, and Yutong Lu. Unifying multimodal transformer for bi- directional image and text generation. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 1138-1147, 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 10196, 2017.
Tachoon Kim, Gwangmo Song, Sihaeng Lee, Sangyun Kim, Yewon Seo, Soonyoung Lee, Se- ung Hwan Kim, Honglak Lee, and Kyunghoon Bac. L-verse: Bidirectional generation between image and text. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16526-16536 16536, 2022.
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, et al. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://github. com/openimages, 2(3):18, 2017.
Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 11523-11532 2022.
Xiujun Li, Xi Yin, Chumyuan Li, Pengchuan Zhang. Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pp. 121-137. Springer, 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- on Learning Representations, 2018.
11

---

Published as a conference paper at ICLR 2023
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11461-11471, 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei. X-linear attention networks for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11971-10980, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp.
PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with unified text-to-text transformer. Mach. Learn. Res., 2 21(140):1-67, 2020.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. In VISIGRAPP (5: VISAPP), 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831 PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: taage-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi. Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 14877, 2022b.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2021.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 202
Zhicong Tang, Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen. Improved vector quantized diffusion models. arXiv preprint 26000, 2022.
Ming Tao, Hao Tang, Fei Wu, Xiao Yuan Jing, Bing- Kun Bao, and Changsheng Xu. Df-gan: A simple and effective baseline for text-to-ine text-to-image synthesis. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Patlern Recognition, pp. 16515-16525, 2022.
12

---

Published as a conference paper at ICLR 2023
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-201 dataset. California Institute of Technology. 2011.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through learning framework. In International Conference on Machine simple sequence-to-sequee PMLR, 2022.
Learning, pp.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- plc visual language model pretraining with weak supervision. In International Conference Learning Representations, 2021.
Wansen Wu, Tao Chang, and Xinmeng Li. Visual-and-language navigation: A survey and taxonomy. arXiv preprint arXiv:2108 / 1544, 2021.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang. Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp- 1316-1324, 2018.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1556 006.03365, 2015.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Dector-quantizer image modeling with improved vggan. In International Conference on Learning Representations, 2021.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206.10789 2022.
Chuanxia Zheng, Long Tung Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. Advances in Neural Information Processing Systems, 35, 2022.
Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Towards language-free training for text-to-image generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17907- 17917, 2022.
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative ad- versarial networks for text-to-image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1802-5810 2019.
13

---

Published as a conference paper at ICLR 2023
A LIMITATIONS AND DISCUSSION
Our work focuses multi-modalle generation, and the task of every modal generation is taken into account. We consequently cannot employ adequate text or visual representations that are difficult to recover, which putting some pressure on the generative model in order to comprehend and represent the original data. This leads to lack of performance our model compared to the state-of-the-art methods in terms of conditional generation quality.
With the permission to manipulate the conditional space for better representation, it can allevi- ate the burden on the generative model to represent the condition. Most of the current modality translation models use a highly abstract representation of a text as a conditional signal, for example, widely used in VQ-Diffuion Gu et al. (2022), Stable Dif-
CLIP Radford et al. (2021) embedding fusion / LDM Rakhimov et al. (2021). Some work has proved that a powerful text encoder (T5 Raf- fel et al. (2020)) and highly abstract text representation could provide better result Saharia al. (2022b).
In future challenge, we will consider obtaining a condensed modal representation embedding, which is also simple to recover. This solution would reduce the strain on the generative model, enabling it concentrate on producing superior generative results. We also found that the first-stage VQ model also partially limits the quality of our image generation. Finding a VQ model with lower information loss rate is also an important way to improve the performance of the model.
discrete
Besides, our model demonstrates sufficient superiority in terms of text-image similarity without optimized by the CLIP loss, as shown in Table contrast, both improved VQ-Diffusion Tang et al. (2022) and OFA Wang et al. (2022) make explicitly or implicitly use of CLIP loss to optimise the model parameters.
B DATASETS AND EXPERIMENTAL DETAILS
B. DATASETS
We demonstrate the feasibility the proposed method on two commonly used datasets: CUB- 200 (Wah al., 2011) and MSCOCO (Lin et al., 2014). The CUB-200 dataset consists of 8,855 training images and 2,933 test images representing 200 species of birds. Each these photos is accompanied by 10 textual descriptions. In MSCOCO, there are 591,753 images utilized for training and 25,014 for testing, with each image corresponding to textual descriptions.
B.2 MODEL DETAILS
We use VQ-GAN (Esser et al., 2021b) with gumhel softmax to compress the images into discrete token sequences, We directly use the pre-trained and public model which is trained on OpenIm- age (Krasin et al., 2017). The compression ratio of the model is 8 x 8 x 3 = 192, and each image compressed into 32 32 token sequence. The codebook size is 2,887 after removing useless codes (Gu et al., 2022).
Furthermore, as the transition matrix of the diffusion model is sensitive to the number of dictionary size and the length of the sequence influences the performance of the denoising neural network, we further investigated the effect of various forms of text word lists. We select the dVAFs with different downsampling factors and dictionary size from (Esser et al., 2021b). We used alternative models of VQ-GAN release, including different downsampling factor with different codebook sizes, from f8 with 8, 8,192 terms Z [ 8192x256 to f 116 with 16,384 entries Z E R 1663842256
For the text portion, we use a BPE tokenizer comparable to (Ramesh et al., 2021; Radford et al., 2021) and set the dictionary size to 8,192, compressing each text description into sequence of length 128. The text and image encoders are fixed in the training phase.
B.3 EXPERIMENTAL DETAILS
For the diffusion process, we set the number diffusion steps to 500. And the noise planning is linear, where an goes from
and 7t goes from 0 to 1. The denoising network architecture is
14

---

Published as a conference paper at ICLR 2023
(4)
FID
Sample Steps
Ours
17.38
500
w/ f8-8192 Gumbel Image Encoder
25.68
100
w/ f16-974 Image Encoder
21.02
100
w/ f16-16384 Image Encoder
24.61
100
Table 5: Ablations studies on Image Encoder. The experiments are conducted on the CUB-200.
described in Sec.3.2, in which the transformer comprises 20 transformer blocks with 16 heads attention and 1024 feature dimension. The model contains 600M parameters. For the ablation model, we use 18 transformer blocks with 16 heads, and the dimension is 256. The model contains 119M parameters. The optimiser for the model is AdamW (Loshchilov & Hutter, 2018), and the learning rate 9e- without warmup. We trained models with batch size of 16 across 8 Tesla A100.
is worth noting the comparison experiments that LAFITE (Zhou et al., 2022) language- model. And LDM (Rombach et al., 2022) is trained on LAION-400M and conducts zero-shot text- to-image generation based on MSCOCO evaluation captions. The results of LDM with a f8-KL, based image encoder and fast sampling strategy, excluding the classifier guidance during inference.
And as caption generation our Pair scenario is unique. Common text generation metrics are difficult to implement due to lack of the references. Here we used Perplexity (PPL) with GPT-2 to evaluate the text quality. The perplexities of generated CUB-200 and COCO captions are 123.32 and 188. 74 whereas the corresponding values for the evaluation set 108.66 and 175.36.
C LOSS FUNCTION
Similar to the continuous domain, the complete expression of the loss function in our model is:
Cub = E E((a)) [DLL (g(x)(20)) [p(ær))
(m0) DKL (9(0)-11<); 122,, æu)il poo ( xx)) po (2 (2x)
(16)
Bggaes (200) (oog (mg a. (at) + log log (= | 10 m0 (eme
D TRUNCATION SAMPLING
As stated (Gu et al., 2022), the truncation sampling is crucial for the discrete diffusion- ap- proach, which prevents the network from sampling tokens with low probability. In our experiments, the truncation rate to 0.88 for Pair and T2/ generation and 0.75 for 12T task, mathematically, we only keep top 88% 75% tokens of pe pa (x;) during inference for image and text tasks, respec- tively. A truncation rate that is too low will cause loss of image detail, but rate that is too high will prohibit the image from creating a distinct geometry.
E PROOF OF THE CLOSED-FORM SOLUTION FOR Q
The solution for Q, inherits the dosed-form attribute from (Gu et al., 2022). Mathematically. given initial state æn at = 0, the probabilities for next time-step / can be obtained:
σ1 B,
s.t. argmax 2% [0, K),
a1 + B
x = Eo
s.t. argmax Zn E K, K + K*),
Q
3
x # To
s.t. argmax To E K),
(17)
81
+ wo
s.t. argmax So
K,, K + K*),
7,
argmax = + K*
15

---

Published as a conference paper at ICLR 2023
Suppose the closed-fore expression of Q, holds at time-step t т, then for the next step t
1:
7++120 To Q++1Q
(18)
The outputs under different conditions can be discussed:
1. When æ = Zo and argmax 2o E [0, K):
Q ++1 3,8++1
+ (0,+1 + 11+1) (a, + 37)
B, (KBr+1 + a++1) + 04 (at+1 + 9t+1)
K [B,(1 -r++1) + Cr 3r 7++1 K] + 0r+1 + Br+1
K [(1 - -1-- (7,)(1 - -
+ KT 0r+1 (1 -1- - -r++1) + T++1 + +
[1 -T++1) - - (1 7 - K3r+1) - 7++1) + ar+1] + +,+1 + 8741
0r+1 + 3r41i
(19)
2. When æ = and argmna E [K, K + K*):
1++100 3" 30, (K* 1) + (ar+1 (²+1) (ar + 3r +)
= 3" B++1 + a++1) + at (at+1 + 11+1)
K 18-(11 --+1 5+ 8" 1+1) |*| + 0,+1 + 7+1
K* [[1- - T, - 71) - 1++1) + K a" 3r (1 - ar+1 - Tr+i]] + 0,+1 + 3 7+1
K* - -7++1) - -1, -1+ KK - - -r+11 + + + +a++1 + +3 r+1
Tr+l + + 1+11
(20)
3. When Do and argmax 2o E [0, K):
Q++130 (ar11 + Br+1) + 3, (K - 1) + a. 37+1
3, + B7+1) K + σ. B7+1
1- 0t 7x (1 (1+1)) c Br+1
K
1 Yr+1
(21)
- (1 11--%am -)++)) + C. (B(+1
K
K
-B441 +
ar+1
+ Tr [11 - 0T+1) - 1]
K
K
-7411i
When + xo and argmax 2g E [K, K + K*):
Q++120 = 3" (ar+1 + B141) + 3
8 3 (K* - 1) + +0,34
8e (ar+1 1141 * K" + 0r
a+ -7, (1 - -+++1) + a. 3
K*
K (1 7 Tr (3, 1 71+1
1
(22)
K*
0++1 ar [1 a7+1) 1]
F 7+1
Kt
K
= = - 7+11
16

---

Published as a conference paper at ICLR 2023
5. When argmax K k
Q Q+1100 = 7, (1+7))
=1- -1-(1-7++1)
(23)
= -++1
Thus, can obtain closed-form solution for Q,
(ā, + 0,) (7, -,) [M) + B,,
s.t. argmax E [0,K),
Q,
(ā+ + 3" )00 + (7, 5*, +)xx + 3"
s.t. argmax E K K + K*)." (24)
where 2 [X] = a + argmax x = K + K
Given indicator functions 1 and 1 for different modals:
1 if argmax zo [0, K),
1 (azo)
0 otherwise.
{1 if argmax E K, K + K*),
(25)
and (m(o))
(0 otherwise.
(.t.1 0 1 Ø;
Eq. 24 can be expressed in:
Q = 00 + 7/ - 10x0))) 1'(00))) wim) + 1xx0), + 1'xx))3)
(26)
F EXTENSION TO MORE MODALITIES
The above description emphasizes textual and visual modalities. However, our proposed method- ology is also applicable other discretisable modalities. Even within the visual domain, we may further consider distinct modalities such as bounding boxes, segmentation masks and edge maps, in addition to RGB images. Specifically, our transition matrix in Eq. is extensible, additional modalities by simply adding new modal quadrants and leaving the mask transition in the final row and column intact. Should the number of states in modality become excessive, will degrade to an absorbing diffusion. It is straightforward that the extended unified transition matrix also meets the criteria of the discrete diffusion model, with the stationary distribution assigning all probability mass to the [MASK] token.
have
The unified objective can also include more modalities. When there are more modalities, po (x consists of more distributions, e.g., pe p=(x)/1) P (=t Pa (x."1)) The architecture of the neural network used for prediction žo can also be altered by adding the mutual attention to cor- responding modal, which is conditioned on the remaining modal sequences. Due to the fact that the complexity of the transformer quadratic the length of the sequence, excessive modalities may incur enormous memory cost.
The above demonstrates the ability of the proposed model handle complex modalities; however we leave the exploration and optimisation of more modalities to future work.
G ADDITIONAL RESULTS
G.1 IMAGE CAPTION
We provide some image caption results based on CUB-200 and MSCOCO dataset in Figs & 7, respectively.
17

---

Published as a conference paper at ICLR 2023
Ref. Image
Ref. Caption:
Ref. Image
Ref. Caption:
in between two cars parking lot.
a cat sitting on the hood of a parked black garage
Sample:
Sample:
a cat stands in front of a black car green car.
a close up of cat laying on top of car.
Ref. Image
Ref. Caption:
Ref. Image
Ref. Caption:
dirt bike parked fent in the woods.
dish of some sort pizza stone
Sample:
Sample:
motorcycle parked in brown grass field.
a person ready for eating piece of pizza plate
Ref. Image
Ref. Caption:
Ref. Image
Ref. Caption:
a black trash bag a restroom next sink.
a man in a wetsuil surling a wave on surf board.
Sample:
Sample:
bath tub in white bathroom sink.
man is rieling small wave surfboard.
Ref. Imaage
Ref. Caption:
Rel. Iurge
Ref. Caption:
person tennis court in tennis wear.
motorcycle parked the gravel in front of gurage
Sample:
Sample:
person standing on temnis court about
a blue motorcycle parked front of a garage.
Figure 6: Samples of the image caption task based on MSCOCO dataset.
18

---

Published as a conference paper at ICLR 2023
Ref. Image:
Ref. Captions:
this bird has all black body with a large orange beak and white eyc.
hird has white eyering wehhed black bird with smubbed, rounded orange bill, stark white eyes, webbed feet, and feathered superciliary. compared to the rest of the body, the head is relatively small, the bird has webbed feet and orange beak.
this pecullirr bird large and mostl grey with purple webhed feet, bright orange, stulbby beak, and head.
black phume
this bird is black with red and has a very short beak.
mevluum sized hird with grey feathers, black head with white stripe leading down fiom the eye, white eye, molamle molaawkssapee leshaped leather on the head, and an orange beak.
the crown of the bird has a distinctive feanure of black feathers protruding from the head. dark gray bird with black webbed feet, red beak crown stands up this bird with webbed feet has an orange bill, white malar stripe, and grayish-brown breast.
Samples:
the bird hus black leatherrs its cleekk neck and webbed feet. this is a dark grey hird with some dark jrey wings and a large light colored beak this big grey bird has a black body and orange bill
this is black white with a long, : orange
this ation bird has white body shor, black beak, grey and black weand orange feet.
this bird has orange beak with black body grav belly and black wings
this hard has a grey body and black wings with a long orange bill.
this bird has a dark gray belly.. an off orange beak and a brightt orange its eve.
Ref. Image:
Ref. Captions:
the small bird light brown with short beak. this bird has a white throat, a short bill, and gray crown.
this small bird with tlat bill and multicolored body and black eyes. diis small bird grey this gray bird with white throat and mohawk at the crown. this particular bird has gray belly and breast with black vecondaries.
small black beak
thin bird has a littl gray mohawk and a short tail.
this bird is grey with white and has long, pointy beak. this particular bird has belly that gray with black spots. this bird has brown crown with brown coverts and black beak.
Samples:
this is small brown bird with dark brown wings grey head, and grey breast. small brown headl and grey hird with a white helly small bird with brown feathers and gry back and wings
this bird has wings that are grey and white with grey belly
this is small gra bird with an all white belly and medium colored wings with black head. small hird with small hill and light dark gray helly this small bird is grey and has white wings and tail and dark gray feathers.
has brow feet.
small grey bird with grey darker on wings
Figure 7: More samples of the image caption task based on CUB-200 dataset.
19

---

Published as a conference paper at ICLR 2023
G.2 TEXT CONDITIONAL GENERATION
We provide more results of pure text to image synthesis in Fig. 8. The resolution of each generated image is 256 x 256.
G.3 CROSS MODAL MODIFICATION
We present samples for modification across vision and language modals under CUB-200 dataset in Fig. The ref. image shows the reference image after the dVAE reconstruction.
G.4 VISION-LANGUAGE PAIR GENERATION
A few additional examples for vision-language pair generation are shown in Fig 10. The resolution of each image is 256 x 256.
20

---

Published as a conference paper at ICLR 2023
Text Promotion:: hird has wings that hrown and yellow hill
Image Samples:
Text Promotion: a completely black bird, small body and head, with black beak.
Image Samples:
Text Promotton: this particular bird has a belly that is tan and white
Image Samples:
Text Promotion: parade of motorcycles is going through group of tall trees
Image Samples:
Text Promotion: city street with multiple bildings and a street light.
Image Samples:
Text Promotion: an all white kitchen with an electric stovetop.
Image Samples:
Figure 8: More samples for the pure text-to-image generation from CUB-200 and MSCOCO.
21

---

Published as a conference paper at ICLR 2023
Masked
praings
this dovkk rnd audd ertwgg hind with Back ving lurs and pentartes und al sasy bill
this is alanic rnd und petiege leed black wiing and prinurics
this is abbis, rts axs. ertieeg hird with Nark ving lors and prataries vellow crontr
this is divtt red secd orange lund with kack wing prinuric
yeimuriss
Samples
dert dirne, noad dis has brcht green head. Statod
this hird has and with lacoeu becual umane foxt Secand loug yellow bill
this band has dies shirk. hess and linose with leoom secondaries and long yellow
this hind has haad, temon belly and Imven vriage and long yellow bill
thi lend has slees abrie hard, ai beorut beizet orange feet long yellow
yellow
Ref. Iingge
Miskee Image
Somples
dis had is velumeae blrck color, mal adl bemon
iidd Bark cake wäth aubbe beak.
this hird is backk in eoder, and los Kadk
this hind is black color. silt ensil shomp
this band blac color 98th slorp blbcks braki.
Bef. Image
Masked Imaz:
Samples
this while Bark Teakk
tas while Bark and Baack boal.
while bivd will black Black feed
whoke lind Mack fixs and Bhack
this -ierk---in hini bnth
Back Milkoor
Figure 9: More samples cross modal vision and language manipulation and infilling. The dark brown portions of the image and description represent the [MASK], while the strike-through repre- sent the caption manipulation.
22

---

Published as a conference paper at ICLR 2023
this bind bloe and white bird with lange black beak.
yellow bird with or ange bill and black head with green feath-
this is black bird with red eye rings and sharp beak.
this bird brown with while and has long pointy beak.
this little pointed bill and yellow colored breast.
this yellow wing and a yellow crown and black hill.
wings are black and has a yel- low and black belly.
bcown belly , black throat and black wings.
particular
a grey beok and yellow throat
white belly and ocange and brown feet and black tail.
arey black blue wings
yellow black wings.
wings
brows
are black and has bodyy
with white breast, sbort beak.
yellow in color. with black shorp beok, and black eye
stop
top
village momntiin.
buy
desert.
plastic
plate hus fast food with solasalad and becadl.
lagg old church and clear blue
plane just landed the
there is baseball
bos of already baked piizz being shaned.
player ready for piic..
pair of couple ing beeod in murket.
trwelingg
many people
mony basebull players are the field.
acress the city.
on blue shuille
kischen windsws
Figure 10: More vision- pair samples from CUB-200 and MSCOCO. Both the image and caption are generated simultaneously.
23


Reviews:
[{"review_id": "pgwDZTSFtDU", "reviewer": "Reviewer_1mAH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper models the joint distribution of texts and images using a discrete diffusion model. The basic idea is to firstly using a VQ model to convert an image to discrete tokens, and then model tokens of both texts and images. This paper designs a specific Markov transition matrix as well as a mutual attention mechanism for this joint modeling task. By joint modeling, this paper can do tasks such as joint sampling of texts and images, text to images, and image captioning.\n", "summary_of_the_review": "See Strength and Weaknesses.", "strength_and_weaknesses": "Strength:\n\nThe experiments are interesting for me. For example, it can mask parts of images and captions, and then generate the masked parts, which might be useful in practice.\n\nQuestions:\n\n1. Missing details of conditional generation. This paper models a joint distribution $p(x^{img}, x^{txt})$, where sampling from the joint distribution is obvious. However, it is not obvious for me to sample from the condition distribution, such as $p(x^{img} | x^{txt})$. How does this work implement it?\n\n2. Missing details of image impainting. The mask is drawn in the pixel space, instead of the token space. So the implementation of impainting is not so obvious. I suggest to add these details in Appendix. Besides, does this method support mask an arbitrary area in the image? How to ensure the consistency in the unmasked part between the original image and the generated image?\n\n3. In Eq.(10), why the conditional distribution is $p_\\theta(x_0^{img} |x_1, x_0^{txt} )$ instead of $p_\\theta(x_0^{img}|x_1)$? I think $p_\\theta(x_0|x_1) = p_\\theta(x_0^{img}|x_1) p_\\theta(x_0^{txt}|x_1)$, and therefore $L_0$ should be $-E_q [\\log p_\\theta(x_0^{img}|x_1) + \\log p_\\theta(x_0^{txt}|x_1)]$.\n\n4. In Eq.(12) and Eq.(13), the left part is unrelated to $x_0^{txt}$, but the right part is related to $x_0^{txt}$. Is this a typo?\n\n5. This work uses a large number of parameters (600M) for relatively small dataset (MSCOCO), but there is still a performance gap between it and the compared method, e.g., FID=25.11 v.s. FID=10.5 on MSCOCO. Perhaps there is still an improvement space on the architecture.\n\nSummary:\n\nOverall, I think this paper is interesting, but it can still be improved by adding more details and fixing some confusions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "diVt9gzVNP", "reviewer": "Reviewer_bEZ6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a unified discrete diffusion model for simultaneous vision-language generation. The proposed model extends multinomial diffusion to the joint text-image tokens with a transition matrix to prevent transiting among modalities. A mutual attention module is proposed to better capture the inter-modal linkages.", "summary_of_the_review": "My rating is mainly based on the novelty and clarity of the paper. I might amend my score if my major concerns are addressed.", "strength_and_weaknesses": "## strengths\n- The proposed method is simple and shows promising results.\n- The proposed unified framework supports various tasks including cross-modal, text-to-image, and image-to-text generation.\n## weakness\n- The paper is not clearly written. For example, I cannot fully understand the proposed mutual attention from Fig. 3. It would be good if the authors could provide e.g. mathematical definitions.\n- One of the main contributions: the unified transition matrix seems a trivial and intuitive extension of VQ-Diffusion or multinomial diffusion to the concatenated text-image tokens. The novelty is slightly limited.\n- Autoregressive models like DALLE also model the joint distribution of text and image. How does the proposed model compare to autoregressive models?\n- From the results shown in Tables 1 & 2, the proposed method does not perform the best in any of the entries. Could the author provide more discussion on the results? For example in Table 1, the T2I FID is much higher than VQ-Diffusion, which is a very close baseline. Any intuition why joint generative vision-language training is not helpful here?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Cs7k4PdRMlG", "reviewer": "Reviewer_cr8c", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multimodal diffusion model, that can perform text-conditioned, image-conditioned and text-and-image-conditioned generation. They utilize discrete diffusion, based on VQ-diffusion [Gu et al, 2022], with the images encoded using discrete VAE and the text encoded using byte-pair encoding. The  discrete diffusion process is designed in the form a unified text-image Markov transition matrix to estimate the joint distribution of language and image. To implement this in practice, they propose a mutual-attention transformer with a fused embedding, which enables the unified diffusion process. They perform experiments for image generation and text-image generation using the CUB and MSCOCO datasets and compare their method with baselines using FID and IS scores. The proposed method is worse than the baseline VQ-diffusion on both datasets, but obtains competitive FID scores. Image caption generation is also evaluated on MSCOCO, where the proposed approach obtains competitive (but not state-of-the-art) results. However, uniquely, the method is able to simultaneously generate both text-and-image and this capability is evaluated by comparing the CLIP similarity between generated image-caption pairs with CLIP scores of eval set pairs, where the proposed method shows slightly better results. ", "summary_of_the_review": "The paper proposes an extension of VQ-diffusion for multimodal discrete diffusion generative model. The results are reasonably good and the novelty of the approach is not somewhat limited. ", "strength_and_weaknesses": "Strengths:\n1. The paper is clearly written and the method is described in detail. The writing, along with the code, makes the paper reproducible.\n2. The method is evaluated thoroughly for image generation, text generation and joint image-text generation using both qualitative and quantitative experiments and compared with baseline methods. \n\n\nComments:\n1. The proposed method is basically an extension of VQ-diffusion for the multimodal case, with a new mutual-attention transformer with a fused embedding proposed to enable multimodal generation. While this extension is not trivial, it is fairly intuitive.  Can the authors highlight the main novelty and technical contributions of the paper? \n2. The generation results for individual modalities are competitive, but worse than VQ-diffusion, indicating that the joint distribution estimation leads to degradation in performance for T2I generation.  Can the authors comment on this?\n3. The multimodal generation results are significantly worse than the autoregressive approach (OFA). While its a very different approach, it would be good for authors to comment on the comparison. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "50QJEuYil1", "reviewer": "Reviewer_tuqJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method for image-text generation that can perform conditional and unconditional image and text synthesis from either blank inputs or by partially masking any of them. The method includes a two pathway for image and text encoding-decoding with a unified diffusion path and matrix, that works with the fusion of the visual and text tokens. The paper is accompanied with code. The experiments include text-to-image synthesis, image captioning, and image and text replacement, which can be accomplished by the very same model. The results are competitive w.r.t. state of the art works, setting a promising direction towards multimodal generation.", "summary_of_the_review": "The paper has the merits to be published at ICLR, and it has the potential to draw the audience's eye. However, some extra efforts are needed towards improving the quality of the manuscript.", "strength_and_weaknesses": "The paper is conceptually easy to follow and understand, and has the merit to propose a unified model for both image to text, and text to image synthesis as well as completion. The paper is reproducible and the computational requirements to achieve good results are not excessive.\n\nThe quantitative results do not seem to surpass previous works on either of the modalities, which might at first set some concerns with the proposed work. However, it is worth noting that the metrics do not fully reflect which model is better, and the visual results are really on par with existing methods (For example the FID score is a metric that below some level is indeed indicating a dataset replication). It is also worth remarking the merit of the method to achieve competitive results in both tasks with the same model. In addition, the multi-modal completion feature is a desirable property of the method which to my knowledge has not been proposed yet in the literature. \n\nWhile the paper is technically sound and easy to follow, it reads quite weak; the writing and presentation need a good effort before the camera ready to improve the paper’s readability. There are several typos and while the mathematical derivation is interesting, some other details might be more attractive to the audience that are left to the supplementary material, such as the experimental details. I would thus suggest the authors to perform a thorough proofreading. \n\nSome well renowned methods are left out of the comparisons, such as DALL-E [Ramesh et al. “Zero-Shot Text-to-Image Generation”], for the task of image generation. While DALL-E is a very big model, showing how the model’s capacity can affect performance would also boost the paper’s contributions. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster