Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation
Authors: Ye Zhu
Yu Wu
Kyle Olszewski
Jian Ren
Sergey Tulyakov
Yan Yan
Venue: ICLR 2023 poster (2023)
Keywords: Contrastive Diffusion
Conditioned Generations
Music Generation
Image Synthesis

Abstract: Diffusion probabilistic models (DPMs) have become a popular approach to conditional generation, due to their promising results and support for cross-modal synthesis. A key desideratum in conditional synthesis is to achieve high correspondence between the conditioning input and generated output. Most existing methods learn such relationships implicitly, by incorporating the prior into the variational lower bound. In this work, we take a different route---we explicitly enhance input-output connections by maximizing their mutual information. To this end, we introduce a Conditional Discrete Contrastive Diffusion (CDCD) loss and design two contrastive diffusion mechanisms to effectively incorporate it into the denoising process, combining the diffusion training and contrastive learning for the first time by connecting it with the conventional variational objectives. We demonstrate the efficacy of our approach in evaluations with diverse multimodal conditional synthesis tasks: dance-to-music generation, text-to-image synthesis, as well as class-conditioned image synthesis. On each, we enhance the input-output correspondence and achieve higher or competitive general synthesis quality. Furthermore, the proposed approach improves the convergence of diffusion models, reducing the number of required diffusion steps by more than 35% on two benchmarks, significantly increasing the inference speed.

Published as a conference paper at ICLR 2023
DISCRETE CONTRASTIVE DIFFUSION FOR CROSS- MODAL MUSIC AND IMAGE GENERATION
Ye Zhu
Yu Wu
Department of Computer Science Illinois Institute of Technology Chicago, IL 60616, USA yzhu960haw's iitieiii.. edu
School of Computer Science
Wuhan University
Wuhan 430000, China
wuyucsewha edu. cn
Kyle Olszewski, Jian Ren, Sergey Tulyakov Snap Inc.
Yan Yan
Department of Computer Science Illinois Institute of Technology Chicago, IL 60616, USA yyan340iit. eeu eduu
Santa Monica, CA 90405, USA (kolszewski, jren, stulyakovy8snap.oom
ABSTRACT
Diffusion probabilistic models (DPMs) have become a popular approach ditional generation, due to their promising results and support for cross-modal synthesis. A key desideratum in conditional synthesis is achieve high corre- spondence between the conditioning input and generated output. Most existing methods learn such relationships implicitly, by incorporating the prior into the variational lower bound. In this work, we take different route-we explicitly enhance input-output connections by maximizing their mutual information. To this end, we introduce a Conditional Discrete Contrastive Diffusion (CDCD) loss and design two contrastive diffusion mechanisms to effectively incorporate it into the denoising process, combining the diffusion training and contrastive learning for the first time by connecting with the conventional variational objectives. We demonstrate the efficacy of our approach in evaluations with diverse multimodal conditional synthesis tasks: dance-to-musie generation, text-to-image synthesis, image synthesis. On each, enhance the input-
well as output correspondence and achieve higher or competitive general synthesis quality. Furthermore, the proposed approach improves the convergence of diffusion mod- els, reducing the number of required diffusion steps by more than 35% on two benchmarks, significantly increasing the inference speed.
INTRODUCTION
Generative tasks that seek to synthesize data in different modalities, such audio and images, have attracted much attention. The recently explored diffusion probabilistic models
et al. (015b) have served
powerful generative backbone that achieves promising results in both
unconditional and conditional generation Kong et al. (2020); Mittal et al. (202T); Lee & Han (202) l
Ho et al. (2020); Nichol & Dhariwal (2021):
1): Dhariwal Nichol (2021) ); Ho al. (2022): Hu al.
(2021). Compared to the unconditional case, conditional generation is usually applied in more
concrete and practical cross- cross-modality scenarios, e.g., video-based music generation Di al. (202))
Zhu et al. (2022a); Gan et al. (2020a) and text-based image generation Gu et al. (2022); Ramesh et al. (2019); Ruan al. (2021) }. Most existing DPM-based conditional synthesis works Gu et al, (2022) Dhariwal & Nichol (2021) learn the connection between the conditioning and
(2021))
the generated data implicitly by adding a prior to the variational lower bound Sohl-Dicksteii et al.
(2015b). While such approaches still feature high generation fidelity, the correspondence between the conditioning and the synthesized data can sometimes get lost, as illustrated in the right column in Fig. To this end, we aim explicitly enhance the input-output faithfulness via their maximized mutual information under the diffusion generative framework for conditional settings this paper. Examples of our synthesized music audio and image results are given in Fig.

---

Published as a conference paper at ICLR 2023
Outputs w/ Weaker Corresonnencee
Input
Outputs from Our Contrastive Diffusion
Dance-to to Music
this bird is gray with white and has long, pointy beak.
Text-to- Imoge
a lot of motorbikes line up down busy street.
Class to Imagee
Class label: Mushroom
Figure 1: Examples of the input (left column) and synthesized output (middle column) from contrastive diffusion model for dance-to-music (Rows 1-2), text-to-image (Rows 3-4), and class- conditioned (Row generation experiments five datasets. The right column shows some synthe- sized data with reasonable quality but weaker correspondence to the input from existing methods Zhu
al. (2022a):
Gu et (2022).
Contrastive methods Oord et al. (2018); Bachman al. (2019); Song Ermon (2020a) have been proven to be very powerful for data representation learning. Their high-level idea aims learn the representation of raw data x based on the assumption that properly encoded benefits the ability of generative model to reconstruct the raw data given prior. This idea can be achieved via optimization of the density ratio p(x(z) Oord et al. (2018) as an entirety, without explicitly
p(z)
modeling the actual generative model p. While the direct optimization of mutual information via generative models p is challenging problem to implement and train Song & Ermon (2020b): [Belghazi] et al. (2018) in the conventional contrastive representation learning field, we show that this can be reformulate effectively done within our proposed contrastive diffusion framework. Specifically, the optimization problem for the desired conditional generative tasks via DPMs by analogy to the above embedding and raw data with our conditioning input and synthesized output. We introduce a Conditional Discrete Contrastive Diffusion (CDCD) loss, and design two contrastive diffusion mechanisms- step-wise parallel diffuusin that invokes multiple parallel diffusion processes during contrastive learning, and sample-wiss auxiliary diffusion, which maintains one principal diffusion process, to effectively incorporate the CDCD loss the denoising process. We demonstrate that with the proposed contrastive diffusion method, we can not only effectively train so as maximize the desired mutual information by connecting the CDCD loss with the conventional variational objective function, but also to directly optimize the generative network p. The optimized CDCD loss further encourages faster convergence of a DPM model with fewer diffusion steps. We additionally present intra- and negarine sampling methods by providing internally disordered and instance-level negative samples, respectively.
To better illustrate the input-output connections, conduct main experiments the novel cross- modal dance-to sonnce-t--muscc generation task Zhuu et al. (2022a), which aims to generate music audio based on silent dance videos. Compared to other tasks such text-to-image synthesis, dance-to-music
2

---

Published as a conference paper at ICLR 2023
generation explicitly evaluates the input-output correspondence in terms of various cross-modle modal aligmment features such as dance-music beats, genre and general quality. However, various generative settings, frameworks, and upplications can also benefit from contrastive diffusion approach, e.g..joint or separate training of conditioning encoders, continuous or discrete conditioning inputs, and diverse input-output modalities detailed in Sec. + Overall, we achieve results superior or comparable to state-of-the-art on three conditional synthesis tasks: Bance-to-musie (datasets: AIST++ Tsuchida et al. (2019); Li al. (2021). TikTok Mance-Music [huu et al. (2022a)) text-to- image (datasets: CUB200 Wah et al. (2011) 1 MSCOCO Lin et al. (2014) and conditiodd image synthesis (dataset: imageNer/Rusakovskk et al. (015)) Our experimental findings suggest three key take-away: I Improving the input-output connections via maximized mutual information is indeed beneficial for their correspondence and the general fidelity of the results (see Fig. I and supplement). ② Both our proposed step-wise parallel diffusion with intra-negative samples and sample-wise auxiliary diffusion with inter-negative samples show state-of-the-an scores in our evaluations. The former is more beneficial for capturing the intra-sample correlations, e.g., musical rhythms, while the latter improves the instance-level performance, e.g., music genre and image class. 3 With maximized mutual information, our conditional contrastive diffusion converge in substantially fewer diffussion steps compared to vanilla DPMs, while maintaining the same or even superior performance (approximately 35 % fewer steps for dance-to-mon mo-siic generation and 40 % fewer for texl-to-image synthesis), thus significantly increasing inference speed.
BACKGROUND
Diffusion Probabilistic Models. Sohl- Dickstein al. (2015b) are a class of generative models that learn to convert a simple Gaussian distribution into a data distribution. This process consists of forward diffusion process and reverse denoising process, each consisting of sequence of T steps that act as a Markov chain. During forward diffusion, an input data sample zo is gradually "corrupted" at each step & by adding Gaussian noise the output of step t 1. The reverse denoising process, seeks to convert the noisy latent variable 2T into the original data sample zo by removing the noise added during diffusion. The stationary distribution for the final latent variable Fr is typically assumed to be normal distribution, p(zr) = ~N(xx II
An extension of this approach replaces the continuous state with a discrete one Sohl- Dickstein et al. (2015a); Hoogeboom et al. (2021); Austin et al. (2021), in which the latent variables T1:7 typically take the form one-hot vectors with K categories. The diffusion process can then be parameterized using multinomial categorical transition matrix defined as q( (x+-)) = Cat(2:: p Dz-1 Qt). where [Qr]s = q(x; = -| = i). The reverse process po [x Ty- can also be factorized as conditionally independent over the discrete sequences Austin et al. 202
In both the continuous and discrete state formulations of DPMs Song & Ermon (2000c); Song et al. (2020b): Kingma et al. p): [Song et al. 20 [Huan et al. (2021); Vahdat et al. (2021) the denoising process py can be optimized by the KL divergence between and P4 in closed forms Song ct al. (2020a); Nichol & Dhariwal (202)) [Ho et al. ( (Hoogeboom et al. (2021) Austin et al.
(2021) via the variational bound on the negative logelikel:oood:
Lvb = Ey (eaa +
Σ D 2p0p (x)) log (2ol)=))) (1)
Cr
4-
Co
Existing conditional generation works via DPMs Gu et al. (2022); Dhariwal & Nichol 1021 usually
learn the implicit relationship between the conditioning and the synthesized data 2g by directly
adding the c as the prior in (T). DPMs with discrete state space provide more controls on the data corruption and denoising compared to its continuous by the flexible designs of transition matrix, which benefits for practical downstream operations such as editing and interactive synthesis Tseng et al. (2000); Cui et al. (202T): Xu et al. 2021). We hence employ contrastive diffusion using discrete state space in this work.
CounterpartAAsttier et al. (202T) Gu al. (2022)
Contrastive Representation Learning. Contrastive learning uses loss functions designed to make neural networks learn to understand and represent the specific similarities and differences between elements in the training data without labels explicitly defining such features, with positive and negative pairs of data points, respectively. This approach has been successfully applied in learning representations of high-dimensional data Oord et al. Bachhaaa
Bachman et al. (2019); He et al. (2020);
Song& & Ermon (2020a); Chen et al. 2020) Lin al. (2021) Many such works seek to maximize the
3

---

Published as a conference paper at ICLR 2023
mutual information between the original data and its learned representation under the framework of likelihood free inference Oord et al. (2018); Song & Ermon (2020a): Wu et al. (202I). The above
problem can be formulated as maximizing a density ratio p(a)=) that preserves the mutual information between the raw data and learned representation 2.
p(n)
To achieve this, existing contrastive methods Oord et al. (2018); Durkan et al. (2020); He et al. (2020) j Zhang et al. (2021) typically adopt neural network to directly model the ratio avoid explicitly considering the actual generative model p(arlz), which has proven to be a more challenging problem [Song& & Ermon (2020b); Belghazi al. (2018). In contrast,
entirety and
show that
by formulating the conventional contrastive representation learning problem under the generative setting, the properties of DPMs enable to directly optimize the model p in this work, which can be interpreted as the optimal version of the density ratio Oord et al. (2018).
Vector-Quantivg Representations for Conditional Generation. Vector quantization is classical technique in which a high-dimensional space is represented using a discrete number of vectors. More recently, Vector- Quantized (VQ) deep learning models employ this technique to allow for compact and discrete representations of music and image data Oord et al. (2017); Razavi et al. (2019); Esser al. (2021b); Dhariwal et al. (2020); Chen et al. (2022). Typically, the VQ-based models use an framework, where the "codebook" contains fixed number of vectors (entries) to represent the original high dimensional raw data. The encoder transforms the input ær into feature embedding that each mapped to the closest corresponding vector in the codebook, while the decoder uses the set of quantized vectors 2 to reconstruct the input data, producing x' as illustrated in the upper part of Fig.
ponder-codebook-deccooo
In this work, we perform conditional diffusion process on the VQ space (i.e., discrete token sequences) as shown in the bottom part of Fig. 2 which largely reduces the dimensionality of the raw data, thus avoiding the expensive raw data decoding and synthesis. As our approach is flexible enough to be employed with various input and output modalities, the exact underlying VQ model we use depends on the target data domain. For music synthesis, we employ a fine-tuned Jukebox DDariwal et al. (2020) model, while for image generation, we employ VQ-GAN Esser et al. (2021b). See Sec. for further details, We refer to 2, the latent quantized representation of z, as zg below to distinguish it from the latent representation at prior stages in the denoising process.
3 METHOD
Here we outline our approach to cross-modal and conditional generation using our proposed discrete contrastive diffusion approach, which is depicted in Fig. 2 In Sec. 3.1 we formulate our Conditional Discrete Contrastive Diffusion loss in detail, and demonstrate how helps to maximize the mutual information between the conditioning and generated discrete data representations. Sec. 3.2 defines two specific mechanisms for applying this loss within a diffusion model training framework. sample- wise and step-wise. In Sec. 3.3 we detail techniques for constructing negative samples designed to improve the overall quality and coherence of the generated sequences.
Given the data pair (c, 2). where e is the conditioning information from given input modality (e.g.. videos, text, or a class label), our objective is to generate a data sample x in the target modality (e.g., music audio or images) corresponding to In the training stage, we first employ and train a VQ-based model to obtain discrete representation 2n of the data x from the target modality. Next, our diffusion process operates the encoded latent representation 20 of x. The denoising process recovers the latent representation 20 given the conditioning c that can be decoded to obtain the reconstruction x'. In inference, we generate 2n based on the conditioning c, and decode the latent VQ representation 20 back to raw data domain using the decoder from the pre-trained and fixed VQ decoder.
3.1 CONDITIONAL DISCRETE CONTRASTIVE DIFFUSION LOSS
We seek to enhance the connection between e and the generated data by maximizing their mutual information, defined as [22:
Σ Po Po(zo, c) log polzale). We introduce set of negative VQ sequences Z' {2',2² 22 ..... z encoded from N negative samples X' = f x .... x }. and pu(sole) Our proposed Conditional Discrete Contrastive Diffusion (CDCD) loss is: define f(zo, c) po(2o)
po(2o)
f(2.,c) c)
LODED = -E log f(z0. c) Eesez
4

---

Published as a conference paper at ICLR 2023
VQ-Raaed Midel
Encoder
Decoder
GT data
data embedding
VQ sequamcc
moonstructed data
Codebook
Conditione Discrate Contrustive Diffieiion
Step Vise Parallel Diffision
Sumple Wise Auxiliary Diffusion
Conditoning
Conditionng Encoders
Engoddess
Conditioning
Conditiccing c
Polao (2,c)
putal (lei.o)
Polid (Fa.))
+
A Aucilisty Transformer--ee besed DPM
DPM stopp
Figure 2: Overview of the proposed pipeline. Our framework includes two major components: a VQ-based encoder-decoder model (top) and conditioned discrete contrastive diffusion generative model on the VQ space (bottom). In the contrastive diffusion stage, we illustrate our proposed step-wise parallel diffusion (bottom left) and sample-wise auxiliary diffusion (bottom right). The variables in green denote those from the principal diffusion process, while the variables in red represent the diffusion invoked by negative samples. Here show audio generation from video input, but demonstrate that this approach extends to different modalities, e.g., text-to-image.
The proposed CDCD loss is similar to the categorical cross-entropy loss for classifying the positive sample as in Oord et al. (2018), where our conditioning e and the generated data 2g corresponds to the original learned representation and raw data, and optimization of this loss leads to maximization of [=20; c). However, the loss in Oord et al. 2018) models the density ratio (20,. c) as an entirety. In case, demonstrate that the DPMs properties (Sohl-Dickstein et al. (2015b); [o et al. (2020); within the diffusion
Austin et a (202T) enable us to directly optimize the actual distribution process for the desired conditional generation tasks. Specifically, show the connections between the proposed CDCD loss and the conventional variational loss Lvb (see (T) in Sec. 3.2) and thus how it contributes to efficient DPM learning. Additionally, we can derive the lower bound for the
mutual information as [ c) 2 log(NV LCDCD (see supplement for details), which indicates that a larger number of negative samples increases the lower bound. These two factors allow for faster convergence of DPM with fewer diffusion steps.
3.2 PARALLEL AND AUXILIARY DIFFUSION PROCESS
The CDCD loss in considers the mutual information between and 2g in a general way, without specifying the intermediate diffusion steps. We propose and analyze two contrastive diffusion mechanisms to efficiently incorporate this loss into DPM learning, and demonstrate that we can directly optimize the generative model pe the diffusion process. We present our slep-wise parallel diffusion and the sumple-wise auxiliary disfusion mechanisms, which are distinguished by the specific operations applied for the intermediate negative latent variables zi.7 for each negative sample x The high-level intuition for the parallel and auxiliary designs is to emphasize different attributes the synthesized data given specific applications. Particularly, we propose the parallel variant to learn the internal coherence of the audio sequential data by emphasizing the gradual change at each time step, while the auxiliary mechanism focuses
the sample-level connections to the conditioning.
Step-Wiss Wise Parallel Diffusion. This mechanism not only focuses on the mutual information between e and zg. but also takes the intermediate negative latent variables zi.T into account by explicitly invoking the complete diffusion process for each negative sample 2 E Z'. As illustrated in Fig. 2 (bottom left), initiate N parallel diffusion processes, among which N are invoked negative
5

---

Published as a conference paper at ICLR 2023
samples. For each negative sample x) X', we explicitly compute its negative latent discrete variables "' In this case, (2) is as follows (see supplement for the detailed derivation):
po(2o.T) NE po(zorle))
[oo PPo(Fa:T [1] Cv)( C Eool( po ( (=):T))
LCDCD-Steg Step = Ez log
DEE
(3)
The equation above factorizes the proposed CDCD loss using the step-wise parallel diffusion mecha- nism into two terms, where the first term corresponds the original variational bound Cvb. and the second term can be interpreted as the negative sum of variational bounds induced by the negative samples and the provided conditioning c. C' is constant as detailed in our supplement.
Sample-Wii Wise Auxiliary Diffusion. Alternatively. our sample-wise auxiliary diffusion mechanism maintains one principal diffusion process. as traditional diffusion training, shown in Fig.
2 (bottom
right). It contrasts the intermediate positive latent variables 21:7 with the negative sample 2j E Z. In this case, we can write the CDCD loss from. 2 as (see supplement for details):
Sancl-Sample == -Ey1-log po c)] E
pe pe 2e, c)].
(4)
As with the step-wise loss, the EDCD-Sample loss includes two terms. The first refers to sampling directly from the positive Eg at arbitrary timestep t. The second sums the same auxiliary loss from negative samples zyj This marginalization operation based on the property of Markov chain in previous discrete DPMs Austin et al. (2021); Gu et al. (2022), which imposes direct supervision from the sample data. The first term similar the auxiliary denoising objective in Austin et al. 2022 Gu et al. (2022).
Both contrastive diffusion mechanisms enable us to effectively incorporate the CDCD loss into our DPM learning process by directly optimizing the actual denoising generative network po-
Final Loss Function. The final loss function for our contrastive diffusion training process is:
L Lvb(z, c) + XCCDCD,
(5)
Lvb conditioning e related, and takes the form of Lt- 1 = DKI. DKI. (g]:( (1 c)) as in Gu et al. (2022), where included as the prior for all the intermediate steps. LEDED refers to either the step-wist parallel diffusion or sample-wise auxiliary diffusion loss. Empirically, we can omit the first term in (3), or directly optimize LCDCD-Step- in which the standard Leb is already included. The detailed training algorithm is explained in the supplement.
3.3 INTRA- AND
SAMPLING
Previous contrastive works construct negative samples using techniques such as image augmenta-
tion Chen et al., (2020); He et al. (2020) or spatially adjacent image patches Oord et al. (2018). In this work, we categorize our sampling methods into intra- and inter-negative samplings as in Fig. 3 For the intra-sample negative sampling, we construct X' based on the given original x. This bears resemblance to the patch- based technique in the image domain Oord et al. (2018). As for the audio data, we first divide the original audio waveform into multiple chunks, and randomly shuffle their ordering. For the inter-sample negative sampling. X' consists of instance-leve negative samples x' that differ from the given data pair x)). In practice, we define negative samples x' to be music sequences with different musical genres from æ in the music generation task, while x' denotes images other than x in the image synthesis task (in practice, we choose x' with different class labels as x).
Based on our proposed contrastive diffusion modes and negative sampling methods, there are four possible contrastive settings: step-wise parallel diffusion with either intra- or inter-negative sampling (denoted as Step-Intra and Step-Inter), or sample-wise auxiliary diffusion with either intra- or inter- negative sampling (denoted as Sample-Intra and Sample-Inted Inter). Intuitively, we argue that Step-Intra because
and Sample-Inter settings are more reasonable compared to Step-Inter and Sample- of the consistency between the diffusion data corruption process and the way to construct negative samples. Specifically, the data corruption process in the discrete DPMs includes sampling and replacing certain tokens with some random or mask tokens at each diffusion step Austin et al. (2021): Gu et al. 20022. which is a chunk-level operation within a given data sequence similar to the ways construct intra-m nergavv samples by shuffling the chunk- orders. In contrast, the sample-wise auxiliary diffusion seeks to provide sample-level supervision, which is consistent with nnter-ne sampling method.

---

Published as a conference paper at ICLR 2023
negative samples
negalive samples
Figure 3: Illustration of intra- and inter-negative sampling for music and image data.
In the interest of clarity and concision, we only present the experimental results for Step-Intra and Sample-Inte Inter settings in Sec. of our main paper. The complete results obtained with other contrastive settings and more detailed analysis are included in the supplement.
EXPERIMENTS
We conduct experiments on three conditional generation tasks: dance-to-musse generation, text-to- image synthesis, and classcconditioned image synthesis. For the dance-to-music task, we seek to generate audio waveforms for complex music from human motion and dance video frames. For the text-to-image task, the objective is to generate images from given textual descriptions. Given our emphasis the input-output faithfulness for cross -rossl generations, the main analysis are based on the dance-to- toaccetto-muss to--usic generation task since the evaluation protocol from Zhu et al. (2022a) explicitly measures such connections in terms of beats, genre and general correspondence for generated music.
4.1 DANCE-TO-MUSIC GENERATION
Dataset. We use the AIST+ Li et al. 2021) dataset and the TikTok Dance-Music dataset Zhu al. 2022a) for the dance-to-musie experiments. AIST+ is subset of the AIST dataset Tsuchida
et al. (2019), which contains 1020 dance videos and 60 songs performed by professional dancers and filmed in clean studio environment settings without occlusions. AIST+ provide human motion data in the form of SMPL Loper et al. (2015) parameters and body keypoints, and includes the annotations for different genres and choreography styles. The TikTok Dance-Music dataset includes 445 dance videos collected from the social media platform. The 2D skeleton data extracted with OpenPose Cao et (2017); Cao et al. (2019) is used as the motion representation. We adopt the official mooss- splits without overlapping music songs for both datasets.
Implementations. The sampling rate for audio signals is 22.5 kHz in our experiments. We use 2-second music samples as in Zhu et al. (2022a) for the main experiments. We fine-tuned the pre-trained Jukebox Dhariwal et al. (2020) for our Music VQ-VAE model. For the motion encoder, deploy a backbone stacked with convolutional layers and residual blocks. For the visual encoder, we extract 13D features Carreira & Zisserman (2017) using a model pre- trained on Kinectics Kaatiicc/Kyyaaa et al.) (2017) the visual conditioning. The motion and visual encoder outputs concatenated to form the final continuous conditioning input to our contrastive diffusion model. For the contrastive diffusion model, we adopt a transformer-based backbone learn the denoising network ps. It includes 19 transformer blocks, with each block consisting of full attention, cross attention and feed forward modules, and a channel size of 1024 for each block. We the initial weight for the contrastive loss as X = 5e 5. The number N of intra- and inter-negativ samples for each GT music sample is 10. The visual encoder, motion encoder, and the contrastive diffusion model are jointly optimized. More implementation details are provided in the supplement.
Evaluations. The evaluation of synthesized music measures both the conditioning- output correspon- dence and the general synthesis quality using the metrics introduced in Zhu et al. (2022a). Specifically. the metrics include the beats coverage score, the beats hit scores, the genre accuracy score, and two subjective evaluation tests with Mean Opinion Scores (MOS) for the musical coherence and general quality. Among these metrics, the beats scores emphasize the intra-sample properties, since they calculate the second-level audio onset strength within musical chunks Ellis (2007), while the genre accuracy focuses on the instance-level musical attributes music styles. Detailed explanations of the above metrics can be found in Zhu et al. (202aa). We compare against multiple dance-to-musi.
7

---

Published as a conference paper at ICLR 2023
Table 1: Quantitative evaluation results for the dance-to-music task on the AIST+ dataset. This table shows the best perfor- Table 2: Quantitative evaluastion mance scores we obtain for different contrastive diffusion steps. results for the dance-to-mine task We report the mean and standard deviations of our contrastive on the TikTok dataset. We set the
diffusion for three inference tests.
default number of diffusion steps
Musical features
Rhythms
Rhythms
Genre Accuracy
Coherence
Quality MOS
to be 80.
Metrics
Coverage
Hit
GT Music
100
I00
88.5
4.7
4.8
Methods
Beats Coverage/HHit
Foley
74.1
69.4
8.1
2.9
Munce2Music
83.5
82.4
7.0
3.0
D2M-GAN
88.47 82.3
CMT
85.5
83.5
11.6
3.0
Ours Vanilla
88.7/ 81.4
D2M-GAN
88.2
84.7
24.4
3.3
3,4
Ours Step-Intra
91.8 86.3
Ours Vanilla
89.0+11.1
83.8+-1.5
25.3-0088
3.3
3.6
Ours Sumple-Inter
90.1/ 85.5
Ours Step Intra
93.911122
90.7.1.5
25.8.10.6
3.5
Ours Sample-nnter
91.8-11.6
86.9 1.4
27.2-0.55
3.6
3.6
25
Vanilla Step Intra
VQ-D* Intra Sample-Inter
Beat coverage ScorUs
Sample-hnter
98.8
93.9
20
22
Throug Pppt (com (campplesig)
.90
89.0
91.5"
81.8
91.7"
20
19.8
19.4
66
10
85. 7
06.7
16.
84..66
5,87
-10
16
14
14.3
74.0
14.7
13.9
80
3.27
1.60
12
1.18
12.9
0.02
12.3
0.37
14.3
0.29
75
10
100
DiElutoo steps
Diffusion steps T
Figure 4: Convergence analysis in terms of diffusion steps for the dance-to-musé task on AIST+ dataset (left) and the text-to-image task on CUB200 dataset (right). We observe that our contrastive diffusion models converge at around 80 steps and 60 steps, resulting 35% steps and 40% steps less compared to the vanilla models that converge at 120 steps and 100 steps, while maintaining superior performance, respectively. We the same number of steps for training and inference.
generation works: Foley Gan et al. (2020a). Dance2Music Aggarwal & Parikh (2021), CMT Di ict al. (202I), and D2M- D2M-GAN Zhu et al. (2022a). The first three models rely on symbolic discrete MIDI musical representations, while the last one also uses VQ musical representation. The major difference between the symbolic MIDI and discrete VQ musical representations lies within the fact that the MIDI is pre-defined for each instrument, while the VQ is learning- based. The latter thus enables complex and free music synthesis appropriate for scenarios like dance videos.
Results and Discussion. The quantitative experimental results shown in Tab, Tab.[and and Tab. Our proposed methods achieve better performance than the competing methods even with vanilla version without contrastive mechanisms. Furthermore, find that the Slep-Intra setting is more helpful in increasing the beats scores, while the Sample-Inter setting yields more improvements for the genre accuracy scores. We believe this due the evaluation methods of different metries. The beats scores measure the chunk-level (i.e., the audio onset strength Ellis (2077) consistency between the GT and synthesized music samples Zhu al. (2022a). while the genre scores consider the overall musical attributes of each sample sequence in instance level. This finding is consistent with our assumptions Sec. 3.3
Convergence Analysis. We also analyze the impact of the proposed contrastive diffusion on model convergence in terms of diffusion steps. The number of diffusion steps is significant hyper-parameter for DPMs Sohl-Dickstein et al. (2015b); Nichol & Dhariwal (202 I): Austin et al. (2021); Gu et al.) ;
(2022) Kingma et al. (2021) that directly influences the inference time and synthesis quality. Previous works have shown that larger number of diffusion steps usually lead to better model performance, but longer inference times Kingma et al.] (202T): Gu et al.) (2022). We demonstrate that, with the improved mutual information via the proposed contrastive diffusion method, we can greatly reduce the number of steps needed. As shown in Fig. (left), we observe that the beats scores reach stable level approximately 80 steps, 35% less than the vanilla DPM that converges in ~ 120 steps. More ablation studies and analysis on this task can be found in the supplement.
8

---

Published as a conference paper at ICLR 2023
Table 3: FID and CLIPScore for text-to-image synthesis on CUB-200 Table 4: FID scores and top 5 and MSCOCO datasets. The VQ-D model with shows the results we reproduced by training using the original code, which can be considered as our baseline. We show the results obtained with default 80 diffusion steps in both training and inference.
classification accuracy using pre-trained ResNet101 He et al. (016) for class-
conditioned image synthesis on ImageNet 256 x 256. Our model follows the Sample- Inter contrastive setting with 80 diffusion steps both training and inference.
Datasets
CUB-200
MSCOCO
Metrics
FID
CLIPSeoree
HD.
CLIPScore
StackGAN
51.89
74.05
StackGANN+
15,30
81.59
SEGAN
18.17
32.28
AunGAN
23.98
35.49
65.66
DM-GAN
16.09
32,64
65.45
Methods FID
Acc.t
DF-GAN
14.81
21.42
66.42
ImageBART
21.19
DAE-GAN
15.19
28.12
VOGAN
15.78
DALLE
56.10
74.66
27.50
IDDPM
12.30
VQ-D (T=100)
12.97
30.17
VQ-D (T=100)
11.89
72.72
VQ-1" (T=80)
14.61
74.96
36.45
65.53
Ours (T=80)
11.74
77.63
Ours Step-Intra (T=80)
12.73
75.32
32.25
66.22
Ours Sumple-Inter (T=80)
12.61
75.50
28.76
66.79
4.2 CONDITIONAL IMAGE SYNTHESIS
Dataset. We conduct text-to-image synthesis on CUB200 Wah et al. (011 and MSCOCO datasets [in et al. (2014). The CUB200 dataset contains images of 200 bird species. Each im- age has 10 corresponding text descriptions. The MSCOCO dataset contains 82k images for training and 40k images for testing. Each image has text descriptions. also perform the class-conditioned image generation ImageNet Deng et al. (2009); Russakovsky al. 2015). Implementation details for both tasks are provided in the supplement.
Evaluations. We adopt two evaluation metrics for text-to-image synthesis: the classic FID score Heusel et al. (2017) as the general measurement for image quality, and the CLIPScore Hessel et al. 2021) evaluate the correspondence between the given textual caption and the synthesized image. For the caas-coondiioneed image synthesis, we use the FID score and a classifier accu- racy for general and input-output correspondence measurement. We compare against text-to-imaae
generation methods including StackGAN Zhang et al. (2017). StackGAN++ Zhang et al. (2018),
SEGAN Tan al. e AttnGAN Xu et al. (2018). DM OM-GAN Zhu al. (2019), DF-GAN lao et al.
(2020),
DAE-GAN Ruan et al. (2021), DALLE Ramesh et al. (2021). and VQ-Diffusion Gu et al.
(2022 For experiments on ImageNet, we list the result comparisons with ImageBART Esser et al.
Fisser et a2021a), VQGANfEsser
(202Tb), IDDPM
& Dhariwal (202)), and VQ-D et al. (2022).
Specifically, VQ-Diffusion Gu et al. (2022) also adopts the discrete diffusion generative backbone, which can be considered as the vanilla version without contrastive mechanisms. Additionally, provide more comparisons with other methods in terms of dataset, model scale and training time in the supplement for more comprehensive and fair understanding of proposed method.
Results and Discussion. The quantitative results are represented in Tab. Lab.Bladd and Tab. We observe that contrastive diffusion achieves state-of-the-as art performance for both general synthesis fidelity and input-output correspondence, and the Sample-Inter contrastive setting is more beneficial compared to Step-Intra for the image synthesis. This empirical finding again validates the contrastive settings in Sec. 3.3) where the Sample-Inter setting helps more with the instance-level synthesis quality. Notably,
assumption regarding
shown in Fig. (right), our contrastive diffusion method shows model convergence at about 60 diffusion steps, while the vanilla version converges at approximately 100 steps on CUB200 Wah et al. (20IT). which greatly increases the inference speed by 40%.
CONCLUSION
While DPMs have demonstrated remarkable potential, improving their training and inference ef- ficiency while maintaining flexible and accurate results for conditional generation is an ongoing challenge, particularly for cross-modal tasks. Our Conditional Discrete Contrastive Diffusion (CDCD) loss addresses this by maximizing the mutual information between the conditioning input and the generated output. Our contrastive diffusion mechanisms and negative sampling methods effectively incorporate this loss into DPM training. Extensive experiments on various cross-modar conditional generation tasks demonstrate the efficacy of our approach in bridging drastically differing domains.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENT
This research partially supported by NSF SCH-2123521 and Snap unrestricted gift funding. This article solely reflects the opinions and conclusions of its authors and not the funding agents.
ETHICS STATEMENTS
As in other media generation works, there are possible malicious uses of such media to be addressed by oversight organizations and regulatory agencies. Our primary objective as researchers always creating more reliable and secure Al and machine learning systems that maximally benefit our society.
REFERENCES
Gunjan Aggarwal and Devi Parikh. Dance2music: Automatic dance-driver music generation. arXiv preprint xrXiv22107.06252 2021.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018.
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In ICCV, 2017.
Jacob Austin, Daniel Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg, Structured denoising diffusion models in discrete state-spaces. In NeurlPS, 2021.
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. NeurlPS, 2016.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurIPS, 2019.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philiphe Morency. Multimodal machine learning: survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 2018.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In ICML. PMLR, 2018.
Jean-Pierre Briot, Gaëtan Hadjeres, and François- David Pachet. Deep learning techniques for music generation, volume 1. Springer, 2020.
7. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh. Openpose: Realtime multi-person
2d pose estimation using part affinity fields. IEEE TPAMI, 2019.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime nulti-person 2d pose estimation using part affinity fields. In CVPR, 2017.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
Ting Chen. Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICLR, 2020.
Yu-Jie Chen, Shin-I Cheng, Wei-Chen Chiu, Hung- Yu Tseng, and Hsin-Ying Lee. Vector quantized image-to-image translation. In ECCV, 2022.
Aiyu Cui, Daniel McKee, and Svetlana Lazebnik. Dressing in order: Recurrent person image generation for pose transfer, virtual try-on and outfit editing. In CVPR, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint / 181.00400 2018.
10

---

Published as a conference paper at ICLR 2023
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In NeurIPS, 2021.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprini urXiv:2005.00341 2020.
Shangzhe Di, Zeren Jiang, Si Liu, Zhaokai Wang, Leyan Zhu, Zexin He, Hongming Liu, and Shuicheng Yan. Video background music generation with controllable music transformer. In ACMMM, 2021.
Wan-Wen Dong. Wen-Yi Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang. Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment. AAAI, 2018.
Conor Durkan, lain Murray, and George Papamakarios. On contrastive learning for likelihood-frie inference. In ICML. PMLR, 2020.
Daniel PW Ellis. Beat tracking by dynamic programming. Journal of New Music Research, 2007.
Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer. Imagebart: Bidirectional context with multinomial diffusion for autoregressive image synthesis. NeurIPS, 2021a.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021b.
Chuang Gan, Deng Huang, Peihao Chen, Joshua Tenenbaum, and Antonio Torralba. Foley music: Learning to generate music from videos. In ECCV, 2020a.
Chuang Gan, Deng Huang, Hang Zhao, Joshua Tenenbaum, and Antonio Torralba. Music gesture for visual sound separation. In CVPR, 2020b.
Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look: Action recognition by previewing audio. In CVPR, 2020.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffuusion model for text-to-image synthesis. In CVPR, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: reference-free evaluation metric for image captioning. In EMNLP, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeulPS, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 2022.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Towards -ooreutoreeressvve language models. arXiv e-prints, pp. arXiv-2102, 2021.
11

---

Published as a conference paper at ICLR 2023
Minghui Hu, Yujie Wang, Tat-Jen Cham, Jianfei Yang, and PN Suganthan. Global context with discrete diffusion in vector quantised modelling for image generation. arXiv preprint arXiv:2112.01799 2021.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, lan Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. In ICLR, 2019.
Chin- Bhin-WeiBue Huang, Jae Hyun Lim, and Aaron C Courville. A variational perspective on diffusion-based generative models and score matching. In NeurIPS, 2021.
Shulei Ji, Jing Luo, and Xinyu Yang. A comprehensive survey on deep music generation: Multi-level representations, algorithms, evaluations, and future directions. arXiv preprint arXiv:2011.06801, 2020.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint
06950, 2017.
Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In ICCV, 2019.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In NeurlPS, 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang. Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In ICLR, 2020.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. In NeurIPS, 2018.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. In NeurIPS, 2019.
Junhyeok Lee and Seungu Han. Nu-wave: diffusion probabilistic model for neural audio upsam- pling. Proc. Interspeech 2021, 2021.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. Controllable text-to-image generation. In NeurlPS, 2019.
Ruilong Li, Shan Yang, David A. Ross, and Angjoo Kanazawa, Ai choreographer: Music conditioned 3d dance generation with aist++. In ICCV, 2021.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV. Springer, 2014.
Yan-Bo Lin, Hung- Yu Tseng, Hsin- Ying Lee, Yen- Yu Lin, and Ming-Hsuan Yang. Exploring cross- video and cross-modality signals for weakly-supervised audio-visual video parsing. In NeurlPS, 2021.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34, 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 171.0 2017.
Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprint arXiv:2103.1609, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2023
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurlPS, 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:/807.03744 2018.
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In ECCV, 2018.
Andrew Owens, Jiajun Wu, Josh McDermott, William T Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In ECCV, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML. PMLR, 2021.
Tanzila Rahman, Bicheng Xu, and Leonid Sigal. Watch, listen and tell: Multi-modal weakly supervised dense event captioning. In ICCV, 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML. PMLR, 2021.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In NeurlPS, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Shulan Ruan, Yong Zhang, Kun Zhang, Yanbo Fan, Fan Tang, Qi Liu, and Enhong Chen. Dae-gan: Dynamic aspect-aware gan for text-to-image synthesis. In ICCV, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.
Tim Sainburg, Marvin Thielk, and Timothy Q Gentner. Finding, visualizing, and quantifying latent structure across diverse animal vocal repertoires. PLoS computational biology, 2020.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015a.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML. PMLR, 2015b.
Jiaming Song and Stefano Ermon. Multi-label contrastive predictive coding. In NeurlPS, 2020a.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. In ICLR, 2020b.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR, 2020a.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models, In NeurlPS, 2020c.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishck Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2020b.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. NeurIPS, 2021.
Hongchen Tan, Xiuping Liu, Xin Li, Yi Zhang, and Baocai Yin. Cemantics-enhanced adversarial nets for text-to-image synthesis. In ICCV, 2019.
13

---

Published as a conference paper at ICLR 2023
Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao Yuan Jing, Fei Wu, and Bingkun Bao. Df- gan: Deep fusion generative adversarial networks for text-to-image image synthesis. arXiv preprint 000.:::0008.0885, 2020.
Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In ECCV, 2018.
Hung- Yu Tseng, Matthew Fisher, Jingwan Lu, Yijun Li, Vladimir Kim, and Ming-Hsuan Yang. Modeling artistic workflows for image generation and editing, In ECCV. Springer, 2020.
Shuhei Tsuchida, Satoru Fukayama, Masahiro Hamasaki, and Masataka Goto. Aist dance video database: Multi-genre, multi-dancer, and multi-camera database for dance information processing. In Proceedings of the 20th International Society for Music Information Retrieval Conference, (ISMIR). 2019.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Advances in Neural Information Processing Systems, 2021.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-20 200-2011 Dataset. Technical Report CNS-TR-201 1-001, California Institute of Technology, 2011.
Liwei Wang, Alexander Schwing, and Svetlana Lazebnik. Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space. In NeurlPS, 2017.
Xin Wang, Yuan-Fang Wang, and William Yang Wang. Watch, listen, and describe: Globally and locally aligned modal attentions for video captioning. In NAACL, 2018.
Yu Wu and Yi Yang. Exploring heterogeneous clues for weakly-superrised audio-visten visual video parsing. In CVPR, 2021.
Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang. Dual attention matching for audio-visual event localization. In ICCV, 2019.
Yu Wu, Linchao Zhu, Xiaohan Wang. Yang, and Fei Wu. Learning to anticipate egocentric actions by imagination. JEEE TIP, 30:1 143-112-1 2021.
Yu Wu, Lu Jiang, and Yi Yang. Switchable novel object captioner. IEEE TPAMI, 45(1):1162-1173. 2023. doi: 10. 109/0//AMM...2222.14445
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In CVPR, 2018.
Zipeng Xu, Tianwei Lin, Hao Tang, Fu Li, Dongliang Hc, Nicu Sebe, Radu Timofte, Luc Van Gool, and Errui Ding. Predict, prevent, and evaluate: Disentangled text-driven image manipulation empowered by pre-trained laniunnl model. arXiv preprint arXiv:2/11.133.. 2021.
Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. CVPR, 2016.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistie image synthesis with stacked generative adversarial networks. In CVPR, 2017.
Han Zhang, Tao Xu. Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE TPAMI, 2018
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for ext-to--maae generation. CVPR, 2021.
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In CVPR, 2019.
14

---

Published as a conference paper at ICLR 2023
Ye Zhu, Yu Wu, Yi Yang, and Yan Yan. Describing unseen videos via cooperative dialog agents. In ECCV, 2020.
Ye Zhu, Yu Wu, Hugo Latapie, Yi Yang, and Yan Yan. Learning audio-visual correlations from variational cross-modal generation. In ICCASP, 2021a.
Ye Zhu, Yu Wu, Yang, and Yan Yan. Saying the unseen: Video descriptions via dialog agents, In TPAMI, 2021b.
Ye Zhu, Kyle Olszewski, Yu Wu, Panos Achlioptas, Menglei Chai, Yan Yan, and Sergey Tulyakov. Quantized gan for complex music generation from dance videos. ECCV, 2022a.
Ye Zhu, Yu Wu, Nicu Sebe, and Yan Yan. Vision+ x: A survey on multimodal learning in the light of data. arXiv preprint vrXiv:2210.02884 2022b.
15

---

Published as a conference paper at ICLR 2023
We present more related work in Appendix A Details about the theoretical deviation and training algorithms are included in Appendix B In Appendix C
we describe additional experimental details and ablation analysis, including the resources required for training and inference. More qualitative examples are provided in Appendix D Further discussions including the broader impact are included in Appendix E
A MORE RELATED WORKS
In addition to the fields of Diffussin Probabilistic Models, Contrastive Representation Learning. and VQ Representations for Conditional Generation discussed in the main paper, work is also closely related to the multi-modal learning and generation fields.
The research topic of multimodal learning, which incorporates data from various modalities such as audio, vision, and language has attracted much attention in recent years Baltrušaitis et al. (2018); Zhu
al. (202bb: Wu et al. 2023). General audio and visual learning works typically seek to investigate their correlations from the intrinsic synchronization nature Aytar et al. (2016); Korbar et al. (2018); Owens & Efros (2018); Owens et al.] (2016); Arandjelovic & Zisserman (2017), and then utilize them in various downstream audio-visual tasks such as audio-visual action recognition Kazakos et al.
(2019;
Gao al.
(2220), audio-visual event localization and parsing Tian et al.) (2018); [huu et al.
;; Wu & Yang (2021), and audio-visual captioning Rahman et al. (0019); Wang et al. (2018). Works generate music from visual or/and motion data have also been widely explored in recent years Gan et al. (2020a); Di et al. (2021); Aggarwal & Parikh (2021): Zhu et al.
a Wu et al.
(2019);
(2022a). For vision and language area, the text generation from visions are extensively explored image and video captioning task /hu et al. (2020) (021b); Anderson et al. (2018); You et al. (2016); Wang et al. (201)). At the same time, works on image/video generation from text have also attracted much attention with recently released largescale models Radford et al. (2021) Li et al. (2019) Ruan ct al. (22TT) Ramesh et al. (202T).
B DETAILED PROOF AND TRAINING
B.1 LOWER BOUND OF CDCD LOSS
We show that proposed CDCD loss has a lower bound related mutual information and number of negative samples N. The derivations below are similar to those from Oord et al. (2018):
Po (sulc) (=()a)
LODCD == Ez[-log
(6a)
pa(za (c) Do(
Σ EEE
pel (o(2/
no(=3)
EEz log [1 + Po(20)
po Po(=6lc)
(6b)
(=
2EZ2 pe (=()
~ EEz log [1 + N Pa
Ez, po(=glc)
(6c)
po co())
po (=))
= Ez log(1 + N (2()
(6d)
Po (z0lc))
2 Ex log(N p=(=0)
(6e)
po (o(2olc)
log(N) T(20, e)-
(6f)
16

---

Published as a conference paper at ICLR 2023
B.2 CONVENTIONAL VARIATIONAL Loss
The conventional variational loss Cvb is derived as follows Sick-Dickstein et al. (2015b):
(xb(2) = Eyl- l-og (o(TT)))) ( (2o) (20)
log (o(
po 11) log
B p(ar))
(x21 (20)
Pa( [4))
g( 1(
log Po (m))
-ogg- (ogpp(TT)
log
g( 1 22, To)
q (x0)
121
p(rT)
po (e( log (0
=B, log
log
( ro)
a(0 [r, v0)
(g)Daagg((xxooo))llllla +
Drul Ipo log (x
t>
B.3 Cvb WITH CONDITIONING PRIOR
Following the unconditional conventional variational loss, we then show its conditional variant with the conditioning c as prior, which has also been adopted in Gu al. (2022).
Lob((r,) c) Lo + L1 + -T LT
Co = log Pe (000=11100
(8)
f--1 1/
LT = (et.
B.4 STEP-WISE AND SAMPLE- WISE CONTRASTIVE DIFFUSION
Below, we show the full derivation for the step-wise parallel contrastive diffusion loss. Given that the intermediate variables from 21.7 are also taken into account in this step-wise contrastive diffusion, we slightly modify the initial notation f(zo, e) c( Pe(sorrec) Pe (30:3)
pa(ao]) from Eq, in the main paper to Po(Po)
c(2,c)
LCDCD-Step = -Ez flog
(9a)
f(2, c) + E
Eviez f(2),c)
Ez log [] + Eezez.f22..
(9b)
((,,c)
po(2o:T)
Po po(
Ez log [1
(9c)
pe po(zor (...Tl (c)
ZTEZ po po(=
po(2o:T)
NE2 po (o(
2 Ez log [1
(same as Eq.(6c)
(9d)
Pe Pe(zo:T
pe (=)_T)
g(21.T120) 20)
N proo (c)
Ez log
q(21.T) 20) (conditional po)
(9e)
po (e(
Do po(20.11c)
log.N Ez E log po(20.T(c) 9 20)
Eog
(9
( 20)
=Lb c) - C Σ (vb(2*,))
(9g)
2EZ
17

---

Published as a conference paper at ICLR 2023
Algorithm I Conditional Discrete Contrastive Diffusion Training. The referenced equations can be found in the main paper.
Input: Initial network parameters 0, contrastive loss weight X, learning rate number of negative samples N. total diffusion steps I' conditioning information contrastive mode m E (Step, Sample).
1: for each training iteration do
2: t~ Uni form (11, ..., T})
3: 21 Sample from g( 12
L.b. + -i=i.....f L. Eq.
5:
if m == Step then
6:
for ... do
7:
z/ t- Sample from q(=/\=/_11 b from negative variables in previous steps
8:
end for
LCDCD -
x EEL, Eq.
else if m Sample then
11:
for jurj=..... ... N do
12:
2, Sample from q(2e =b: from negative variables in step
13:
end for
14:
LCDCD = N Σ C2gg
> Eq. 4
15:
end if
16:
+ Lob ALoDCD Eq.
8 VoL.
18:
end for
In the above (9g), C stands for a constant that equals to log.V, which can be further adjusted by the weight select for the CDCD loss in Eq. S
Similarly for the sample-wise auxiliary contrastive diffusion, the loss can be derived as follows:
f(20.0) c)
CCDCD-Samie
-Ez [log
(10a)
f(20, c) + >>EE f(
po(20)
NEz, po(c8le)
Ezz log [1
(10b)
po (0
po(=6)
((210T/20)
N pa pa(20
2 Ez E.g logl
(10c)
po (o(2olc)
9 ( 200
po (c)
N Ez E (o(
~ Egg lloo
(10d)
(2 (20)
g (z0)
Egl (-og po (20 Ze, c)]
C Σ E,I-log log po 2,, c)].
(10e)
#/eZ'
Note that from high-level perspective, our contrastive idea covers different concepts, while conventional contrastive learning usually focuses only on the negative samples. In our case, due the unique formulation of diffusion models that bring the diffusion steps into the methodology design, we consider the contrast within the context of "negative samples" and "negative steps" (also corresponds to the "negative intermediate steps").. In the deviation above, we use the symbols Z and q to distinguish between these two concepts.
B.5 CONDITIONAL DISCRETE CONTRASTIVE DIFFUSION TRAINING
The training process for the proposed contrastive diffusion explained in Algo.
C ADDITIONAL EXPERIMENTAL DETAILS AND ANALYSIS
C.I DANCE-TO-MUSIC TASK
Implementation. The sampling rate for all audio signals is 22.5 kHz our experiments. We use 2-second music samples in Zhu et al. (202aa) for main experiments, resulting in 44,100 audio
18

---

Published as a conference paper at ICLR 2023
data points for each raw music sequence. For the Music VQ-VAE, fine-tuned Jukebox Dhariwal jet al. (2020) on our data to leverage its pre-leamed codebook from a large-scale music dataset (approximately 1.2 million songs). The codebook size K is 2048, with token dimension d_ = 128, and the hop-length I. is 128 in our default experimental setting. For the motion module, we deploy a backhone stacked with convolutional layers and residual blocks. The dimension size of the embedding we use for music conditioning is 1024. For the visual module, we extract I3D features Carreira & Zisserman (017)) using a model pre-trained on Kinectics Kay et al. (2117) as the visual conditioning information, with a dimension size of 2048. In the implementation of our contrastive diffusion model, we adopt transformer- based backbone to learn the denoising network po. includes transformer blocks, in which each block is consists of full-attention, tross-attention and feed-forward network, and the channel size for each block is 1024. We set the initial weight for the contrastive loss as A = 5e - 5. The numbers of intra- and inter-negative samples for each GT music sample are both 10. The AdamW Loshchilov & Hutter (017) optimizer with 31 = 0.9 and 82 = 0.96 is deployed in our training, with learning rate of 4.5e- - 4. We also employ an adaptive weight for the denoising loss weight by gradually decreasing the weight as the diffusion step increases and approaches the end of the chain. The visual module, motion module, and the contrastive diffusion model are jointly optimized. The architecture of adopted motion encoder is shown in Tab. [] which is the same as in Zhu et al. (2022a).
Table 5: Architecture for the motion encoder:
x 1, stride=1, Conv 256, LeakyReLU
Residual Stack 256
3 x 1, stride=1, Conv 512, LeakyReLU
Residual Stack 512
3 1, stride=1, Conv 1024, LeakyReLU
Residual Stack 1024
3 x 1 stride=1, Cony 1024, LeakyRel.U
4 x 1, stride=1, Conv I
Other than the aforementioned implementation details, we also include the mask token technique that bears resemblance to those used in language modelling [Devlin et al. (2018) and text-to-image synthesis Gu et al. (2022) for our dance-to-music generation task. We adopt truncation rate of 0.86 in our inference.
MOS Evaluation Test. We asked total of 32 participants to participate in our subjective Mean Opinion Scores (MOS) music evaluations Zhu et al. (2022a); Kumar et al. (2019), among which 11 of them are female, while the rest are male. For the dance-music coherence test, we fuse the generated music samples with the GT videos as post-processing. We then asked each evaluator to rate 20 generated videos with score of (least coherent) to 5 (most coherent) after watching the processed video clip. Specifically, the participants are asked to pay more attention to the dance-music coherence in terms of the dance moves corresponding to the music genre and rhythm, rather than the overall music quality, with reference to the GT video clips with the original music. As for the overall quality evaluations, we only play the audio tracks without the video frames to each evaluator, As before, they are asked to rate the overall music quality with a score of (worst audio quality) to 5 (best audio quality).
Training Cost. For the dance2music task experiments on the AIST+ dataset, we use 4 NVIDIA RTX A5000 GPUs, and train the model for approximately days. For the same task on the TikTok dance-music dataset, the training takes approximately 1.5 days on the same hardware.
Complete Results for Contrastive Settings. As discussed in our main paper, there are four possible combinations for contrastive settings given different contrastive diffusion mechanisms and negative sampling methods. Here, we include complete quantitative scores for different contrastive settings in Tab. We observe that all the four contrastive settings, including the Step-Inter and Sample- Intra settings that are not reported in our main paper, help to improve the performance. As we noted, amongst all the settings, Step-Intra and Sample-Inter are more reasonable and yield larger improvements for intra-sample data attributes (i.e., beats scores) and instance-level features (i.e., genre accuracy scores).
19

---

Published as a conference paper at ICLR 2023
Table 6: Complete quantitative evaluation results for the dance-to- mo-music generation task the AIST+ dataset. We report the mean and standard deviations of our contrastive diffusion for three inference tests.
Musical features
Rhythms Coverage
Rhythms
Genre
Coherence MOS
Quality
Metrics
Hit 1
Accuracy
MOS 1
GT Music
100
100
88.5
4.7
4.8
Foley Gan al. (2020a
74.1
69.4
8.1
2.9
Dance2Music
Aggurwal Pankh 2021
83.5
82.4
7.0
3.0
CMT
85.5
83.5
11.6
3.0
D2M-GAN et 2022a
88.2
84.7
24.4
3.3
3.4
Ours Vamila
89.0.1.11
83.8=1.5
25.3110.8
Ours Step-Intra
93,911.2
90.7=1.5
25.8=0.6
3.6
Ours Step-Inter
92.4+1.0
88.9=1.7
24.3=0.7
Ours Sample-Intra
91.5+1.7
84.6=1.6
26.0=008
Ours Sample-Inter
91.811.6
86.9=144
27.2.0.5
Table 7: Ablation results for different music lengths on the AIST+ dataset.
Length
Methods
Beats Coverage f
Beats Hit
Genre Acc. 1
D2M-GAN Zhu al. (2022a)
88.2
84.7
24.4
Ours Vanilla
89.0
83.8
25.3
28
Ours Step-Intra
93.9
90.7
25.8
Ours Sample-Inter
91.8
86.9
27.2
D2M-GAN Zhu al. (2022a)
87.1
83.0
23.3
Ours Vanila
86.2
81.8
24.6
4s
Ours Step-Intra
93.1
86.4
25.3
Ours Sample-Inter
91.4
83.9
26.1
D2M-GAN Zhu et al. (2022a)
Ours Vanilla
84.8
81.1
22.7
6s
Ours Step-Intra
87.9
83.2
22.9
Ours Sample-Inter
86.3
81.6
23.0
Ablation on Music Length. Although we use 2-second musical sequences in the main experiments to make for consistent and fair comparisons with Zhu et al.) (2022a), framework also synthesize longer musical sequences. In the supplementary, we show our generated music sequences in 6- seconds. The quantitative evaluations terms of different musical sequence lengths are presented where we show better performance when synthesizing longer musical sequences.
with
Tab. 7
C.2 TEXT-TO-IMAGE TASK
Implementation. For the text-to-image generation task, we adopt VQ-GAN Esser et al. (2021b) as the discrete encoder and decoder. The codebook size K is 2886, with token dimension d, 256. VQ- GAN converts a 256 x 256 resolution image to 32 x 32 discrete tokens. For the textual conditioning, we employ the pre-trained CLIP Radford al. (202 I model encode the given textual descriptions. The denoising diffusion model pe has 18 transformer blocks and channel size of 192, which is similar model scale the small version of VQ-Diffusion Gu al. (022). We use
5e 5 as the contrastive loss weight. Similar to the dance-to- music task, we also use the adaptive weight that changes within the diffusion stages. We keep the same truncation rate 0.86 as in our dance-to-music experiment and in Gu al. (2022). Unlike in the dance-to sance-to-muusc experiments, where we jointly learn the conditioning encoders, both the VQ-GAN and CLIP models are fixed during the contrastive diffusion training.
Training Cost. For the text2image task experiments on the CUB200 dataste, the training takes approximately days using 4 NVIDIA RTX A5000 GPUs. For the same experiments the MSCOCO dataset, we run the experiments on Amazon Web Services (AWS) using 8 NVIDIA Tesla V100 GPUs. This task required 10 days of training.
20

---

Published as a conference paper at ICLR 2023
Table 8: Comparisons in terms of the task, datasets, model scale, and training time between
proposed CDCD and some other diffusion-basee works.
Tasks
Dataset
Method
Puram.
Training time
Dance-to-Musii
AIST+ et 2021
Ours CDCD
350M
V100 days
Dunce-to-Mush
TikTok Zhu A2224
Ours CDCD
350M
V100 days
Text-to-Iman
CUB200 wab a.
2011
Ours CDCD
35M
20 V100 days
Text-to-Imuge
Li alal
Ours
35M
V100 days
Class conditioned
ImageNar
Ours CDCD
370 M
160 V100 days
Image Synthesis
ImageNet
ADMDhariwal & Nichol 2021
150-1000 V100 days
Imuge Synthesis
ImageNet
LDM LDM Rombuch all
400M
V100 days
Outguts w/ Weaker
Inpur
Outputs from Our Contmstive Diifusion Model
Conespondence
this bird is all black and has a long pointy beak
white and black spotted
bird.
this bird pointy yellow hill, black head and wings, yelloo throat.
wings that
brownand has an orange bill
this bird black and
green color, black beak
Figure 5: More qualitative results from our text-to-image experiments on CUB200 dataset. We show examples of the text input (left column), the synthesized images from our contrastive diffusion model with 80 diffusion steps and FID score 12.61 (middle column), and the output from existing method Gu et al. 2022 with 100 diffusion steps and a FID score 12.97.
C.3 CLASS-CONDITIONED IMAGE SYNTHESIS TASK
Implementation. For the class-conditions image synthesis, also adopt the pre-trained VQ- GAN Esser et al. (b)21b) as the discrete encoder and decoder. We replace the conditioning encoder with class embedding optimized during the contrastive diffusion training. The size of the conditional embedding is 512. Other parameters and techniques remain the same, as in the text-to-image task.
Training Cost. For the class-conditioned experiments on the ImageNet, V100 GPUs running on AWS. This task required 20 days of training.
8 NVIDIA Tesla
D MORE QUALITATIVE RESULTS
D.1 GENERATED MUSIC SAMPLES
For qualitative samples of synthesized dance music sequences, please refer to our anonymous page in the supplement with music samples. In addition to the generated music samples on AIST+ TTuchida eet al. (2019); Li et al. (202T) and TikTok Dance-Music Dataset Zhu et al. (2022a), we also include some qualitative samples obtained with the music editing operations based the dance-music genre
21

---

Published as a conference paper at ICLR 2023
Outpurs W Weaker
Input
Outyuts Contrastive Ditlusion Model
Cumspunilnoe
kitchen and dining room area with large window.
an office ith desk computer chair and laptop.
the plane flying through a clear blue sky.
motorists and bicyclists in a busy city square.
small wooden table covered with delicious vegetabless
many sheep graze a grassy pasture in a valley.
assorted Items a counter kitchenette.
bathroom with walkk-nn closet and Jacuzzi tub.
train is running by station.
Figure 6: More qualitative results from our text-to to--tag--geeee experiments on COCO dataset. We show examples of the text input (left column), the synthesized images from our contrastive diffusion model with 80 diffusion steps and FID score of 28.76 (middle column), and the output from existing method Gu et al. (2022) with 100 diffusion steps and a FID score 30.17.
annotations from AIST++ Specifically, we edit the original paired motion conditioning input with different dance-muss genre using a different dance choreographer.
Discussion on Musical Representations and Audio Quality. It is worth noting that we only compare the overall audio quality with that of D2M-GAN Zhu al. (2022a). This due to the nature of the different musical representations in the literature of deep-learning based music generation Gan cet al. (2020a): Dong et al. (2018); Huang et al. (2019); Gan et al. (2020b); Aggarwal Parikh (2021). There are mainly two categories for adopted musical representations in previous works: dre-defined symbolic and learning- brased representations/ Ji et al. (2020): Briot et al. (2020). For the former symbolic music representation, typical options include ID piano-roll and 2D MIDI-basa representations. While these works benefit from the pre-defined music synthesizers and produce music that does not include audio noise, the main limitation that such representations
22

---

Published as a conference paper at ICLR 2023
Outgrats w/ w/W Wenker Comsspondano
Input
Outpuls from Our Contrastive Dillusion Model
Class label: snail
Class label: bookcase
Class label: fountain
Class label:
Class label: mirrowave
Class label: plate
Class label: lakeside
Class label: dough
Class label: cheeseburger
Figure 7: More qualitative results from our dlass-conditional image synthesis experiments on ImageNet. We show examples of the text input (left column), the synthesized images from our contrastive diffusion model with 80 diffusion steps and FID score of 11.74 (middle column), and the output from existing method Gu al. (2022) with 100 diffusion steps and FID score 11.89.
usually limited to single specific instrument. which hinders their flexibility be applied in wider and more complex scenarios such as dance videos. In contrast, the learning- baasnd music representations (i.e., musical VQ in our case) rely on well-trained music synthesizers decoders, but can be used as unified representation for various musical sounds, e.g., instruments or voices. However, the training of such music encoders and decoders for high-quality audio signals itself remains challenging problem. Specifically, high-quality audio is form of high-dimensional data with extremely large sampling rate, even compared to high-resolution images. For example, the sampling rate for CD-quality audio signals is 44.1 kHz, resulting in 2,646,000 data points for one-minute musical piece. To this end, existing deep learning based works Dhariwal et al. (2020); Kumar et al. (2019) for music generation employ methods to reduce the number of dimensions, e.g., by introducing hop
23

---

Published as a conference paper at ICLR 2023
lengths and a smaller sampling rate. These operations help to make music learning and generation more computationally tractable, but also introduce additional noise in the synthesized audio signals.
In this work, adopt the pre-trained JukeBox model Dhariwal al. (2020)
music encoder and decoder for the musical VQ representation. The adopted model has a hop length of 128, which corresponds the top-level model from their original work Dhariwal et al. 2020). Jukebox employs models: top- middle-, and bottom-level, with both audio quality and required computation increasing from the first the last model. As an example, in the supplemental HTML page, we provide music samples directly reconstructed from JukeBox using the top-level model we employ in our work, compared to the ground truth audio. While they allow for high-quality audio reconstruction (from the bottom-leve model, with a hop length of 8), it requires much more time and computation not only for training but also for the final inference, e.g., . hours to generate 20-second musical sequence. As the synthesized music from the top-level model includes some audible noise, we apply a noise reduction operation (Sainburg et al. 2020). However, the overall audio quality is not primary factor that we specifically address in this work cross-modal conditioning and generation, it largely depends on the specific music encoder and decoder that are employed. This explains why we report similar MOS scores in terms of the general audio quality.
D.2 SYNTHESIZED IMAGES
We present more qualitative examples for text-to-image synthesis and class-conditioned image synthesis in Fig. 5 Fig. 6 and Fig. 7
E FURTHER DISCUSSION ON THE CDCD LOSS
In this section, we provide our further discussion on the proposed CDCD loss in terms of various aspects, including its relevance to the existing auxiliary losses, the impact of the CDCD strength, as well as additional experimental results.
E.1 CDCD AS AUXILIARY Loss
While the diffusion models are typically trained and optimized with the conventional variational lower bound loss Lus we described in the main paper and Appendix B.2 there are several different types of auxiliary losses proposed to further regularize and improve the learning of diffusion models. Specifically, Dhariwal & Nichol (2021) introduces the idea of classifier based guidance for the diffusion denoising probabilistic models with continuous state space. Classifier- frree guidance is proposed in Ho & Salimans (2022). In the area with discrete diffusion formulations Austin al. (202T): Gu et al. 2002)) an auxiliary loss that encourages the model to predict the noiseless token at the arbitrary step is adopted and proven help with the synthesis quality.
Similar to the previous cases, we consider the proposed CDCD loss as a type of auxiliary losses, which seeks provide additional guidance better learn the conditional distribution p(x|c). Specifically, the classifier-fre guidance Ho & Salimans (2022) propose to randomly discard conditioning while learning a conditional diffusion generative model, which bears resemblance to our introduced downsampled contrastive steps in Appendix E.3
E.2 IMPACT OF CDCD STRENGTH
We further show the ablation studies on the parameter X, which is the weight of our proposed CDCD loss that characterize the strength of this contrastive regularizer.
We conduct the dance-to-music generation experiments with different values of X, and show the results in Tab. 9 As we observe from the table that the performance in terms of the beat scores are relatively robust for different À values ranging from 4e to 5e - 5. At the same time, we empirically observe that with a large value of X, the model converges faster with less training epochs.
In case of the image synthesis task, we are rather cautious on the strength of the imposed contrastive regularizer. Intuitively, the proposed CDCD loss encourages the model to learn slightly different distribution for negative samples, which could impose a trade-off between the one for the actual data given a specific conditioning. Therefore, while a larger value of À helps with the learning speed,
24

---

Published as a conference paper at ICLR 2023
Table 9: Ablation results terms of À the dance-to-music task.
Value
Beats Coverage
Beats Hit
Se-4
91.0 93.4
90.2
le-5
93.3
5e-5
93.9
90.7
Table 10; Ablation results in terms of À on the dance-to-music task.
Method
Beats Coverage
Beats Hit
Full (Ta 100)
90.5 93.9
90.7
T. 80
93.4
To 60
92.7
90.4
empirically set the À to be 5e 5. Note that this value adapted from the weight for other auxiliary
losses in previous works Gu al. 2022).
E.3 DOWNSAMPLED CONTRASTIVE STEPS
While we show the performance of complete step-wise contrastive diffusion in the main paper, we discuss here alternative way implement the proposed method with less computational cost, by downsampling the contrastive steps in the diffusion process. Specifically, we randomly downsampled the steps with the proposed CDCD loss, which shares the similar spirit as in the class-free guidance Ho & Salimans| (2022) to randomly drop out the conditioning. The experimental results are listed in Tab. 10 where there little performance drop with downsampled contrastive steps.
25


Reviews:
[{"review_id": "quKDG1kqRr", "reviewer": "Reviewer_3LEn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a so-called Discrete Contrastive Diffusion model for cross-modal music and image generation. The main idea is to explicitly enhance input-output connections by maximizing their mutual information instead of implicitly learning such relationships. Specifically, the authors try to combine diffusion training and contrastive learning via the conventional variational objectives. Experiments on three different tasks (i.e., dance-to-music generation, text-to-image synthesis, and class-conditioned image synthesis) verify the effectiveness of the proposed method by showing higher synthesis quality and faster inference speed compared to other existing diffusion models. \n\n", "summary_of_the_review": "Due to enough novelty and academic contributions as analyzed above, I suggest accepting this paper after some minor revisions.", "strength_and_weaknesses": "Strengths:\nTo the best of my knowledge, this paper is the first to combine diffusion training and contrastive learning to design an effective generative model. The authors propose the CDCD loss and design two contrastive diffusion mechanisms to achieve this goal. The enhanced diffusion model is designed in a reasonable manner. The paper is well-written and well-organized. The proposed method is clearly described with all necessary analyses and mathematical details. Experiments have been conducted to demonstrate the effectiveness of the proposed method by showing higher synthesis quality and faster inference speed compared to other existing diffusion models.\n\nWeaknesses:\nBy zooming in the synthesized images shown in Fig. 6 and Fig. 7, it seems to me that they are not as good as some SOTA results obtained by DALLE-V2, Imagen, etc. The authors need to explain the reasons and show/compare more qualitative results on the tasks of text-to-image synthesis and class-conditioned image synthesis in their supplemental materials. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XCNs8U5CYi", "reviewer": "Reviewer_6GAW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a Conditional Discrete Contrastive Diffusion (CDCD) loss to enhance input-output connections by maximizing their mutual information. The author also designs two contrastive diffusion mechanisms to  incorporate  $L_{CDCD}$  into the denoising process.  Diverse multimodal conditional synthesis tasks have been evaluated and the proposed methods achieve higher or competitive general synthesis quality. \n", "summary_of_the_review": "\nThis paper introduces a simple yet effective method to improve the input-output connections via maximized mutual information. And both proposed DPMs are proven effective and achieve state-of-the-art scores in our evaluations. Compare to previous methods, the proposed methods show better performance. Overall, I think this paper is good.\n\n", "strength_and_weaknesses": "###### Strength：\n\n· This work first combines diffusion training and contrastive learning, which provides a new perspective on the research of conditional DPMs.\n\n· The proposed object function and contrastive diffusion mechanisms are shown promising quantitative and visual results. \n\n· The frame work improves the convergence of diffusion models, reducing the number of required diffusion steps by more than 35% on two benchmarks.\n\n· Proper INTRA- AND INTER-NEGATIVE SAMPLING methods are adopted and relevant experiments revealed their effectiveness. \n\n###### Weakness：\n\nI did't see any weaknesses", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QlJw7QGuHq", "reviewer": "Reviewer_MzmQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Existing diffusion-based cross-modal generation methods mainly establish the cross-modal relationships by incorporating the cross-modal prior model into the variational lower bound of the diffusion model. However, the authors claim that this method may lead to the loss of the cross-modal correspondence in the denoising process. To overcome this, the authors propose Conditional Discrete Contrastive Diffusion (CDCD) loss, which enhances the cross-modal relationships by constructing negative samples and introducing contrastive loss in the training.", "summary_of_the_review": "The idea of this paper makes sense. However, the experimental results are not satisfactory. There are some issues with the proposed approach. Therefore, I believe that the paper is below the acceptance bar.\n\n=======================================================================================================\n\nThank the authors for addressing my concerns. After reading the authors’ responses and peer reviews, I decide to change my score to 6. My main concern is the experiment results fall behind the VQ-D-B. The new experiment results show that the contrastive loss improves the FID of VQ-D-B from 19.7 to 18.44, which addresses my main concern.", "strength_and_weaknesses": "Strength:\n1. The idea of introducing contrastive loss to enhance the cross-modal relationship is reasonable. The proposed pipeline is well demonstrated in Figure2.\n2. The proposed approach is clearly described, which makes the manuscript easy to follow.\n\nWeaknesses:\n1. The authors should provide more evidence to support the claim that incorporating prior into the variational lower bound can lead to the loss of the cross-modal correspondence.\n2. Would the objective of enhancing cross-modal relationships contradict to increase the sample quality? How would the authors balance the variational loss and contrastive loss?\n3. In the construction of inter-negative samples, the authors take all the images x’ other than x as negative samples. In this way, similar images may also be considered negative samples. How would the authors address this?\n4. In the text-to-image generation task, the authors use the VQ-diffusion-S as the baseline. The results of the proposed approach slightly outperform the VQ-diffusion-S while falling behind the VQ-diffusion-B greatly. The authors should verify the effectiveness of the proposed approach on larger models.\n5. In Table3, the performance of the proposed approach falls behind the DF-GAN.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KuNwhzlr7-", "reviewer": "Reviewer_uFmy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a regularizer that enforces that the condition embedding and the latent embedding of the input datum share the maximal mutual information between them. This regularizer then helps the diffusion process because the condition is rich with information about the latent embeddings, and thus helping the denoising process. The regularizer term is inspired by a contrastive loss in which the numerator indirectly measures the mutual information between the condition and latent embeddings, and the denominator includes information from conditions and embeddings that should not be related to each other (i.e., mimicking negative samples). The paper also discusses two ways of applying this regularizer: step-size parallel diffusion and sample-wise auxiliary diffusion. The former allows parallel evaluations of the denoising diffusion process while the latter does not. The paper presents experiments on dance-to-music generation, text-to-image synthesis, and class-conditioned image synthesis showing improvements over the included baselines.", "summary_of_the_review": "Overall, I think the idea of using negative samples by means of a contrastive-based loss  is interesting. However, there are practical concerns that make me doubt the proposed method may be useful to speed up the diffusion processes. This is because while the proposed method indeed requires fewer steps, it is unclear if the step became more costly in terms of time or even computational resources. Second, the reproducibility given the state of the narrative falls short.", "strength_and_weaknesses": "Strengths:\n\nI find the idea of using negative samples as part of the contrastive-loss-based regularizer to maximize the mutual information between the condition and latent interesting and novel. I think using these negative samples can indeed inform the model about what not to generate and constraint better the data generation.\n\nWeaknesses:\n1. While I find the idea novel, I think the method is quite elaborate and can imply more computational resources. \n- First, including negative samples as part of the loss will increase computation making the computational cost even more expensive for a denoising diffusion process. \n- Second, while the proposed stepwise diffusion allows parallelization, it still requires more resources that can increase the cost of an already expensive denoising diffusion process.\n\n2. Insufficient experiments:\n- The paper lacks an ablation study about the parameter $\\lambda$ which controls the contribution to the total loss of the proposed regularizer. According to Section 4.1, $\\lambda$ was set to 5e-5, which I find the value too low. It is unclear how to set this parameter from the experiments. More importantly, what the impact of increasing the value of $\\lambda$ and thus enforcing the regularizer stronger on performance is not clear.\n- The paper also misses an ablation study about the latent encoder. What is the effect of not even using one? Wouldn’t a latent encoder likely reduce the information (in the information theoretical sense) from the original input. Can the proposed methods work on raw signals, i.e., latents are the input signals directly.\n- The experiments in paragraph “Results and Discussion” and Fig. 4 state that because the proposed method requires fewer steps to converge the proposed method is faster to converge. While the experiments show a reduction in steps, it is unclear about the cost of each step in terms of time in the proposed method. I think having a paper demonstrating that the proposed method indeed reduces the time of convergence is more important than the number of steps.\n- The experiment in Table 3 is missing a more appropriate baseline: Stable Diffusion. I think using Stable Diffusion instead of DALLE makes more sense because Stable Diffusion also uses a latent representation while DALLE does not.\n\n3. From the theoretical perspective, is there a proof showing that the proposed regularizer combined w/ the variational-bound-based loss still preserves the Langevin dynamics in some way? I think discussing the theoretical guarantees can be informative.\n\n================= Post-Discussion =================\n\nAfter engaging with the authors in the discussion, I still think the paper can benefit from reporting wall-clock time of the training phase, add more extensive ablation studies, and add Stable Diffusion as a baseline. For the most part, most of my concerns about clarity were addressed. Nevertheless, because I think there are missing experiments, I cannot champion the paper fully as I think the paper can benefit from another revision. I will slightly increase my rating to 6 - marginally above the acceptance threshold.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster