Images that Sound: Composing Images and Sounds on a Single Canvas
Authors: Ziyang Chen
Daniel Geng
Andrew Owens
Venue: NeurIPS 2024 poster (2024)
Keywords: Multimodal Learning
Audio-Visual Learning
Creative AI
Diffusion Models
Primary area: speech_and_audio
TL;DR: We compose pretrained text-to-image and text-to-audio diffusion models to create spectrogram that can be seen as images and played as sound.

Abstract: Spectrograms are 2D representations of sound that look very different from the images found in our visual world. And natural images, when played as spectrograms, make unnatural sounds. In this paper, we show that it is possible to synthesize spectrograms that simultaneously look like natural images and sound like natural audio. We call these visual spectrograms *images that sound*. Our approach is simple and zero-shot, and it leverages pre-trained text-to-image and text-to-spectrogram diffusion models that operate in a shared latent space. During the reverse process, we denoise noisy latents with both the audio and image diffusion models in parallel, resulting in a sample that is likely under both models. Through quantitative evaluations and perceptual studies, we find that our method successfully generates spectrograms that align with a desired audio prompt while also taking the visual appearance of a desired image prompt.

Images that Sound: Composing Images and Sounds on a Single Canvas
Ziyang Chen
Daniel Geng
Andrew Owens
University of Michigan
https://ificl witpiiicclggi.u........ github. io/ mmgges- that sound/
Colar image
"castle geayscale, lithograph
"slgers, crayscale, black background"
Spectrogram
Frequancy cy
"bell ringing"
reiger growling"
Waveform
Figure 1: Images that sound. We diffusion models to generate visual spectrograms (second row) that look like natural images, which call images that sound. These spectrograms he converted into natural sounds (third row) using pretrained vocoder, or colorized to obtain more visually plcasing results (first row). Please refer to our website to listen to the sounds.
Abstract
Spectrograms are 2D representations of sound that look very different from the images found in our visual world. And natural images, when played as spec- trograms, make unnatural sounds. In this paper, we show that it is possible synthesize spectrograms that simultaneously look like natural images and sound like natural audio. We call these visual spectrograms images that sound. Our approach is simple and zero-shot, and leverages pre-trane rreitraine text-to-image and text-to-spectrogram diffusion models that operate in a shared latent space. During the reverse process, denoise noisy latents with both the audio and image dif- fusion models in parallel, resulting in a sample that is likely under both models. Through quantitative evaluations and perceptual studies, find that method successfully generates spectrograms that align with a desired audio prompt while also taking the visual appearance of desired image prompt. Please our project page for video results: https: / /ificl. ..t.ggitio github. 10/ images------- that whaatssuuue sound/
Introduction
The spectrogram is ubiquitous low-dimensional representation for audio machine learning that plots energy within different frequencies over time. But also widely used as a tool for converting
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

IMAGES
SPECTROGRAMS
IMAGES THAT SOUND
"casLle"
"bell ringing"
"castle" "bell ringing"
"kicten"
"cat mecwing"
"kicten" & "cot mecwing"
Figure 2: Images vs. spectrograms. We show grayscale images generated from Stable Diffusion [96] on the left, followed by log-mel spectrograms generated from Auffusion [118] in the middle, and our generated images that sound results on the right.
sound into a visual form that can be-ai least partially-perceivee by sight. For example, in this representation (Fig. 2), event onsets look to a human observer like lines, and speech looks like a sequence of waves and bands. This insight is commonly used within the audio community, which frequently repurposes pretrained visual networks for audio tasks, often with only relatively minor modifications [48, 90, 69, 68, 34, 118, 112].
We hypothesize that the success of spectrograms these roles is due in part to the fact that they share many statistical properties with the distribution of natural images, providing visual structures like edges and textures that the human visual system can readily process. Given the statistical similarities between images and sounds, we ask whether it is possible automatically generate examples that lie at the intersection of both modalities. We create images that sound (Fig. 1). 2D matrices that look semantically meaningful when viewed images, but that also sound meaningful when played as a spectrogram. This generative modeling problem is challenging, because it requires modeling a distribution that is induced by two very different data sources, and no relevant paired data is available.
We are motivated by the "spectrogram art" that has been made by a variety of artists [10], most famously by musicians Aphex Twin [2], Venetian Snares [113] and Nine Inch Nails [85]. These artists manipulate their songs display desired image when they are visualized as spectrograms, such as by showing the artist's s face or album art. In current practice, there is steep trade-off between the quality of the image and the sound, since is difficult to simultaneously control the interpretation of a signal in both modalities. As a result, existing artwork often comes across to the listener as dissonant or as random noise, rather than as natural sounds. By contrast, we aim to generate signals that natural as possible in both modalities, such towers that simultaneously sound like ringing bells or images of tigers that make a roaring sound (Fig. 1).
In this work, we pose this problem as multimodal compositional generation task and propose simple, zero-shot method that composes off-the-sheit models from different modalities. Inspired by prior work on compositionality in diffusion models [72, 29, 42, 41], we denoise using both a noise estimate from the spectrogram model and a noise estimate from the image model. This is possible because these two models perform diffusion in the same latent space. The result a sample that is simultaneously likely under the (text-conditional) distribution of spectrograms and images. The spectrograms are then converted to waveforms using pretrained vocoder. In addition, we show that these black-and-whitt images may be colorized, resulting in color images whose grayscale versions can be played as spectrograms.
sstttt--ppectogrram and text-to-image diffusion
Surprisingly, we find that off-the-shelf diffusion models trained on different modalities can be composed together to obtain samples that function as both an image and a sound. Often these examples reuse visual elements in unexpected ways (e.g., in Fig. 1, a line is both the onset of a bell chime and the contour of a bell tower). We provide qualitative results, as well as quantitative comparisons and human study results against baselines, indicating that our method produces spectrograms that better align with both the audio and image prompts. Our contributions are summarized as follows:
We propose images that sound, a type of multimodal art that can be both understood as an image or played as sound.
We show that we can compose pretrained diffusion models from different modalities in a zero-shot fashion to produce examples the intersection of image and spectrogram distributions.
We encourage the reader to listen popular examples of spectrogram art [10].

---

We propose alternative methods for generating images that sound, one based on score distillation sampling [11, 93] and another based on simply subtracting an image from spectrogram. We find through qualitative and quantitative experiments that our method outperforms baseline approaches and generates high-quality samples.
Related Work
Diffusion models [102, 54, 106, 27, 104] are a class of generative models that learn to reverse forward process that iteratively corrupts data. Typically, this forward process adds Gaussian noise and the reverse process learns to denoise the data by predicting the added noise. Diffusion models have variety of applications, including text-conditioned image generation [27, 96, 84, 26, 98], video generation [53, 56, 101, 6, 45, 115], image and video editing [94, 97, 79, 49, 9, 31, 40J, audio generation [118, 70, 71, 32, 43, 76]. 3D generation [74, 57, 12, 73, 8, 38], and camera pose estimation [121]. J. In this work, we use Stable Diffusion [96]. a latent diffusion model trained for text-conditioned image generation, as well as Auffusion [118], a text-conditioned audio generation model trained to produce log-mel spectrograms. Auffusion is finetuned from Stable Diffusion, similar to Riffusion [34], and as result, the two methods share a latent space. This is crucial for our technique, which jointly diffuses these shared latents.
Diffusion models.
Compositional generation. One property of diffusion models is that they admit straightforward technique to compose concepts by summing noise estimates. This may be understood by viewing noise estimates gradients of conditional data distribution [105, 106], and the sum of these gradients as pointing in the direction that maximizes multiple conditional likelihoods. This approach has been applied enable compositions of text prompts globally [72], spatially [7, 29], transformations of images [42], and image components [41]. We go beyond these works by showing that diffusion models from two different modalities can successfully be composed together.
Audio-visual learning. A variety of works have learned cross-modal associations between vision and sound. Some approaches establish semantic correspondence, i.e., which sounds and visuals are commonly associated with one another [3, 108). Previous work has used this cue to learn cross- modal representations [5, 83, 80, 46, 44, 69, 68] and mndiolvisual sound localization [4, 58, 81, 59, 100, 91, 75]. Some researchers focus on the temporal correspondence between audio and visual streams [64, 87, 33, 16, 107, 62] to study source separation [122, 1, 37], Foley sound generation [61, 28, 117, 78], and action recognition [39, 60, 86). Others also explore the spatial correspondence between them [21, 35, 120, 19, 22, 77], including spatial sound generation [36, 82, 14, 67] and audio-visual acoustic learning [13, 103, 24, 18, 20]. Differing from the works above, our focus is to explore the intersection of the distributions between spectrograms and images, where we create spectrograms that can be understood as visual images and can also be played as sounds.
Audio steganography. Audio steganography is the practice of concealing information within an audio signal. Artists have explored it for creative expression [111, 1 110]. Aphex Twin embedded a visual of his face in the audio waveform of the track "Formula" [2]. Noam Oxman creates animal portraits made of musical notations [88]. Other work has proposed deep learning methods for steganography, such as hiding video content inside audio files with invertible generative models [1 [119], hiding audio data inside an identity image [123], and audio watermarking [15, 89, 99]. Our approach can be viewed steganography method that hides an image within audio track, and only
revealed when the track is converted to a spectrogram.
3 Method
Our goal is to generate spectrograms that simultaneously represent both a sound and an image, each of which is specified by a text prompt. When the spectrogram is converted into a waveform, the sound matches the audio prompt, while when it is visually inspected, it should take the appearance of a given visual prompt (Fig. 1). To do this, we sample from the joint distribution of images and spectrograms, using fff-the-shelf diffusion models trained each modality independently.
3

---

"castle e with
bel1 towers, gräyscale"
Latent Image Diffusion
A
Weighted Avg.
shared latent space
2
Latent Audio Diffusion
(()
"bell xinging"
iterative denoising
Latent Decoder
Vocoder
Z
waveform
log-mel spectrogram
Figure 3: Composing audio and visual diffusion models. We generate the visual spectrogram that be visualized as an image or played as sound. Given noisy latent z,, we apply visual and audio diffusion models, each guided text prompt, compute noise estimates ( and ( respectively. We obtain the multimodal noise estimate g( by a weighted average, then use it as part of the iterative denoising process. Finally, decode the clean latent to spectrogram and convert into wavcform using pretrained vocoder (or by Griffin-Lim [47]).
3.1 Preliminaries
Diffusion models. Diffusion models [54, 106] iteratively denoise standard Gaussian noise, Xy N( I), to generate clean samples, Xy, from some learned data distribution. timestep in the reverse diffusion process, the noise predictor, сØ, takes the intermediate noisy sample, X,, and the condition y, such as a text prompt embedding, to estimate the noise < y. Following DDIM [104], obtain the next, less noisy, sample X;-1 at the previous timestep via:
X, = Va
Xx
ai éa (xei U,
v 11 a:-1 of (x + o,e,
(1)
02
where e; is independent Gaussian noise, a is a predefined coefficient, and ar controls the randomness level which set to 0 for deterministic sampling. We may also optionally apply classifier-free guidance (CFG) [55] by modifying the noise estimate as:
(x;;; y, L) 6 xx; 0,() (co(xe;B,)) ep (x:; 2,()).
where y denotes the strength of the conditional guidance and is the unconditional embedding of the empty string. This often results in much higher- nigher-quality samples.
Latent diffusion. Latent Diffusion Models (LDMs) [96] perform the diffusion process in latent space rather than in pixel space. A pretrained encoder and decoder pair, and D, translates between pixel space and latent space. The latent space typically much more compact and information-dense. which makes diffusion in this space more efficient. We use pretrained LDMs in our approach, due to the availability of audio and visual models with the same latent space.
3.2 Multimodal Denoising
Our goal is to generate an example x E Rmxn that would be likely to appear under both visual and audio distributions, Pa(.) and (.). We formulate this sampling from a product of expert models [52]: Pan (x) x P. (x) (x). We follow recent work on the compositional generation that
Reeent work has called this conjunction [30]. since conceptually the samples are roughly from the intersection of both distributions.

---

samples from this distribution using the score functions from pretrained diffusion models [30]. In contrast to these approaches, however, our two models are trained on two different modalities.
We create our spectrograms using two pretrained latent diffusion models. One trained to generate images, "'."( and the other generate spectrograms, (,,a(t>"))) both operating in the same latent space. We show an overview of our method in Fig. 3. Given a noisy latent, Zę, and text prompts Ye and ye corresponding to the desired image and spectrogram prompt respectively, we compute two CFG noise estimates (Eq. (2)):
e Edi. [Z 2,t) + Tou (Eg,o yo. t) Cs Øtt))
(3)
(A) = cs.a (2;; 23, () + T/a (Eo,a ya: - 60.a (2e; 23,)) ())
(4)
where "/ and To are the corresponding visual and audio guidance scales. We then combine the noise estimates from both modalities by applying weighted averaging, producing a multimodal noise estimate that steers the denoising process toward sample that is likely under the distribution of both images and spectrograms:
(5)
where X and ( are the weights of the audio and visual noise estimates at timestep t respectively.
With this new noise estimate 2( we perform a step of DDIM (Eq. (1)) to obtain a less noisy latent, Ze-1. Repeating this process we obtain the clean latent Zo. which is then decoded using the decoder 'D to obtain the spectrogram & = D(zo). This spectrogram can further be converted to a waveform using pretrained vocoder or colorized to RGB image whose grayscale version is the spectrogram.
Warm-starting. We find useful to warm-start the denoising process. In Sec, 4.5, we experiment with warm-starting using only the spectrogram noise estimates only the image noise estimates.
This can be represented by using w and w!! as the relative weight on the audio and the visual noise estimates respectively. We let
(2)
(1) We
1
Wa
X
(6)
(t)
(1)
(+)
(t)
Wa
Wid
we
with wa H(.T 1) and wh H(,T 1) being Heaviside step functions, and la and by indicating the proportion of the reverse process that has audio or visual denoising respectively. When to 1.0 and l, = 1.0, we warm-start with only image denoising, and vice-versa. The above ensures that the weights ( and ((()) sum to one, and are equally weighted after varm-starting.
(8)
Colorization. After we generate a spectrogram, 8, we can optionally colorize it to create a more visually appealing result. Since our spectrograms fall outside the distribution of pre-trained coloriza- tion models, we use Factorized Diffusion [41] to colorize, which samples a diffusion model while projecting the noisy intermediate images such that they equal & when turned into grayscale. In doing so, the denoising process synthesizes only the "color component" of the sampled image, while the "grayscale component" is constrained to equal the generated spectrogram. Note that this method is similar to prior work [63, 23, 106, 1 114]. We choose this particular method due to its simplicity.
4 Experiments
We evaluate our methods using quantitative metries and human studies. We also present qualitative comparisons and an analysis of our method, and why it works.
4.1 Implementation Details
Models. We select a pair of off-the-shelf latent diffusion image and audio models that share the same latent space, encoder, and decoder. For the image model, we use Stable Diffusion v1.5² [96]. For the audio model, we use Auffusion' [1]8] which finetunes Stable Diffusion vl.5 on log-mel spectrograms. To synthesize audio from the log- spectrograms, we consider two options: following [118] and using off- the-shelf HiFi-GAN [66] vocoder, the Griffin-Lim algorithm [47, 92]. We use HiFi-GAN for our main experiments. In Sec. 4.5, we evaluate the choice of vocoder and verify that resultant waveforms do indeed encode to visually interpretable spectrogram.
Steble Diffusion. v1-5 hugging face model card
Auffusion hugging model card
5

---

Table 1: Quantitative evaluation images that sound. We report CLIP, CLAP, FID, and FAD metrics, along with 95% confidence intervals shown gray. The best results are highlighted bold.
Method
Modality
CLIP (
CLAP (%)
FID.L.
FAD
Stable Diffusion [96]
V
34,5 (b0.1)
2.2 (2:022
41.74
Auffusion [118)
A
22.5 (20.1)
48.3 (20.6)
290.29
Imprint
A & V
27.2 (40.2)
32.3 (+:.0)
244.84
29.42
SDS
& V
25.4 (+-022
23.4 (+1.4)
273.03
32.57
Ours
&&V
28.2 (1:0.1)
33.5 (4:0.9
226.46
19.21
Hyperparameters. We begin the reverse process with random latent noise 2y E R4x32x122 the same shape that Auffusion was trained on. Despite the image model not being trained on this specific size, we found that it nevertheless produces visually appealing results. We set the classifier guidance scales T/ and Ta to be between 7.5 and 10 and denoise the latents for 100 inference steps with warm-start parameters of La2 1.0, Lu 0.9 to preserve audio priors. We decode the latent variables into images of dimension 3 x 256 x 1024. By averaging across each channel, we obtain spectrograms corresponding to 10 seconds of audio. We re-normalize the spectrograms for visualization.
Baselines. As there is no previous work in this domain, we propose two baseline approaches. The first, inspired by Diffusion Illusions of Burgert el al. [11], uses multimodal score distillation sampling (SDS). We optimize single-channel image x = g(0), where y is an implicit function parameterized by 0, using two SDS losses: one from the image diffusion model @u and the other from the audio diffusion model Φ.. This results in a gradient of:
LSDS (x = s(0)) = deda Be, (w).(1) (000)
0x 00
Ee, wa(l) (c!))
0x ae
-
where Aaaa is the weight of the image SDS gradient and is the noise added to the image or latents. We implement this with pixel-based diffusion model DeepFloyd IF [26]. as we find performs better than Stable Diffusion with the SDS loss, and Auffusion [118]. This model thus does not require shared latent space between vision and audio. We refer to this baseline as the SDS.
The second baseline involves taking existing images and subtracting them from existing spectrograms, multiplied by some scaling factor, inspired by [25]. This works when the spectrograms have high power, as the subtraction does not significantly affect the audio but still imprints an image into the spectrogram. We obtain spectrograms and images for this baseline via Auffusion and Stable Diffusion. This approach, which we call imprint, is simple but can be surprisingly effective. All methods use the same vocoder and post-processing for fairness. Please see Appendix A.3 for more details.
4.2 Quantitative Evaluation
We start by quantitatively evaluating the quality of our generated images that sound, examining how well the generated examples match the provided text prompts for each modality.
Experimental setup. Following the evaluation of Visual Anagrams [42]. we create two sets of text prompt pairs. We randomly select 5 discrete (onset-based) and 5 continuous sound category names from VGGSound Common [17] audio prompts. We randomly chose 5 objects and 5 scene classes for image prompts, formatted as "a painting of [class], grayscale", This yields a total of 100 prompt pairs. We report Stable Diffusion and Auffusion performances single-modality
Table 2: Human study, We show win-rates of our spectrograms against those generated by the SDS and imprint baselines. The first row indicates which audiovisual prompt pair is evaluated, formatted as [audio prompt] / [visual prompt], ), with the last column being the average of all seven prompt pairs. Note that 50% win-rate is chance performance, and such method outperforms the baselines in the vast majority of cases. Also note that this a bast-case evaluation please see Sec. 4.3 for details. All results reported are win-rate against the baseline with a 95% confidence interval in gray (N = 100).
Baseline Metrie
bell/castle bark/dog
birds/garden meow/kitee racecarrracccar tigertiger
train/train
Average
andio quality (44.0)
69.4 (24.2)
95.9 (:0.5)
75.5 (4:.7)
88.8 (22.0)
70.4
88.8 (=2.0)
77.4 (20.7)
SDS
visual quality
(44.7)
(1.0
:8.0:40.4)
32.7 (4-44)
69 (54.2)
(8-4.)
94.9 (s1.0)
67.8 (50.8)
alignment
58.2
63.3 (44.0)
:3.9:
62.2 (b:47))
82.7 (H2 8)
$9.2 (2:
91.8 (11.0)
73.0 (20.5)
audio quality
ca3.)
73.7 t::.0
53.7:a5.0
54.7 (85.0)
86.3 (a: 4
85.3 (8:05))
85.3
74.4 (5077)
Imprint
visual quality
92.6
86.3 (42.4)
66.3 (44..))
68.4 (4:4.3)
66.3 (2445)
77.9 (63.5)
56.8 (24.9)
73.5 (40.5)
alignment
(42.1)
(7.4) (22.2)
60.0 (64..5)
65.3 (4:4.0)
86.3 (52.4)
80.0 (2:.2)
85,3 (=2.6)
78.9 (4006)

---

Ours
Imprint baseline
SDS
painting Lowezs, grayscale" "bell ringing"
"a painting of furry kittens, grayscale" "a kitten mecwing for attention"
painLing of auto racing game, grayscale" "a race car passing by and d' sappear ing"
Figure 4: Qualitative comparison. We show our qualitative results along with the imprint and SDS baselines given visual (first) and audio (sccond) prompts. Please zoom in for better viewing.
benchmarks to establish upper and lower bounds. We generate 10 samples for each prompt pair, except for the SDS baseline, for which we generate 4 samples due its slower speed.
Evaluation metric. Following [50], we use the CLIP [95] score to measure the alignment between spectrograms and image text prompts, and analogously we use the CLAP [116] score to evaluate the aligmment of audio with audio text prompts. An ideal method should excel at both simultaneously. We also report FID [51] and FAD [65] evaluate the quality of generated examples where we use the results of Stable diffusion and Auffusion reference sets respectively.
Results. We show our quantitative results in Tab. Our method outperforms baselines across all metrics and performs comparably to single-modality models, which serve as rough upper bounds for each modality, This demonstrates our approach's 's ability to generate meaningful images that sound, sampling from the intersection of natural image and spectrogram distributions. Stable Diffusion achieves low CLAP score, indicating how poorly randomly sampled natural image acts as a spectrogram. We observe that the SDS baseline often fails to optimize both modalities together. In contrast, our method achieves a higher success rate and generates more diverse results. Our method is significantly faster, generating one sample in 10 seconds compared the SDS baseline's 2-hour optimization time using NVIDIA L40s. The imprint baseline imprints the image onto the spectrogram, potentially degrading the sound pattern and leading lower CLAP score. Note that FID and FAD are distribution- based metrics, and as our task focuses on generating examples that lie small subset of the natural image and spectrogram distribution, higher FID scores, in general, are expected.
4.3 Human Studies
Experimental setup. We also perform two-alternative forced choice (2AFC) studies to evaluate our results. We construct seven paired text prompts by hand, ensuring semantic correlations between image and audio prompts, such as pairing visual of dogs with the sound of dogs barking. Using these prompts, we generate samples using our method, the SDS baseline, and the imprint baseline, and hand-pick the best examples for evaluation. This best-case evaluation is useful as participants from MTurr are not expected to have prior knowledge about spectrograms, let alone domain expertise. Moreover, this evaluation matches the intended use case of our method, in which user repeatedly queries the model for a result that they prefer based on artistic merit and quality. Participants, are presented with one sample from method, and a corresponding sample from a baseline, and are

---

"a pond full of vater lilies, grayscale, 1ithograph sty_e"
painting corgis, grayscale, black background"
Spectrogram
"dog barking"
"frog croaking"
Waveforn
WI
colorful corgia"
colorful
Color image
"a painting of auto recing geme, greyscale"
Ka painti king of blooning garden, grayecale"
Spectrogram
passing stsappearing"
"birds singing
Waveform
"a coloriul photo of auto racing game"
"a colorful photo Blooning garden"
Color image
Time
Figure 5: Qualitative examples with colorization results. We present 4 examples alongside their image
prompts, audio prompts, and colorization prompts. Please refer our website for video results.
asked choose (1) the sample that looks most like the image prompt, (2) the sample that sounds most like the audio prompt, and (3) the sample in which the visual structure of spectrogram best aligns time. The first two questions
with that of image
as perceptual versions of CLIP and CLAP scores. The third question is designed to evaluate how well the visual structure of audio matches with the images the spectrogram is played. Please Appendix A.3 for further details and discussion.
Results. Win-rates between our method and baselines are presented in Tab. 2, broken down by prompt pair. We also include averaged win-rates over all prompt pairs in the final column. As can be seen, our method outperforms the two baselines in most cases. Human evaluators consistently rate our spectrograms as being higher in audio and visual quality, and as being better "visually-synced" on average method is 2-3 times as likely to be chosen the better sample than baselines.
4.4 Qualitative Results
Results. We present qualitative results from our method well as baselines in Fig. with additional results from our method in Figs. 5, 8 and 9 in the appendix. Audio of all results can be found on our website. As can seen (and heard) our approach generates more visually appealing samples with better sound quality than compared to the baselines. The SDS baseline often focuses on one modality, to the detriment of the other, and in general, generates audio lower quality. Morcover, the method suffers from the characteristic oversaturation of SDS-based results. The performance of the imprint baseline highly dependent on the independently generated spectrogram and image and tends to fail when the two misaligned, in the castle example, or when the spectrogram has low energy, as the subtracted image hard to see in already low energy regions of the spectrogram, such as with the kitten example. Interestingly, we found that method often combines visual and acoustic elements. For example, the onsets of the bells ringing in Fig. coincide with the towers of the castle, and the spectrogram patterns of meowing hidden as stripes and edges the kittens.
We show additional hand-picked results from our approach in Fig. with colorization results, in which we see more examples of our method blending acoustic and visual elements, such as the

---

Table Ablations. We conduct the ablation study of the guidance scale (left) and the warm-starting (right). To
evaluate overall performance, we normalize each score by boundaries in Tab. and then sum them.
Method
Variation
CLIP (
CLAP (56)
Mothod
b
la
CLIP (5t
CLAP (S)
Overall
7ex Te 5.0
28.0
29.3
1.0
1.0
28.7
30.8
1.14
Ours
Tes No 7.5
28.2
31.9
Ours
0.9
1.0
29.0
28.2
27.4
33.5
1.15
1.08
To:
28.2
33.5
35,9
1.14
ORIGINAL
HIFL-GAN
GRIFFIN-LIM
Figure 6: Cycle consistency check the vocoders. We show the original log mel-spectrogram decoded from latents and log mel-spectrograms obtained from waveforms synthesized by HiFi-GAN Griffin-Lim.
water lilies corresponding to the frogs croaking, the corgis corresponding the dogs barking, and the flowers corresponding to the birds chirping. Please see more results Fig. 8 of Appendix A.2.
Multimodal compositionality. Prior work [72, 29, 7, 42, 41] shows that diffusion models may be "composed" to generate samples that are likely under two or more different probability distributions. Our method can be seen as extending this idea of compositionality to multiple modalities. On the face of it, the distribution of spectrograms and the distribution of natural images would seem to be completely disjoint. However, as our results show, perhaps surprisingly, some overlap exists. We believe that this is possible for two reasons. First, spectrograms and images are both fairly flexible, allowing for significant amounts of perturbation or changes in style before becoming unrecognizable. And second, images and spectrograms share certain low-level characteristics, such edges, curves, and comers, indicating a certain amount of similarity. Please see Appendix A. for more analysis.
However, we find that not all compositions can be successful as shown in Fig. 9 of Appendix A.2. Moreover, careful selection of prompts crucial to creating good results. For example, incorporating terms such as "1 "1ithoge thograph style" or "black background" encourages the visual model to create areas of silence, which results better quality, as shown in Fig. 5.
4.5 Ablations
Vocoder. To extract waveforms from our generated spectrograms we use HiFi-GAN [66], a neural vocoder. Given that our spectrograms incredibly out of distribution, concern with this setup is that the vocoder will ignore spectrograms and generate waveforms that do not match the inputs. To ameliorate this concern, conduct cycle consistency check by re-encoding the neural vocoder's predicted waveform back into a spectrogram by performing an STFT. As can be seen in Fig. 6, the recomputed spectrograms
very similar to the original spectrogram, with only slightly less sharpness and some blurred textures". This suggests we truly create visual spectrograms that look like images. We also experiment with using Griffin-Lim [92] as a vocoder, with similar results HiFi-GAN as shown in Fig. 6. We opt to use HiFi-GAN as our default vocoder as it outperforms Griffin-Lim in audio quality, with Griffin-Lim attaining CLAP score of 0.302, compared to 0.335 obtained from HiFi-GAN. Please see more results in Fig. of Appendix A.2.
Warm-starting. We also conduct an ablation study on our warm-starting strategy by varying which modality is warm-started and by how many steps. Results are presented in Tab. 3, where i, and t, are defined Sec. 3.2. We find that warm-starting the denoising process with either image or audio diffusion yields higher scores in the corresponding modality, as that modality effectively gets free reign to set the high-level features of the final result. We find that allowing the audio diffusion model to denoise alone for the first 10% of the timesteps results in an attractive balance between CLIP and CLAP scores. Therefore, we adopt l, = 0.9 and la 1.0 for our main experiments.
We note that perfect cycle consistency is not generally possible since vocoders fundamentally lossy.

---

Guidance scale. We also explore different guidance scales "/ and for our method. We present results Tab. 3. We find that higher guidance scales generally yield better results on both modalities. We hypothesize that the higher guidance scales more strongly encourage the sample to come from the "intersection" of the conditional spectrogram and conditional image distributions, resulting in better alignment with both text prompts.
5 Discussion and Conclusion
In this work, we demonstrate that, perhaps surprisingly, there is a non-trivial overlap between the distribution of natural images and the distribution of natural spectrograms. We show this by sampling from the intersection of these two distributions, resulting in spectrograms that look like real images but also sound like real sounds. The method we proposed is simple and zero-shot, and leverages the compositional nature of diffusion with cross-mode models. We see our work as advancing multimodal compositional generation and opening up new possibilities for multimodal art.
Limitations. One limitation of our method is that it cannot generate examples that have both high-fideling audio and image. We show failure cases, which occur for many prompts, in Fig. 9. Some of these failures may be due to the strict constraints of the problem, since realistic examples may not always exist at the intersection of both distributions. Our method is also limited by the quality of the audio diffusion model, whose performance lags behind that of visual models.
Potential negative societal impacts. The image and audio generation models that our method leverages are becoming progressively more powerful, and care must be taken in their deployment. Moreover, our method could potentially used for steganography, secretly embedding images within audio. This capability may be used for deception, and we believe it deserves further consideration.
Acknowledgements. We thank Ang Cao, Linyi Jin, Jeongsoo Park, Chris Donahue, Alexei Efros, Prem Seetharaman, Justin Salamon, Julie Zhu, and John Granzow for their helpful discussions. This project supported by the Sony Research Award and Cisco Systems. Daniel is supported by the National Science Foundation Graduate Research Fellowship under Grant No. 1841052.
References
[1] T. Afouras, Owens, S. Chung, and Zisserman. Self-supervised learning of audio-visua objects from video. European Conference on Computer Vision (ECCV). 2020.
[2] Aphex Twin. Formula, 1994. audio track. 2,
[3] Arandjelovic and Zisserman. Look, listen and learn. In Proceedings of the IEFE international conference on computer vision, pages 609-617, 2017.
[4] Arandjelovic and A. Zisserman. Objects that sound. In Proceedings of the European conference on computer vision (ECCV), pages 435 451, 2018.
[5] Asano, Patrick, Rupprecht, and Vedaldi. Labelling unlahelled videos from scratch with multi-modal self-supervision. Advances in Neural Information Processing Systems, 33:4660-4671, 2020.
3
[6] O. Bar- Tal, H. Chefer, O. Tov, C. Herrmann, R. Paiss, S. Zada, A. Ephrat, J. Hur, Y. T. Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12945 2024.
[7] O. Bar-Tal, I. Yariv, Y. Lipman, and T. Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. arXiv preprint arXiv:2302.081 2023. 3.9
[8] R. Bensadoun, T. Monnier, Y. Kleiman, F. Kokkinos. Y. Siddiqui, M. Kariya, O. Harosh, R. Shapovalov, Meta gen. arXiv preprint arXiv:2407. 02599, 2024.7
Graham, Garreau,
[9] T. Brooks, Holynski, and A. Efros. Instructpix2pix: Learning follow image editing instructions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392- 18402, 2023. 3
[10) B. Buckle. Spectrogram art: A short history of musicians hiding visuals inside their tracks. Available from: https: : //mi xmag. net httt///eatue////eeeeeeeeeeeeeeee spect rom art-mun -musiiccc cesssi-appe aphea-phe-eeee twin, 2022. Mix- mag article. 2
10

---

[11] R. Burgert, X. Li, A. Leite, K. Ranasinghe, and M. S. Ryoo. Diffusion illusions: Hiding images in plain
sight. arXiv preprini arXiv:2312.05817, 2023. 3, 6, 18
[12] A. Cao, J. Johnson, A. Vedaldi, and D. Novotny. Lightplane: Highly-scalable components for neural 3d fields. arXiv preprint arXiv:2404. 0224404499766. 19760, 2024.
[13] C. Chen, R. Gao, P. Calamia, and K. Grauman. Visual acoustic matching. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 3
[14] C. Chen, A. Richard, R. Shapovalov, V. K. Ithapu, N. Neverova, K. Grauman, and A. Vedaldi. Novel- view acoustic synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognittinn, pages 6409-6419, 2023. 3
[15] G. Chen, Y. Wu, S. Liu, T. Liu. X. Du, and E. Wei. Wavmark: Watermarking for audio generation. arXiv preprint Arvvv:3308. / 12770, 2023, 3
[16] H. Chen, W. Xie, T. Afouras, A. Nagrani, A. Vedaldi, and A. Zisserman. Audio-visual synchronisation the wild. arXiv preprint arXiv:21 2021.
[17) H. Chen, Xie, A. Vedaldi, and A. Zisserman. Vggsound: large-scal audio-visual dataset. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721-725. IEEE, 2020. 6
[18] M. Chen, K. Su, Shlizerman. Be everywhere-hear everything (bee): Audio scene reconstruction by sparse audio-visum samples. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023. 3
[19] Chen, D. Fouhey, and A. Owens. Sound localization by self-supervised time delay estimation. European Conference on Computer Vision (ECCV), 2022. 3
[20] Chen, 1. D. Gebru, C. Richardt, A. Kumar, Laney, A. Owens, and Richard. Real acoustic fields: An audio-visual room acoustics dataset and benchmark. In The IEFF/ / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2024. 3
[21] Chen, X. Hu, and A. Owens. Structure from silence: Learning scene structure from ambient sound. In 5th Annual Conference on Robot Learning, 2021. 3
[22] Chen, S. Qian, and A. Owens. Sound localization from motion: Jointly learning sound direction and camera rotation. International Conference on Computer Vision (ICCV), 2023.
[23] J. Choi, S. Kim, Y. Jeong, Y. Gwon, and S. Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108. 02238. 2021. 5. 17
[24] S. Chowdhury. S. Ghosh, S. Dasgupta, A. Ratnarajah, U. Tyagi, and D. Manocha. Adverb: Visually guided audio dereverheration. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7884-7896, 2023. 3
[25] Classical Music Reimagined, Fun with spectrograms! how to make an image using sound and music. Available from: https: //www. http://wwwwwttttto youtube. com/ ca ID Y 2017. Youtube video.
[26] DeepFloyd Lab StabilityAI. DeepFloyd IF: novel state-of-the-ee open-source text-to-image model with a high degree of photorealism and language understanding. https: : /wwww/wwww//..o////./ wwwpww.eepel....cc aii deepfloyd- 1,, 2023. 3, 6, 17
[27] Dhariwal and Nichol. Diffusion models heat gans on image synthesis. Advances neural information processing systems, 04-8780-8794, 2021.
[28] Y. Du, Z. Chen, J. Salamon, B. Russell, and A. Owens. Conditional generation of audio from video via foley analogies. Computer Vision and Pottern Recognition (CVPR), 2023. 3
[29] Y. Du, C. Durkan, R. Strudel, J. B. Tenenbaum, S. Dieleman, R. Fergus. J. Sohl- -il--Dikstein. A. Doucet, and W. S. Grathwohl. Reduce, reuse, recycle: Compositional generation with eaagyybbased diffusion models and meme. In International Conference on Machine Learning, pages 8489 8510. PMLR, 2023. 2, 3, 9
[30] Y. Du, S. Li, and I. Mordatch. Compositional visual generation with energy based models. Advances in Neural Information Processing Systems, 33:6637 6647, 2020. 5,5
[31] D. Epstein, A. Jabri, B. Poole, A. Efros, and A. Holynski. Diffusion self-guidance for controllable image generation. Advances in Neural Information Processing Systems, 36:16222-166 16239, 2023. 3
11

---

[32] Z. Evans, C. Carr. J. Taylor, S. H. Hawley, and J. Pons. Fast tining-conditioned latent audio diffusion. arXiv preprint mrXiv:2402.04825. 2024.
[33] C. Feng, Z. Chen, and A. Owens. Self-supervised video forensics by audio-visual anomaly detection. Computer Vision and Pattern Recognition (CVPR), 2023. 3
[34] S. Forsgren and H. Martiros, Riffusion Stable diffusion for real-time music generation, 2022. 2,3
[35] R. Gao, C. Chen, Z. Al-Halah, C. Schissler. and K. Grauman. Visualechoes: Spatial visual representation learning through echolocation. In European Conference on Computer Vision (ECCV), 2020. 3
[36] R. Gao and K. Grauman. 2.5d visual sound. In Conference on Computer Vision and Pottern Recognition (CVPR), 2019. 3
[37] R. Gao and K. Grauman. Visualvoice: Audio-visual speech separation with cross-modal consistency. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 3
[38] R. Gao, A. Holynski, P. Henzler, A. Brussee, R.
P. Srinivasan, J. T. Barron, and B. Poole. Cat3d: Create anything in 3d with multi-view diffusion models. arXiv preprint arXiv:2405.103/4 2024. 3
[39] R. Gao, T.-H. Oh, K. Grauman, and L. Torresani. Listen look: Action recognition by previewing audio. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10457-10467, 2020. 3
[40] D. Geng and A. Owens. Motion guidance: Diffusion-basse based image editing with differentiable motion estimators, arXiv preprint arXiv:240I. 18085, 2024. 3
[41] D. Geng. I. Park, and A. Owens. Factorized diffusion: Perceptual illusions by noise decomposition. arXiv:2404. 11.44044116555 16655, April 2024. 2,3, 17
[42] D. Geng, L. Park, and A. Owens. Visual anagrams: Generating multi-view optical illusions with diffusion models. In CVPR, 2024. 2, 6,9
[43] D. Ghosal, N. Majumder, A. Mehrish, and S. Poria. Text-to-audio generation using instruction tuned Ilm and latent diffusion model. arXiv preprint arXiv:2304.1373 2023. 3
[44] R. Girdhar, A. El-Nouby. Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 51800151190 2023. 3
[45] R. Girdhar, M. Singh, A. Brown, Q. Duval, S. Azadi, S. S. Rambhatla, A. Shah, X. Yin, D. Parikh, and
I. Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709, 2023. 3
[46] Y. Gong. A. Rouditchenko, A. H. Liu, D. Harwath, L. KKrlinsky, H. Kuehne, and J. Glass. Contrastive mudio-visual masked autoencoder. arXiv preprint arXiv:2210.07839, 2022. 3
[47] D. Griffin and J. Lim. Signal estimation from modified short-time fourier transform. IEEE Transactions on acoustics, speech, and signal processing, 32(2):23--243, 1984. 4,5
[48] G. Gwardys and D. Grzywezak. Deep image features in music information retrieval. International Journal of Electronics and Telecommunications, 60:321-326, 2014.
[49] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch. and D. Cohen-Or. Prompt-to- torommpt image editing with cross attention control. arXiv preprint 2208v:208..0662
2022. 3
[50] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.0888
2021. 7
[51] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.7 7
[52] G. F. Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 2002. 4
[53] J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022. 3
12

---

[54] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020. 3.4
[55] J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint axXivv2207.125596 2022. 4
[56] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet. Vidco diffusion models. Advances in Neural Information Processing Systems, 35:8633-866 3-466,, 2022. 3
[57] L. Höllein, A. Cao, A. Owens, J. Johnson, and M. Nießner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 7909-7920, Octoher 2023. 3
[58] X. Hu, Z. Chen, and A. Owens. Mix and localize: Localizing sound sources in mixtures. Computer Vision and Pattern Recognition (CVPR), 2022. 3
[59] C. Huang. Y. Tian, A. Kumar, and C. Xu. Egocentric qudio-visual object localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22910-22921, 2023.
[60] J. Huh, J. Chalk, E.. Kazakos, D. Damen, and A. Zisserman. Epic-sounds: A large-scale dataset of actions that sound. ICASSP 2023-2023 IEFF International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 1-5. JEEE, 2023.
[61] lashin and E. Rahtu, Taming visually guided sound generation, arXiv preprint arXiv:2110008791, 2021. 3
[62] Tashin, Xie, Rahtu, and A. 7isserman. Synchformer: Efficient synchronization from sparse cues. arXiv preprint arXiv:2401.16423 2024. 3
[63] B. Kawar, M. Elad, S. Ermon, and Song. Denoising diffusion restoration models, Advances in Neural Information Processing Systems, 35:23593- 22606622666222 2022. 5, 17
[64] E. Kidron, Y. Y. Schechner, and M. Elad. Pixels that sound. In 2005 IEEE Computer Society Conference on Computer Vision and Puttern Recognition (CVPR 005, volume I, pages 38-95, IEEE, 2005. 3
[65] K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi. Fr), 'cchet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv: 18t.:08220000000 084666, 2018. 7
[66] J. Kong, J. Kim, and J. Bac. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 13212022-1 2020. 5,
[67] S. Liang, C. Huang, Y. Tian, A. Kumar, and C. Xu. Av-nerf: Learning neural fields for real-world audio-visual scene synthesis. Advances in Neural Information Processing Systems, 36, 2024. 3
[68] Y.-. Lin and G. Bertasius. Siamese vision transformers are scalable audio-visual learners. arXiv preprint arXiv:2403.1 19638893....8,,.. 2024. 2,3
[69] Y.-B. Lin, Y.-L. Sung. J. Lei, M. Bansal, and G. Bertasius. Vision transformers are parameter-efficiee audio-visual learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2299-2309, 2023. 2,3
[70] H. Liu, Z. Chen, Y. Yuan, X. Mei, X. Liu, D. Mandie, W. Wang, and M. D. Plumbley. Audioldm: Text-to-aude generation with latent diffusion models. arXiv preprint arXiv:2301.1250. 2023. 3
[71] H. Liu, Q. Tian, Y. Yuan, X. Liu, X. Mei, Kong, Y. Wang, W. Wang, Y. Wang, and M. D. Plumb- ley. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. arXiv preprint TrXiv:2308.05734 2023. 3
[72] N. Liu, S. Li, Y. Du, A. Torralba, and J. B. Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pages 423-439 Springer. 2022. 2, 3,
[73] R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick. Zero-1-to-3 Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9298-9309, 2023. 3
[74] S. Luo and W. Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2837-2845, 2021. 3
[75] T. Mahmud, Y. Tian, and D. Marculescu. T-vsl: Text-guided visual sound source localization in mixtures. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024. 3
13

---

[76] N. Majumder, C.-Y. Hung, D. Chosal, W.-N. Hsu, R. Mihalcea, and S. Poria. Tango 2: Aligning diffusion- based lext-to-audio generations through direct preference optimization. arXiv preprint arXiv:2404.080 09956. 2024.
[77] Majuumder, Z. Al-Halah, and K. Grauman. Learning spatial features from audio-visual correspondence in egocentric videos. arXiv preprint arXiv:337.004760 2023.
[78] X. Mci, Nagaraja, G. L. Lan, Z. Ni, E. Chang, Y. Shi, and V. Chandra. Folcygen: Visually-guide audio generation. arXiv preprini arXiv:2309. 2023. 3
[79] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108. 2021. 3
[80] H. Mittal, P. Morgado. U. Jain, and A. Gupta. Learning state-aware visual representations from audible interactions, Advances in Neural Information Processing Systems, 01-23755-33777 2022. 3
[81] S. Mo and P. Morgado, Localizing visual sounds the easy way. In European Conference Computer Vision, pages 218-234. Springer, 2022.
[82] P. Morgado, N. Nvasconcelos, T. Langlois, and O. Wang. Self-supervised generation of spatial audio for 360 video. Advances neural information processing systems, 31, 2018. 3
[83] Morgado, N. Vasconcelos, and Misra. Audio-visual instance discrimination with cross- mossl agreement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 2477512286. 2021. 3
[84] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorcalistic image generation and editing with text-guided diffusion models, 2021.
[85] Nine Inch Nails. Year zero, 2007. Music Album. 2
[86] M. A. Nugroho, S. Woo. S. Lee, and C. Kim. Audio-visual glance network for efficient video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10150-10159, 2023.
[87] A. Owens and A. A. Efros, Audio-visual scene analysis with supe-supe multisensory features. In Proceedings of the European conference on computer vision (ECCV), pages 631-648, 2018.
[88] N. Oxman. Sympawnies: animal portraits made of musical notations. Available from: https: //www youtube. com/ @ @Sympawnies is, 2023. Youtube Channel. 3
[89] P. O'Reilly, 7. Jin, J. Su, and B. Pardo. Maskmark: Robust neuralwatermarking for real and synthetic speech. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4650-4654. IEEE, 2024. 3
[90] K. Palanisamy, D. Singhania, and A. Yao. Rethinking models for audio classification. arXiv preprint arXiv:2007.11154, 2020. 2
[91] Park, A. Senocak, and J. S. Chung. Can clip help sound source localization? In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 5711-5720, 2024. 3
[92] Perraudin, Balazs, and L. Spndergaard. fast griffin-lim algorithm. In 2013 IEEE workshop on applications of signal processing to audio and acoustics, pages 1-4. IEEE, 2013. 5,9
[93] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. ICLR, 2023. 3
[94] C. Qi. X. Cun, Y. Zhang, C. Lei, X. Wang, Y. Shan, and Q Chen. Fatezero: Fusing attentions for zero-shot text- bast- video editing. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15932-15942, 2023.
[95] A. Radfordd J. W. Kim, C. Hallacy, A. Ramesh, Goh, Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on muchine learning, pages 8748-8763. PMLR, 2021.
[96] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022. 2, 5,
14

---

[97] C. Saharia. W. Chan, H. Chang, C. Lee. J. Ho. T. Salimans, D. Fleet, and M. Norouzi. Palette: Image-to- image diffussin models. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022. 3
[98] C. Saharia, W. Chan, S. Saxena, I.. Li, J. Whang. E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes. T. Salimans, J. Ho. D. J. Fleet, and M. Norouzi. Photorealistic text-to-image diffusion models with deep language understanding, 2022.
[99] R. San Roman, P. Fernandez, H. Elsahar, A. Défossez, T. Furon, and T. Tran. Proactive detection of voice cloning with localized watermarking. In Forty-first International Conference on Machine Learning. 2024.
3
[100] A. Senocak, H. Ryu, Kim, T.-H. Oh, H. Pfister, and S. Chung, Sound source localization all about cross-modal alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7777-7787, 2023. 3
[101] U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni, et al. Make-a-videor Text-to-video generation without text-video data. arXiv preprint arXiv:200.14799) 2022. 3
[102] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning. volume 2365,2265,222 Lille, France, 07-09 09 Jul 2015. PMLR. 3
Proceedings of Machine Learning Research, pages
[103] A. Somayazulu, Chen, and K. Grauman. Self-supervised visual acoustic matching. Advances in Neural Information Processing Systems, 36, 2024."
[104] J. Song, Meng, and Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020. 3,4
[105] Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances neural information processing systems, 32, 2019. 3
[106] Song. J. Sohl-Dickstein,
Kingma, A. Kumar, Ermon, and Poole. Score- generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020. 3, 3,4,55 5, 17
[107] J. Sun, L. Deng, Afouras, A. Owens, and A. Davis. Eventfulness interactive video alignment, ACM Transactions on Graphics (TOG), 42(4):1-10, 2023.
[108] K. Sung-Bin, A. Senocak, H. Ha, A. Owens, and T.-H. Oh. Sound to visual scene generation by audio-to-visual latent alignment. Computer Vision and Pattern Recognition (CVPR), 2023. 3
[109] M. Tancik, P. Srinivasan, Mildenhall, S. Fridovich- Keiil,, N. Raghavan, U. Singhal, R. Ramamoorthi,
J. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547. 2020. 18
[110] The Beatles. Lucy in the sky with diamonds, 1967. 3
[111] Tool. 10,000 days, 2006. Volcano Entertainment.
[112] Ulyanov. Audio texture synthesis style transfer. https: http://ynnnntnno.t...mmm 1o/ mudio-texture- synthes and---n--- serddstylee
2016. 2
[113] Venetian Snares. Songs about my cats, 2001. Music Album. 2
[114] Y. Wang. J. Yu, and J. Zhang. Zero-shot image restoration using denoising diffusion null-space model. The Eleventh International Conference on Learning Representations, 2023. 17
[115] Z. Wu, Y. Ge, X. Wang. S. W. Lei, Y. Gu, Y. Shi, W. Hsu, Y. Shan, X. Qie, and M. Z. Shou. Tune-a- video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023. 3
[116] Wu, K. Chen, T. Zhang, Hui, T. Berg- KinkpKikkattrikk, and S. Dubuov. Large-scale contrastive language- audio pretraining with feature fusion and keyword-to-caplen augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
7
[117] Z. Xie, S. Yu, Li, Q. He, C. Chen, and Y.GG. Jiang. Sonicvisionlm: Playing sound with vision language models. arXiv preprint arXiv:2401.04394 2024.
15

---

[118] J. Xue, Y. Deng, Y. Gao, and Y. Li. Auffusion: Leveraging the power of diffusion and large language models for text-lo-audio generation. arXiv preprint arXiv:2401.01044 00.4044444,, 2024. 2, 3, 5, 18
[119] H. Yang, H. Ouyang. V. Koltun, and Q. Chen. Hiding video in audio via reversible generative models. In Proceedings the IEEE/CVF international conference computer vision, pages 1100-1109 2019.
[120] K. Yang, B. Russell, and J. Salamon. Telling left from right: Learning spatial correspondence of sight and sound. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 9932 9941, 2020. 3
[121] J. Zhang, A. Lin. M. Kumar, T.-H. Yang. D. Ramanan, and S. Tulsiani. Cameras rays: Pose estimation via ray diffusion. In International Conference on Learning Representations (ICLR). 2024.
[122] H. Zhao, C. Gan, W.-C. Ma, and A. Torralba. The sound of motions. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 1735 1744, 2019.
[123] L. Zhao, H. Li, X. Ning, and X. Jiang. Thinimg: Cross-modal steganography for presenting talking heads images. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5553-5562, 2024.
16

---

A.1 Multimodal Compositionality Analysis
Model capabilities. Through our experiments, we observed that our method generally performs well with "continuous" sound events (e.g., racing cars train whistles) and simple visual prompts. Continuous sounds typically produce spectrograms with high energy distributed across time and frequencies, resulting in "white" spectrograms. This allows our model to effectively reduce sound energy, creating visual patterns that align with the audio.
Simple visual prompts with object scene nouns provide the diffusion models with more flexibility during denoising, enabling sampling from the overlapping distributions of images and spectrograms. However, more complex prompts could push the models into highly constrained distributions where images that sound are less likely to be generated.
Generating discrete sounds (e.g., dog barking or birds chirping) is more challenging due to their sparse energy distribution. In these cases, the models are more constrained, making it difficult to produce visual content with clear edges and structures aligned with sound onsets, leading to less satisfactory results sometimes.
Additionally, we emphasize that some prompt pairs may not have overlapping image and spectrogram distributions, making it impossible to create meaningful examples. For instance, combining the visual prompt starry night with the audio prompt playing guitar leads to conflict, where the image modality tends toward dark image, while the audio suggests a brighter one.
Style words. Visual diffusion models are capa- ble of generating RGB images, but spectrograms are only one channel. We therefore use style words like "grayscale" or "b back background" nudge the image denoising process toward a dis- tribution that matches the spectrograms. As sug- gested by the reviewer, we conducted ablation experiments by removing the grayscale style word. The results are shown in Tab. The model
Table 4: Ablation on style words.
Method
CLIP
CLAP
FID
FAD
w/o "grayseale"
28.1
33.7
237.12
18.09
w/ "grayscale"
28.2
33,5
226.46
19.21
produces similar results, but (as expected) the image quality slightly decreases while the audio quality slightly improves.
A.2 Qualitative results
More qualitative results. We show more qualitative results from our method with different prompts in Fig. 8. Please
website for video results. We also provide random examples with random prompt pairs in Fig. 9 with the last two rows as failure cases. For the failure cases, we can see that they either have good audio quality but lose clear visual patterns (mouutainss/irreworss) have clear visual appearances but noisy audio (dogs/trains).
Vocoder analysis. We show more examples of the vocoder cycle consistency experiment in Fig. As can be seen, the spectrograms from HiFi-GAN are quite similar to the original ones decoded from latents, indicating that our method does not find adversarial examples against the vocoder, but truly does find spectrograms that look like images.
A.3 Implementation Details
Colorization. We use DeepFloyd IF [26] following Factorized Diffusion [41] for colorizing spectrograms. This technique colorizes a grayscale image by using a pretrained diffusion model zero-shot to generate the color component, and is similar prior work such [63, 23, 106, 1 114]. We use it due to its simplicity. We colorize spectrograms of size x 256 x 1024 by directly feeding these into the diffusion model, which we found produced reasonable results despite the fact that the model was not trained for this size. We use prompts of the form "a colorful photo of [image prompt] and denoise for 30 steps with guidance scale of 10. Additionally, the denoising at step 7 of 30 gave better results, which we hypothesize works because it gives the model a stronger prior for what the structure of the image is than starting from pure noise.
found that starting
halttpe://hugiigggeee wtt.a//ggee..e.o///eoeeoooo. DeepFloyd
17

---

SDS baseline. We follow Diffusion Illusions [11] to implement our SDS baseline with an implicit image representation. We use Fourier Features Networks [109] with a learnable MLP to generate images of size 1 256 x 1024. We use stage of DeepFloyd IF-M to perform image score distillation sampling. We randomly make eight overlapping 256 x 256 crops and resize them to 64 64 to compute the averaged image SDS loss with guidance scale of 80. For the audio modality, we use Auffusion [I [1].] As Auffusion is a latent diffusion model, we encode the images into 4 x 32 x 128 latents and perform the audio SDS loss with guidance scale of 10, which we found gave the best performance in the audio-only generation. We set the weight the image SDS loss Aads to 0.4 to ensure balanced optimization for both modalities. We use the AdamW optimizer with a learning rate of 10 and weight decay of 10-3, and optimize the Fourier Feature Network for 40,000 steps. We also apply the waar-start strategy to this method by optimizing the audio SDS loss only for the first 5,000 steps by setting Ands to zero. We note this method does not require a shared latent codebook between image and audio diffusion models.
Imprint baseline. We begin by generating images, Ximg. and spectrograms, Xspee. of size 256 x 1024 using Stable diffusion and Auffusion, respectively, both with guidance scale of 7.5. Next, we the generated images as masks by converting them into inverse grayscale images and scaling them by a factor p. This mask is then applied to the gener- ated spectrogram to obtain the final result, given by Xsps (1 gray(1 11 Xing) )) The hyperparameter controls the strength of energy reduction: larger values yield clearer visual patterns but poorer audio quality, and vice versa. To strike a good balance, we set p 0.5. The imprint baseline takes 10 seconds generate a sample on NVIDIA L40s.
Algorithm 1 Pseudocode in a PyTorch-liin like style the imprint baseline.
get imagea and img stable_diffan diffusion apea auffusior (text_a)
Irom LDMs
requce the energy give image masks mask apea mask spec audio voooder (apec)
(1.0 Lng.mean
Prompt selection. We present the image and audio prompts used for the quantitative evaluation in Tab. 5, We use 10 prompts for each modality, for total of 100 prompt pairs.
Table 5: Text prompts for the quantitative evaluation.
Image prompts
Audio prompts
painting of castles, grayscale painting of dogs, grayscale painting of kittens, grayscale painting tigers, grayscale painting of auto racing game, grayscale painting of mountains, grayscale painting of a garden, grayscale painting of forest, grayscale painting a farm, grayscale painting of a beach, grayscale
dog barking
cal meowing
bird chirping. tweeting
tiger growling church bell ringing
race car, auto racing
train whistling fireworks banging
people cheering playing acoustic guitar
A.4 Human Studies
Participants for the human study were recruited from Amazon Mechanical Turk (MTurkk), and were paid 0.50 USD for a task lasting less than min. We use a total seven prompt pairs and compare them against two baselines: the SDS baseline and the imprin/ baseline. For each method and prompt pair, we hand-selected two high-quality samples for a total of 84 videos. Each video is about 10 seconds long, and includes vertical line moving from left right, indicating the current temporal position in the spectrogram. All participants were shown 14 pairs of videos-seven pairs comparing our method to the SDS baseline, and seven pairs comparing our method to the imprint baseline, all randomly selected and blinded. The participants are then asked to answer three questions:
1. Which video LOOKS most like [visual prompt]
2. Which video SOUNDS most like a [audio prompt ]?
3. In the video, we play the image as a sound, from left to right. which video does the [visual prompt] better align with the [audio prompt] sounds?
18

---

ORIGINAL
HIFI-GAN
GRIFFIN-LIM
Figure 7: More results on the vocoder cycle consistency check. We show the original log mel--ppetroggam decoded from latents and log mel-spectrograms obtained from waveforms synthesized by HiFi-GAM and Griffin- Lim.
The first two questions are designed to evaluate the quality of the audio and image generated by the methods, and their alignment with the respective prompts. The third question seeks to understand how well the visual structure or texture of the generated image and spectrogram align. However, note
able guarantee that the participants had prior experience with spectrograms. To mitigate this to an extent, we include the description as a preamble to the third question. Also, note that abbreviated versions of the audio and visual prompts avoid excessively long questions. We provide the prompt pairs we used for human studies in Tab. 6 for reference, and screenshots of our survey including the title block well as the first video pair and associated questions in Fig. 10.
Table Text prompts the human studies. note that prompts are paired.
Image prompts
Audio prompts
painting of castle towers, grayscale
bell ringing
painting of cute dogs, grayscale
dog barking
painting of blooming garden with many birds, grayscale
birds singing sweetly
painting of furry kittens, grayscale
kitten meowing for attention
painting of auto racing game, grayscale
a race car passing by disappearing
painting of trains, grayscale painting of tigers, grayscale
train whistling
tiger growling
19

---

painting of cute dogs, grayscale" "dog barking"
painting blooming garden many bird, grayscale" "blrds singing sweelly"
"a painting of tigers, grayscale" & "tiger growling"
"a painting of trains, grayscale" "train whistling" Figure 8: More qualitative results. We show more qualitative results of our approach. Please zoom in for better viewing.
20

---

"a painLing of kilLens, grayscale"
"dog backing"
"a painting of Farn, grayscale" "bird chirping, tweeting"
painting beach, grayscale" "people cheering"
"a painling of mounLains, grayscale" "fizeworks banging"
painti of dogs, grayscale" "train whi st ing"
Figure 9: Random results. We show random results from our approach, using random audio and visual text prompt pairs. We provide failure cases in the last two rows. Please zoom in for hetter viewing.
21

---

Survey on Audiovisual Perception v2 [SOUND REQUIRED]
Video 2
bell
PLEASE TURN ON YOUR SOUND FOR THIS Su SEVEY
Thenk for portoppating this survey We oreating imgees ther alto be read os sound, and need your hels evalunting thei que we will show pairs of videooo you will anower quections about each palt
Image pronpt: patnting of castle towers, grayscate
ahared
VIDEO PAIR
Audio bell. ringing
Video
bell
Worhh bie
Shoee
Watch hullitee
Image painting castle grayscale
Which video LODKS most like 'castle tower?*
Video
Video 2
Which video SOUNDS most like "bello ringing?
pronpt: bell ringing
Video
Watch on
Yellube
In the video, we play the image as sound, from left to right which "castle"
'bells sounds?
Video
Video 2
Figure 10: Human study screenshots. We show screenshots from our human study survey, Here we show the title block, well the first pair of vidcos. The full survey contains 14 vidco pairs.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We confirm that the claims in the abstract and introduction (Sec. 1) accurately reflect our contribution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide discussion about the limitations of our approach in Sec. 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
Our paper does not include theoretical results or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: provide all implementation details in Sec. 4.1 and Appendix A.3. We also include all details of the human study experiments in Sec. 4.3 and Appendix A.3.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification: We provide all code in our supplemental material, including code for baselines. We will release all code on acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: /tpp:://nii h/tp///iippp//////// cc/
publ c gu des CodeSulom s. onPol cyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (ht (https: / /nips. cc/ tnips.cc/public///// cuublic/guidee
CodeS Submi ss www.onnoooloooooooooooo.ooo compolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the experimental setup details for quantitative evaluation in Sec. 4.2 and Appendix A.3 and for human study in Sec. 4.3 and Appendix A.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, the appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report performance with 95% confidence intervals for our main experiments in Tabs. and 2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
25

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify that we use NVIDIA L40s for all the experiments including baselines in Sec. 4.2, where we also specify that our method takes about 10 seconds to generate sample.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ /tt////neuue /wwurisuuu..////.
Answer: [Yes]
Justification:
We confirm that the research conducted conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss potential societal impacts in Sec. 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We use pretrained diffusion models with well-known risks. We refer readers to the model cards of these diffusion models for further discussion of safeguards. Our work does not introduce significant risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use Stable Diffusion v1.5, DeepFloyd IF, and Auffusion. We cite all these models, and provide links to model weights and licenses in Sec. 4.1 and Appendix A.3.
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw thoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We provide all details about the human study in Sec. 4.3 and Appendix A.3, including full text and screenshots.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The organization where this research was conducted has IRB approval for perceptual studies conducted over Mechanical Turk.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "s8DMEkCfn1", "reviewer": "Reviewer_mkQc", "summary": "This paper proposes a very creative idea, synthesize spectrograms that simultaneously look like natural images and also sound like natural audio, which they call images that sound. The method is rather simple, and leverages two pre-trained diffusion models, one text-to-image and the other text-to-spectrogram. A method is proposed to leverage the shared latent space and generate samples that are likely under both models. Both qualitative and quantitative results are provided, demonstrating the effectivenss of the proposed method.", "questions": "See weakness.", "limitations": "Yes, the authors have faithfully discussed the limitations of the proposed framework.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The biggest strength of the paper is the idea, very beautiful! Though making such kind of art is not completely new, this is the first method that did this as a scalable task. It nicely leverage state-of-the-art diffusion techniques and creates a model that can generate such artistic images that sound.\n\n- The paper is also very nicely written, with clear motivation, problem statement and formulation, and nice illustrations and figures.\n\n- The related work is complete and sufficient details are provided for reproducibility.\n\n- The paper presents both quantiative and qualitative evaluation with nice ablation study, demonstrating the effectiveness of the proposed method.", "weaknesses": "There are no major weaknesses. One can argue that the method is too simple: it's basicially just combining existing techniques on diffusion, leveraging pre-trained diffusion models StableDiffusion and Auffusion, and also getting inspirations from recent image diffusion papers. Having said that, the system just works and it's more an idea paper, so I don't mind it's a simple method.\n\nAnother weakness or question is that it would be good to provide some analysis of the shared latent space. Is there way to better interpret it? For example, would it be possible to doing some interporation in the latent space such that we can see some meangingful changes in both the image domain and audio domain.\n\nApart from just being cool, it would also be nice to include more discussions on some more concrete potential applications of such a system.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive and valuable feedback. Below are our responses.\n\n&nbsp;\n\n**Simplicity**\n\nWe appreciate the reviewer's comment that this is a creative idea. We believe that its simplicity is in fact a major strength. While our method is simple, it is not obvious that noise estimates from diffusion models of two different modalities can be combined together during the reverse diffusion process. To our knowledge, we are the first to combine diffusion models from different modalities for multimodal composition. That this can be done by using applying ideas from compositional generation to a new domain is a benefit, rather than a drawback.\n\n&nbsp;\n\n**Shared latent space**\n\nWe thank the reviewer for the suggestion. We use the same pretrained VAE encoder and decoder to map between the latent and pixel spaces for the reverse diffusion process. Since we rely on an existing latent space, we did not specifically explore its interpolation capabilities. However, we see this as an interesting direction for future work.\n\n&nbsp;\n\n**Potential applications**\n\nWe thank the reviewer for the suggestion. Our work focuses on exploring the ``intersection\" between the distribution of spectrograms and images, with *images that sound* being one artistic application. As we discussed in the paper, our approach could potentially be used in steganography to secretly embed images within audio for message delivery or vice versa.\n\n&nbsp;\n\nWe thank the reviewer for their time and hope these comments have addressed their questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bjlKSoebpj", "reviewer": "Reviewer_RECb", "summary": "The authors propose to leverage pre-trained text-to-image and text-to-spectrogram diffusion models, and de-noise noisy latents with both models in parallel during reverse process. They show that the proposed method can generate spectrogram aligned with audio prompt while having visual appearance of the image prompt with quantitative analysis as well as human study.", "questions": "- How are text prompts for both modalities (y_v, and y_a) selected, in Table 2, some of the pairs are the same objects, some of them are different, are there any rationale how these selections are made?", "limitations": "- The text prompts curated in this work is limited to only a few objects, it would be beneficial to include some methods to curate a richer and diverse prompts.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Proposed method has very interesting application, and can potentially provide new research direction in multimodal representation learning and generation.\n- Proposed method leverage pre-trained diffusion models in both modalities to achieve listenable spectrograms with meaningful visual semantic properties, which has application in creative domain.\n- This paper is clearly written and easy to follow, human study provides different perspective of proposed method compared to other baselines, and ablation study for vocoder, warm-starting and guidance scale presented in section 4.5 provide more thorough heuristics.", "weaknesses": "- Text prompts used for generating both spectrogram and image are mostly templating based, and composed only with simple object nouns. It would be interesting to show some generated examples on more complicated prompts with compositions of objects, this helps to provide a view into how proposed method can be scaled up for future directions.\n- There is lacking in depth discussion in the choice of including \"grayscale\", \"lithograph style\", \"black background\", and the effect of excluding these style words. It might help to provide potential future directions in controlling the styles between the two modalities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive and comprehensive feedback. Below are our responses.\n\n&nbsp;\n\n**Text prompt design**\n\nWe note that our paper already contains some examples with prompts that describe visual scenes, such as in Fig. 8, where we use the relatively complex text prompt of `\"a painting of a blooming garden with many birds, grayscale\"`. We intentionally used simple text prompts with object or scene nouns for most experiments to allow the diffusion models to have *more flexibility* during the denoising process, since we aim to sample from the intersection of the image and spectrogram distributions. In this setting, the models are much more constrained than they would be in normal synthesis. Introducing more complex prompts could push the models to sample from even more constrained distributions where *images that sound* are less likely to exist, leading to less satisfactory results. \n\n&nbsp;\n\n**Style words**\n\nVisual diffusion models are capable of generating RGB images, but spectrograms are only one channel. We therefore use style words like `grayscale` or `black background` to nudge the image denoising process toward a distribution that matches the spectrograms. As suggested by the reviewer, we conducted ablation experiments by removing the grayscale style word. The results are shown in the table below. The model produces similar results, but (as expected) the image quality slightly decreases while the audio quality slightly improves.\n\n| Exp | CLIP ↑| CLAP ↑ | FID ↓  |  FAD ↓ | \n|:----------|:----------:|:----------:|:----------:|:----------:|\n| w/o `grayscale` | 28.1 | **33.7** |   237.12  |  **18.09** |\n| w/ `grayscale` | **28.2**  | 33.5  |  **226.46** |  19.21 | \n&nbsp;\n\n**Prompt selection**\n\nFor human studies, we manually create prompts based on the prompt banks from the quantitative evaluation by augmenting them and ensuring semantic correspondence between image and audio to create artistic examples for evaluation. Please see Tab.5 in the appendix for the exact prompts used.\n\n&nbsp;\n\nWe thank the reviewer for their time and hope these comments have addressed their questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YrA3ithhaV", "reviewer": "Reviewer_quDM", "summary": "The paper proposes using pretrained text-to-image and text-to-speech diffusion and leveraging their compositional property to generate spectrograms that look like images and can be also be converted into meaningful sounds. The work is motivated by applications in art. The paper also curates a set of text prompts for conditioning the diffusion models. Besides, it evaluates the generated spectrograms using both automatic and human evaluation. In automatic evaluation, it measures both intra-modal similarity between the generated output and the modality-specific text prompts using CLIP-style models. In human evaluation, human subjects are asked to rate the output compared to baselines' on the basis of both intra- and cross-modal similarity between the generated output and the modality-specific text prompts. Furthermore, to tackle the lack of baselines for the task, the paper introduces two baselines by adapting existing methods, and shows that the proposed method performs better than the baselines across different evaluation types and metrics. Finally, the paper also provides good qualitative examples that show the promise of the idea.", "questions": "Could the rebuttal comment (more) on the following\n1. What kind of statistical properties is the paper referring to in L21-23: \"We hypothesize ... share statistical properties ...readily process\"? \n\n2. importance of shared latents (L72 and elsewhere) and if it is possible to provide an analysis/ablation to support the claim. See weakness 2 for details. \n\n3. how meaningful the quantiative metrics are (see weakness 3 for details) and if it is possible to report other standard image and audio generation metrics (see weakness 4 for details)\n\n4. i) why the model wins against SDS by a large margin even when \"SDS baseline often fails to optimize both modalities together, producing either spectrogram or image content only\" (L208-9). See weakness 5i for details.\n    ii) why the samples for human evaluation are cherrypicked. See weakness 5ii for details.\n    iii) why the cross-modality alignment metric makes sense when the prompts to the two diffusion models are not shared. See weakness 5iii for details.\n\n5. how the CLIP and CLAP scores are weighted while determining which model is better (details in weakness 6 and also related to Q3)\n\n6. the rationale behind limiting the prompts to the ones listed in table 4 given that the individual diffusion models work with a much broader set of prompts", "limitations": "The paper has discussed its limitations and societal impact.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Interesting idea: the idea of leveraging pretrained diffusion models and their compositionality to render spectrograms that look like real images and sound like real sounds is interesting and could have useful applications in art, as mentioned in the paper\n\n2. Extensive evaluation and good qualitative results: the paper extensively evaluates its model using both automatic and human evaluation, different evaluation metrics, and compares against different baselines despite the lack of existing methods for the task. Besides, it does important ablations of its method, which helps better understand the role of different design choices. Finally, the paper provides good qualitative examples, which help further demonstrate the strengths of the idea.\n\n3. Useful baselines: to tackle the lack of existing methods, the paper proposes two meaningful baselines for the task, which not only facilitate better model evaluation, but can potentially be useful for future work on this topic", "weaknesses": "1. Text is unclear / poorly structured at parts:\n    i) L21-23, \"We hypothesize ... share statistical properties ...readily process\": what kind of statistical properties is the paper referring to? Clear examples in the next version could help a reader.\n    ii) It's probably better to put the application/motivation para (L29-37) before the para in L21-28. The current order leaves a reader guessing the use of the work for quite a bit (at least that was the case with me)\n\n2. Importance of shared latents (L72 and elsewhere): I am not entirely convinced by the current content of the text why a model that does not share latents won't work. There is no model analysis/ablation for this claim as well. I think that if the model is able to work even with different prompts for the text-to-image and  text-to-audio model, it might work with separate latent space as well. \n\n3. Quantitative metrics are meaningful? if CLIP and CLAP are given equal weightage in table 1, auffusion is the strongest model, but obviously it isn't. This makes me wonder if these standalone quantitative metrics are even meaningful for this task.\n\n4. Lack of other standard quantiative metrics: the paper does not evaluate the generated outputs using standard metrics [1, 2] like FID, inception score, etc. \n\n5. Human study setup and results are not entirely clear.\n   i) In L208-9, the paper says that most of the time the SDS baseline collapses to either modality. If that is true, how come the win rate against SDS is so much higher than 50% for both audio and visual quality?\n   ii) Why are the examples hand-picked (L218) for human evaluation, shouldn't they be randomly sampled instead?\n   iii) It's not clear how the cross-modality alignment evaluation (L222-3, and upper and lower last row in table 2) makes sense when the text prompts to the two diffustion models are independent\n\n6. L278, \"attractive balance between CLIP and CLAP scores\": the choice of weightage on CLIP and CLAP for determining t_a and t_v in table 3 seems arbitrary. If they are given equal weightage, the best values would be t_v = 0.8 and t_a = 1.0?\n\n7. What's the rationale behind limiting the prompts to the ones listed in table 4 given that the individual diffusion models work with a much broader set of prompts?\n\n8. Minor:\n   i) L264, \"simply found adversarial examples against the vocoder\": the meaning of this phrase was unclear to me\n   \nRefs:\n[1] High-resolution image synthesis with latent diffusion models. Rombach et al.  CVPR 2022.\n[2] Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation. Arxiv 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comprehensive feedback. Below are our responses.\n\n&nbsp;\n\n**Importance of shared latent spaces**\n\nThis seems to be a misunderstanding: the latent spaces *must* be shared during joint sampling because if they weren't, then the latent vector would be decoded to two completely different images in the two modalities, which would trivialize into a different problem. Moreover, diffusion would fail if the latents did not span the same space or if the noise schedule differed. Finally, we stress our paper *does* include a model that works even without a shared latent space. The SDS version of the model (L174-182) successfully avoids this constraint because it directly backpropagates into pixels. It uses a pixel diffusion method (DeepFloyd) and an audio latent diffusion model (Auffusion). While it is outperformed by our joint sampling approach, this result suggests that the shared latent space is not strictly necessary. We will clarify this. \n\n&nbsp;\n\n**Writing of introduction**\n\nStatistical similarities between images and audio have been well established in previous work [A, B], which we would be happy to cite for clarity. However, we note that we have already provided several examples in the introduction: 1) we point out that both spectrogram and images have similar patterns that are famous \"objects of study\" in the natural image statistics (e.g., lines and edges) (L22-23), 2) the fact that frozen visual features have been surprisingly successful for audio models (L18-20). If there were no common statistical similarities between the two signals, then neither property would hold. Regarding the presentation order, we feel that the image statistics motivation helps readers understand the importance of the problem and motivates the generative modeling approach, which is why we described it before the task itself. We note that Fig.1 and its caption already present the work in a style that is quite similar to the reviewer's suggestion.\n\n&nbsp;\n\n**Quantitative metric**\n\nWe report Stable Diffusion and Auffusion as single-modality models, serving as upper and lower bounds for reference. CLIP and CLAP metrics are meaningful when compared with multimodal baselines, showing how well the generated results align with their respective prompts. These two networks have different common output ranges, so naively summing them without calibration would emphasize one score at the expense of the other. To fairly evaluate overall performance, we normalize each score based on the lower and upper bounds listed in Tab.1 and then sum them. As a result, in Tab.3, $t_v=0.9, t_a=1.0$ achieves the highest score of 1.154, while $t_v=0.8, t_a=1.0$ scores 1.139. For the results of FID and FAD metrics, please refer to the general response. These details will be included in the revised version.\n\n&nbsp;\n\n**SDS baseline**\n\nWe note that *collapsing to a single modality* does not necessarily mean that that modality is generated perfectly. It simply means that the other modality is not generated at all. Therefore, despite the collapse, the result may still be of poor quality, especially considering that the method is still jointly optimizing two losses. Additionally, due to the limitations of the SDS loss [83], the generated images tend to be overly saturated, while the audio is often distorted. As a result, the outcomes are generally inferior to our main method. For more examples, please refer to the supplementary video.\n\n&nbsp;\n\n**Human study** \n\nFor the human study, we compare our 3 proposed models' ability to generate examples that are qualitatively and artistically appealing. For each method, we generate a fixed number of samples per prompt pair and hand-select the best qualitative result. This also avoids having the 2AFC task performance determined by a model's failure rate (e.g., SDS collapsing to one modality), which would dominate results. We note that the random results used in the quantitative evaluation can be found in Fig.8. We will clarify this in our manuscript. \n\n&nbsp;\n\n**Cross-modality alignment**\n\nWe explain the cross-modality alignment metric in L623-629. This metric measures how well the visual structure of the image aligns with that of the spectrogram, rather than measuring *semantic* alignment, e.g., how an image edge of castle towers corresponds to the spectrogram onset pattern of ringing bells, or how an image of dogs matches the audio pattern of barking. Please see Fig.10 for the survey question. We'll clarify this in a revision.\n\n&nbsp;\n\n**Prompt selection**\n\nOur quantitative evaluation closely follows Visual Anagrams [38], randomly selecting 5 discrete (onset-based) and 5 continuous sound classes from VGGSound Common as audio prompts. For image prompts, we randomly chose 5 object and 5 scene classes, creating 100 prompt pairs through random combinations for evaluation and generating 10 samples for each. Our method scales easily for more prompts, whereas SDS baselines took more than 1,000 hours in total to generate current results for evaluation, taking two hours per example. Therefore, we kept the evaluation to a manageable scale.\n\n&nbsp;\n\n**Phrase of adversarial**\n\nWe thank the reviewer's suggestion. By this phrase, we meant spectrograms that the vocoder ignores, i.e., generating waveforms that do not match the inputs. We will rephrase this in a revision.\n\n&nbsp;\n\nWe hope these comments have addressed the reviewer’s questions. We would like to thank them for their time and ask that they consider raising their score for our paper.\n\n[A] Mynarski & McDermott. *Learning Mid-Level Auditory Codes from Natural Sound Statistics*, Neural Computation 2018.\n\n[B] McDermott & Simoncelli. *Sound texture perception via statistics of the auditory periphery: evidence from sound synthesis.* Neuron 2011.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0y9hNUqzX0", "reviewer": "Reviewer_w6xJ", "summary": "This paper explores the feasibility of synthesizing spectrograms (images that sound) that simultaneously look like natural images and sound like natural audio. This paper proposes a zero-shot approach, which leverages pre-trained text-to-image and text-to-spectrogram diffusion models that operate in a shared latent space, and denoises noisy latents with both the audio and image diffusion models in parallel. This paper shows quantitative evaluations and perceptual studies to claim the proposed method is able to generate spectrograms that align with a desired audio prompt while also matching the corresponding visual appearance.", "questions": "Please see the weaknesses section for my major concerns.", "limitations": "The authors have explained the current limitations, and potential negative social impact in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The topic introduced in this paper is very interesting and inspiring. \n+ The paper is clearly written, providing detailed explanations for easy understanding and reimplementation. The proposed method's capacity is fairly claimed and supported by convincing experimental results.", "weaknesses": "- Technical contribution is limited. The proposed method combines two existing latent diffusion models (audio and visual), weighted integrate noises from these two processes to ensure semantic consistency for both modalities. Additionally, another existing diffusion model is used for colorization. The technical innovation appears incremental, raising concerns about the novelty of the contribution.\n- The capability of the proposed model is not high. Despite the inspiring topic, the generated audio and visual quality show a noticeable performance gap compared to state-of-the-art generators. Ensuring consistency of both modalities in one image may result in less diverse outputs for audio and visual generation, potentially failing to accurately reflect prompt details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. Below are our responses.\n\n&nbsp;\n\n**Technical contribution**\n\nThe reviewer appears to have taken a narrow, algorithm-centric view of what constitutes a contribution. We note that this seems to be the major complaint in their (unusually short) review. Our work in fact makes a number of contributions. It is the first to propose the idea of generating examples that are natural in visual and audio signals, thereby exploring the intersection of two seemingly very different distributions. We are not aware of any work in the field of multimodal learning that has addressed this problem before. Beyond the novelty of this problem formulation (as acknowledged by mkQc, RECb, and quDM), the fact that we can find such examples is an interesting empirical contribution; it was not obvious *ex ante* that this could be done. Finally, we feel that our approach's simplicity is a major *strength*, not a drawback. While our approach draws inspiration from [38, 65], those approaches only combined noise from a single diffusion model. Showing that techniques from compositional generation can be adapted to this highly novel multimodal domain, and that (perhaps surprisingly) noise estimates from two different modalities' diffusion models can be successfully combined together, goes far beyond previous work.\n\n&nbsp;\n\n**Colorization**\n\nTo clarify, we provide colorized versions of the spectrograms in several qualitative examples using a post-processing procedure, since spectrograms are limited to grayscale. This is neither part of the evaluation nor a key component of our main method.\n\n&nbsp;\n\n**Model capabilities**\n\nFirst, we want to stress that generating examples that can be both viewed as images and listened to as sound in a zero-shot manner is extremely challenging: it requires using a single signal to represent two very different modalities without any supervision. As noted in our limitations section, the nature of this task *forces* the model to produce examples that may not be as natural as those from specialized models in each individual modality, since many visual patterns are improbable under audio models, and vice versa. Interestingly, our model generates high-quality results in both modalities despite these constraints. We have also shown that our method significantly outperforms several challenging baselines and ablations, providing further evidence for its abilities. \n\n&nbsp;\n\nWe hope these comments have addressed the reviewer’s questions. We would like to thank them for their time and ask that they consider raising their score for our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thorough comments and appreciate the recognition of the creativity of our work, described as "a beautiful idea" (mkQc) and "an interesting topic" (RECb, quDM, w6xJ). The acknowledgment of the "thorough evaluation" (mkQc, RECb, quDM) of our method and baselines is also valued. We answer some common questions below and provide new experimental results. 

&nbsp;

**Our Contributions**

We want to emphasize that our paper goes beyond just a novel application of diffusion models. It also pioneers the exploration of the intersection between two very different distributions (images and audio spectrograms), a domain that has not been explored before. We use learned image and audio distributions from diffusion models to probe the overlapped distribution between these two modalities for free. 

&nbsp;

**FID and FAD Evaluation**

Following reviewer quDM’s suggestion, we evaluated FID and FAD scores using generated examples from Stable Diffusion and Auffusion as reference sets respectively. As shown in the table below, our approach achieves the best performance. Note that FID and FAD are distribution-based metrics, and as our task focuses on generating examples that lie in a small subset of the natural image and spectrogram distribution, higher FID scores, in general, are expected. 

| Method                          |         Modality         |       FID ↓    |      FAD ↓   |
|:---------------------------------|:------------------:|:----------:|:---------:|
| Stable Diffusion | $\mathcal{V}$  | --       | 41.74   |
| Auffusion | $\mathcal{A}$  | 290.29   | --      |
| Imprint                         | $\mathcal{A}$ \& $\mathcal{V}$ | 244.84   | 29.42   |
| SDS                             | $\mathcal{A}$ \& $\mathcal{V}$ | 273.03   | 32.57   |
| Ours                            | $\mathcal{A}$ \& $\mathcal{V}$ | **226.46** | **19.21** |

Meta-review: This paper describes a method to generate spectrograms called "images that sound", where the spectrogram of perceptually natural audio looks like a natural corresponding image. This suggests that the distributions of natural images and spectrograms of natural sounds have surprising overlap.

Reviewers note that the paper is very well-written and appreciate the "beautiful idea" that is the core contribution of the paper. The majority of reviewers find the objective and subjective evaluations to be thorough, complete, and convincing. Sufficient details are provided for replication. 

The main weaknesses brought up by reviewers include concern about novelty, since the paper combines two prior latent diffusion models and is perhaps overlay simple; I tend to disagree that this is a major point of weakness, since this is more of an idea paper. Another point of concern is about general applicability of this method, as well as the quality of generated outputs (I tend to think quality of generated output isn't as much of a concern, given this is a challenging task, and more of an idea paper). The more critical reviewers pointed out a number of concerns including lack of clarity in some details, important of shared latents, meaningfulness of quantitative metrics, lack of standard metrics, some lack of clarity in human study, parameters of generative model sampling, and some other miscellaneous details. I think the author rebuttal and discussion provided clarity for these issues, and I encourage the authors to include these clarifications in the camera-ready draft.

Overall, with a consensus of reviewers recommending acceptance, I think this is a paper of interest to many audio, vision, and audio-visual researchers at NeurIPS that proposes and validates an interesting idea, and should thus be accepted.
Decision: Accept (poster)