Images that Sound: Composing Images and Sounds on a Single Canvas

Authors: ziyang  chen
daniel  geng
andrew  owens
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-22 22:07:03

Abstract: Spectrograms are 2D representations of sound that look very different from the images found in our visual world. And natural images, when played as spectrograms, make unnatural sounds. In this paper, we show that it is possible to synthesize spectrograms that simultaneously look like natural images and sound like natural audio. We call these visual spectrograms *images that sound*. Our approach is simple and zero-shot, and it leverages pre-trained text-to-image and text-to-spectrogram diffusion models that operate in a shared latent space. During the reverse process, we denoise noisy latents with both the audio and image diffusion models in parallel, resulting in a sample that is likely under both models. Through quantitative evaluations and perceptual studies, we find that our method successfully generates spectrograms that align with a desired audio prompt while also taking the visual appearance of a desired image prompt.

Main paper: 
 introduction: 
 The spectrogram is a ubiquitous low-dimensional representation for audio machine learning that plots the energy within different frequencies over time. But it is also widely used as a tool for converting 38th Conference on Neural Information Processing Systems (NeurIPS 2024)."castle" "bell ringing" "castle" & "bell ringing""kitten" "cat meowing" "kitten" & "cat meowing"Figure 2 : Images vs. spectrograms. We show grayscale images generated from Stable Diffusion [96] on the left, followed by log-mel spectrograms generated from Auffusion [118] in the middle, and our generated images that sound results on the right.sound into a visual form that can be-at least partially-perceived by sight. For example, in this representation (Fig. 2 ), event onsets look to a human observer like lines, and speech looks like a sequence of waves and bands. This insight is commonly used within the audio community, which frequently repurposes pretrained visual networks for audio tasks, often with only relatively minor modifications [48, 90, 69, 68, 34, 118, 112] .We hypothesize that the success of spectrograms in these roles is due in part to the fact that they share many statistical properties with the distribution of natural images, providing visual structures like edges and textures that the human visual system can readily process. Given the statistical similarities between images and sounds, we ask whether it is possible to automatically generate examples that lie at the intersection of both modalities. We create images that sound (Fig. 1 ), 2D matrices that look semantically meaningful when viewed as images, but that also sound meaningful when played as a spectrogram. This generative modeling problem is challenging, because it requires modeling a distribution that is induced by two very different data sources, and no relevant paired data is available.We are motivated by the "spectrogram art" that has been made by a variety of artists [10] , most famously by musicians Aphex Twin [2] , Venetian Snares [113] and Nine Inch Nails [85] . These artists manipulate their songs to display a desired image when they are visualized as spectrograms, such as by showing the artist's face or album art. In current practice, there is a steep trade-off between the quality of the image and the sound, since it is difficult to simultaneously control the interpretation of a signal in both modalities. As a result, existing artwork often comes across to the listener as dissonant or as random noise, rather than as natural sounds. 1 By contrast, we aim to generate signals that are as natural as possible in both modalities, such as towers that simultaneously sound like ringing bells or images of tigers that make a roaring sound (Fig. 1 ).In this work, we pose this problem as a multimodal compositional generation task and propose a simple, zero-shot method that composes off-the-shelf text-to-spectrogram and text-to-image diffusion models from different modalities. Inspired by prior work on compositionality in diffusion models [72, 29, 42, 41] , we denoise using both a noise estimate from the spectrogram model and a noise estimate from the image model. This is possible because these two models perform diffusion in the same latent space. The result is a sample that is simultaneously likely under the (text-conditional) distribution of spectrograms and images. The spectrograms are then converted to waveforms using a pretrained vocoder. In addition, we show that these black-and-white images may be colorized, resulting in color images whose grayscale versions can be played as spectrograms.Surprisingly, we find that off-the-shelf diffusion models trained on different modalities can be composed together to obtain samples that function as both an image and a sound. Often these examples reuse visual elements in unexpected ways (e.g., in Fig. 1 , a line is both the onset of a bell chime and the contour of a bell tower). We provide qualitative results, as well as quantitative comparisons and human study results against baselines, indicating that our method produces spectrograms that better align with both the audio and image prompts. Our contributions are summarized as follows:• We propose images that sound, a type of multimodal art that can be both understood as an image or played as a sound. • We show that we can compose pretrained diffusion models from different modalities in a zero-shot fashion to produce examples at the intersection of image and spectrogram distributions.Audio steganography. Audio steganography is the practice of concealing information within an audio signal. Artists have explored it for creative expression [111, 110] . Aphex Twin embedded a visual of his face in the audio waveform of the track "Formula" [2] . Noam Oxman creates animal portraits made of musical notations [88] . Other work has proposed deep learning methods for steganography, such as hiding video content inside audio files with invertible generative models [119] , hiding audio data inside an identity image [123] , and audio watermarking [15, 89, 99] . Our approach can be viewed as a steganography method that hides an image within an audio track, and is only revealed when the track is converted to a spectrogram.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We confirm that the claims in the abstract and introduction (Sec. 1) accurately reflect our contribution. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We provide a discussion about the limitations of our approach in Sec. 5. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 Our goal is to generate spectrograms that simultaneously represent both a sound and an image, each of which is specified by a text prompt. When the spectrogram is converted into a waveform, the sound matches the audio prompt, while when it is visually inspected, it should take the appearance of a given visual prompt (Fig. 1 ). To do this, we sample from the joint distribution of images and spectrograms, using off-the-shelf diffusion models trained on each modality independently. a respectively. We obtain the multimodal noise estimate ε(t) by a weighted average, then use it as part of the iterative denoising process. Finally, we decode the clean latent z0 to a spectrogram and convert it into a waveform using a pretrained vocoder (or by Griffin-Lim [47] ).Diffusion models. Diffusion models [54, 106] iteratively denoise standard Gaussian noise, x T ∼ N (0, I), to generate clean samples, x 0 , from some learned data distribution. At timestep t in the reverse diffusion process, the noise predictor, ϵ θ , takes the intermediate noisy sample, x t , and the condition y, such as a text prompt embedding, to estimate the noise ϵ θ (x t ; y, t). Following DDIM [104] , we obtain the next, less noisy, sample x t-1 at the previous timestep via:EQUATIONwhere ϵ t is independent Gaussian noise, α t is a predefined coefficient, and σ t controls the randomness level which we set to 0 for deterministic sampling. We may also optionally apply classifier-free guidance (CFG) [55] by modifying the noise estimate as:EQUATIONwhere γ denotes the strength of the conditional guidance and ∅ is the unconditional embedding of the empty string. This often results in much higher-quality samples.Latent diffusion. Latent Diffusion Models (LDMs) [96] perform the diffusion process in a latent space rather than in pixel space. A pretrained encoder and decoder pair, E and D, translates between pixel space and latent space. The latent space is typically much more compact and information-dense, which makes diffusion in this space more efficient. We use pretrained LDMs in our approach, due to the availability of audio and visual models with the same latent space.Our goal is to generate an example x ∈ R m×n that would be likely to appear under both visual and audio distributions, p a (•) and p v (•). We formulate this as sampling from a product of expert models2 [52] : p av (x) ∝ p a (x)p v (x). We follow recent work on the compositional generation that samples from this distribution using the score functions from pretrained diffusion models [30] . In contrast to these approaches, however, our two models are trained on two different modalities.We create our spectrograms using two pretrained latent diffusion models. One trained to generate images, ϵ ϕ,v (•, •, •), and the other to generate spectrograms, ϵ ϕ,a (•, •, •), both operating in the same latent space. We show an overview of our method in Fig. 3 . Given a noisy latent, z t , and text prompts y v and y a corresponding to the desired image and spectrogram prompt respectively, we compute two CFG noise estimates (Eq. ( 2)):EQUATIONEQUATIONwhere γ v and γ a are the corresponding visual and audio guidance scales. We then combine the noise estimates from both modalities by applying weighted averaging, producing a multimodal noise estimate that steers the denoising process toward a sample that is likely under the distribution of both images and spectrograms:EQUATIONwhereλ (t)a and λ(t)v are the weights of the audio and visual noise estimates at timestep t respectively.With this new noise estimate ε(t) , we perform a step of DDIM (Eq. ( 1)) to obtain a less noisy latent, z t-1 . Repeating this process we obtain the clean latent z 0 , which is then decoded using the decoder D to obtain the spectrogram x = D(z 0 ). This spectrogram can further be converted to a waveform using a pretrained vocoder or colorized to an RGB image whose grayscale version is the spectrogram.Warm-starting. We find it useful to warm-start the denoising process. In Sec. 4.5, we experiment with warm-starting using only the spectrogram noise estimates or only the image noise estimates. This can be represented by using w(t)a and w(t)v as the relative weight on the audio and the visual noise estimates respectively. We letEQUATIONa = H(t a T -t) and w(t) v = H(t v T -t)being Heaviside step functions, and t a and t v indicating the proportion of the reverse process that has audio or visual denoising respectively. When t a < 1.0 and t v = 1.0, we warm-start with only image denoising, and vice-versa. The above ensures that the weights λ (t) a and λ (t) v sum to one, and are equally weighted after warm-starting. Colorization. After we generate a spectrogram, x, we can optionally colorize it to create a more visually appealing result. Since our spectrograms fall outside the distribution of pre-trained colorization models, we use Factorized Diffusion [41] to colorize, which samples a diffusion model while projecting the noisy intermediate images such that they equal x when turned into grayscale. In doing so, the denoising process synthesizes only the "color component" of the sampled image, while the "grayscale component" is constrained to equal the generated spectrogram. Note that this method is similar to prior work [63, 23, 106, 114] . We choose this particular method due to its simplicity.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification: We provide all code in our supplemental material, including code for baselines.We will release all code on acceptance. Guidelines: Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experiments: 
 We evaluate our methods using quantitative metrics and human studies. We also present qualitative comparisons and an analysis of our method, and why it works.Models. We select a pair of off-the-shelf latent diffusion image and audio models that share the same latent space, encoder, and decoder. For the image model, we use Stable Diffusion v1.53 [96] . For the audio model, we use Auffusion4 [118] , which finetunes Stable Diffusion v1.5 on log-mel spectrograms. To synthesize audio from the log-mel spectrograms, we consider two options: following [118] and using off-the-shelf HiFi-GAN [66] vocoder, or the Griffin-Lim algorithm [47, 92] . We use HiFi-GAN for our main experiments. In Sec. 4.5, we evaluate the choice of vocoder and verify that our resultant waveforms do indeed encode to a visually interpretable spectrogram. Hyperparameters. We begin the reverse process with random latent noise z T ∈ R 4×32×128 , the same shape that Auffusion was trained on. Despite the image model not being trained on this specific size, we found that it nevertheless produces visually appealing results. We set the classifier guidance scales γ v and γ a to be between 7.5 and 10 and denoise the latents for 100 inference steps with warm-start parameters of t a = 1.0, t v = 0.9 to preserve audio priors. We decode the latent variables into images of dimension 3 × 256 × 1024. By averaging across each channel, we obtain spectrograms corresponding to 10 seconds of audio. We re-normalize the spectrograms for visualization.Baselines. As there is no previous work in this domain, we propose two baseline approaches. The first, inspired by Diffusion Illusions of Burgert et al. [11] , uses multimodal score distillation sampling (SDS). We optimize a single-channel image x = g(θ), where g is an implicit function parameterized by θ, using two SDS losses: one from the image diffusion model ϕ v and the other from the audio diffusion model ϕ a . This results in a gradient of:EQUATIONwhere λ sds is the weight of the image SDS gradient and ϵ is the noise added to the image or latents.We implement this with pixel-based diffusion model DeepFloyd IF [26] , as we find it performs better than Stable Diffusion with the SDS loss, and Auffusion [118] . This model thus does not require a shared latent space between vision and audio. We refer to this baseline as the SDS.The second baseline involves taking existing images and subtracting them from existing spectrograms, multiplied by some scaling factor, inspired by [25] . This works when the spectrograms have high power, as the subtraction does not significantly affect the audio but still imprints an image into the spectrogram. We obtain spectrograms and images for this baseline via Auffusion and Stable Diffusion. This approach, which we call imprint, is simple but can be surprisingly effective. All methods use the same vocoder and post-processing for fairness. Please see Appendix A.3 for more details.We start by quantitatively evaluating the quality of our generated images that sound, examining how well the generated examples match the provided text prompts for each modality.Experimental setup. Following the evaluation of Visual Anagrams [42] , we create two sets of text prompt pairs. We randomly select 5 discrete (onset-based) and 5 continuous sound category names from VGGSound Common [17] as audio prompts. We randomly chose 5 objects and 5 scene classes for image prompts, formatted as "a painting of [class], grayscale". This yields a total of 100 prompt pairs. We report Stable Diffusion and Auffusion performances as single-modalityTable 2 : Human study. We show win-rates of our spectrograms against those generated by the SDS and imprint baselines. The first row indicates which audiovisual prompt pair is evaluated, formatted as [audio prompt]/[visual prompt], with the last column being the average of all seven prompt pairs. Note that 50% win-rate is chance performance, and as such our method outperforms the baselines in the vast majority of cases. Also note that this is a best-case evaluation -please see Sec. 4.3 for details. All results reported are % win-rate against the baseline with a 95% confidence interval in gray (N = 100). "a painting of castle towers, grayscale" & "bell ringing" "a painting of furry kittens, grayscale" & "a kitten meowing for attention" "a painting of auto racing game, grayscale" & "a race car passing by and disappearing" benchmarks to establish upper and lower bounds. We generate 10 samples for each prompt pair, except for the SDS baseline, for which we generate 4 samples due to its slower speed.Evaluation metric. Following [50] , we use the CLIP [95] score to measure the alignment between spectrograms and image text prompts, and analogously we use the CLAP [116] score to evaluate the alignment of audio with audio text prompts. An ideal method should excel at both simultaneously. We also report FID [51] and FAD [65] to evaluate the quality of generated examples where we use the results of Stable diffusion and Auffusion as reference sets respectively.Results. We show our quantitative results in Tab. 1. Our method outperforms baselines across all metrics and performs comparably to single-modality models, which serve as rough upper bounds for each modality. This demonstrates our approach's ability to generate meaningful images that sound, sampling from the intersection of natural image and spectrogram distributions. Stable Diffusion achieves a low CLAP score, indicating how poorly a randomly sampled natural image acts as a spectrogram. We observe that the SDS baseline often fails to optimize both modalities together. In contrast, our method achieves a higher success rate and generates more diverse results. Our method is significantly faster, generating one sample in 10 seconds compared to the SDS baseline's 2-hour optimization time using NVIDIA L40s. The imprint baseline imprints the image onto the spectrogram, potentially degrading the sound pattern and leading to a lower CLAP score. Note that FID and FAD are distribution-based metrics, and as our task focuses on generating examples that lie in a small subset of the natural image and spectrogram distribution, higher FID scores, in general, are expected.Experimental setup. We also perform two-alternative forced choice (2AFC) studies to evaluate our results. We construct seven paired text prompts by hand, ensuring semantic correlations between image and audio prompts, such as pairing a visual of dogs with the sound of dogs barking. Using these prompts, we generate samples using our method, the SDS baseline, and the imprint baseline, and hand-pick the best examples for evaluation. This best-case evaluation is useful as participants from MTurk are not expected to have prior knowledge about spectrograms, let alone domain expertise. Moreover, this evaluation matches the intended use case of our method, in which a user repeatedly queries the model for a result that they prefer based on artistic merit and quality. Participants, are presented with one sample from our method, and a corresponding sample from a baseline, and are Results. Win-rates between our method and baselines are presented in Tab. 2, broken down by prompt pair. We also include averaged win-rates over all prompt pairs in the final column. As can be seen, our method outperforms the two baselines in most cases. Human evaluators consistently rate our spectrograms as being higher in audio and visual quality, and as being better "visually-synced"; on average our method is 2-3 times as likely to be chosen as the better sample than baselines.Results. We present qualitative results from our method as well as baselines in Fig. 4 , with additional results from our method in Figs. 5, 8 and 9 in the appendix. Audio of all results can be found on our website. As can be seen (and heard) our approach generates more visually appealing samples with better sound quality than compared to the baselines. The SDS baseline often focuses on one modality, to the detriment of the other, and in general, generates audio of lower quality. Moreover, the method suffers from the characteristic oversaturation of SDS-based results. The performance of the imprint baseline is highly dependent on the independently generated spectrogram and image and tends to fail when the two are misaligned, as in the castle example, or when the spectrogram has low energy, as the subtracted image is hard to see in already low energy regions of the spectrogram, such as with the kitten example. Interestingly, we found that our method often combines visual and acoustic elements. For example, the onsets of the bells ringing in Fig. 4 coincide with the towers of the castle, and the spectrogram patterns of meowing are hidden as stripes and edges on the kittens.We show additional hand-picked results from our approach in Fig. 5 with colorization results, in which we can see more examples of our method blending acoustic and visual elements, such as the water lilies corresponding to the frogs croaking, the corgis corresponding to the dogs barking, and the flowers corresponding to the birds chirping. Please see more results in Fig. 8 of Appendix A.2.Prior work [72, 29, 7, 42, 41] shows that diffusion models may be "composed" to generate samples that are likely under two or more different probability distributions.Our method can be seen as extending this idea of compositionality to multiple modalities. On the face of it, the distribution of spectrograms and the distribution of natural images would seem to be completely disjoint. However, as our results show, perhaps surprisingly, some overlap exists. We believe that this is possible for two reasons. First, spectrograms and images are both fairly flexible, allowing for significant amounts of perturbation or changes in style before becoming unrecognizable.And second, images and spectrograms share certain low-level characteristics, such as edges, curves, and corners, indicating a certain amount of similarity. Please see Appendix A.1 for more analysis.However, we find that not all compositions can be successful as shown in Fig. 9 of Appendix A.2. Moreover, careful selection of prompts is crucial to creating good results. For example, incorporating terms such as "lithograph style" or "black background" encourages the visual model to create areas of silence, which results in better quality, as shown in Fig. 5 .Vocoder. To extract waveforms from our generated spectrograms we use HiFi-GAN [66] , a neural vocoder. Given that our spectrograms are incredibly out of distribution, one concern with this setup is that the vocoder will ignore spectrograms and generate waveforms that do not match the inputs. To ameliorate this concern, we conduct a cycle consistency check by re-encoding the neural vocoder's predicted waveform back into a spectrogram by performing an STFT. As can be seen in Fig. 6 , the recomputed spectrograms are very similar to the original spectrogram, with only slightly less sharpness and some blurred textures 5 . This suggests we truly create visual spectrograms that look like images. We also experiment with using Griffin-Lim [92] as a vocoder, with similar results to HiFi-GAN as shown in Fig. 6 . We opt to use HiFi-GAN as our default vocoder as it outperforms Griffin-Lim in audio quality, with Griffin-Lim attaining a CLAP score of 0.302, compared to 0.335 obtained from HiFi-GAN. Please see more results in Fig. 7 of Appendix A.2.Warm-starting. We also conduct an ablation study on our warm-starting strategy by varying which modality is warm-started and by how many steps. Results are presented in Tab. 3, where t a and t v are defined in Sec. 3.2. We find that warm-starting the denoising process with either image or audio diffusion yields higher scores in the corresponding modality, as that modality effectively gets free reign to set the high-level features of the final result. We find that allowing the audio diffusion model to denoise alone for the first 10% of the timesteps results in an attractive balance between CLIP and CLAP scores. Therefore, we adopt t v = 0.9 and t a = 1.0 for our main experiments.Guidance scale. We also explore different guidance scales γ v and γ a for our method. We present results in Tab. 3. We find that higher guidance scales generally yield better results on both modalities. We hypothesize that the higher guidance scales more strongly encourage the sample to come from the "intersection" of the conditional spectrogram and conditional image distributions, resulting in better alignment with both text prompts. 
discussion and conclusion: 
 In this work, we demonstrate that, perhaps surprisingly, there is a non-trivial overlap between the distribution of natural images and the distribution of natural spectrograms. We show this by sampling from the intersection of these two distributions, resulting in spectrograms that look like real images but also sound like real sounds. The method we proposed is simple and zero-shot, and leverages the compositional nature of diffusion with cross-modal models. We see our work as advancing multimodal compositional generation and opening up new possibilities for multimodal art.Limitations. One limitation of our method is that it cannot generate examples that have both high-fidelity audio and image. We show failure cases, which occur for many prompts, in Fig. 9 . Some of these failures may be due to the strict constraints of the problem, since realistic examples may not always exist at the intersection of both distributions. Our method is also limited by the quality of the audio diffusion model, whose performance lags behind that of visual models.Potential negative societal impacts. The image and audio generation models that our method leverages are becoming progressively more powerful, and care must be taken in their deployment. Moreover, our method could potentially be used for steganography, secretly embedding images within audio. This capability may be used for deception, and we believe it deserves further consideration. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss potential societal impacts in Sec. 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: We use pretrained diffusion models with well-known risks. We refer readers to the model cards of these diffusion models for further discussion of safeguards. Our work does not introduce significant risk.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We use Stable Diffusion v1.5, DeepFloyd IF, and Auffusion. We cite all these models, and provide links to model weights and licenses in Sec. 4.1 and Appendix A.3.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Our paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes] Justification: We provide all details about the human study in Sec. 4.3 and Appendix A.3, including full text and screenshots.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [Yes] Justification: The organization where this research was conducted has IRB approval for perceptual studies conducted over Mechanical Turk.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Model capabilities. Through our experiments, we observed that our method generally performs well with "continuous" sound events (e.g., racing cars or train whistles) and simple visual prompts. Continuous sounds typically produce spectrograms with high energy distributed across time and frequencies, resulting in "white" spectrograms. This allows our model to effectively reduce sound energy, creating visual patterns that align with the audio.Simple visual prompts with object or scene nouns provide the diffusion models with more flexibility during denoising, enabling sampling from the overlapping distributions of images and spectrograms. However, more complex prompts could push the models into highly constrained distributions where images that sound are less likely to be generated.Generating discrete sounds (e.g., dog barking or birds chirping) is more challenging due to their sparse energy distribution. In these cases, the models are more constrained, making it difficult to produce visual content with clear edges and structures aligned with sound onsets, leading to less satisfactory results sometimes.Additionally, we emphasize that some prompt pairs may not have overlapping image and spectrogram distributions, making it impossible to create meaningful examples. For instance, combining the visual prompt starry night with the audio prompt playing guitar leads to a conflict, where the image modality tends toward a dark image, while the audio suggests a brighter one. More qualitative results. We show more qualitative results from our method with different prompts in Fig. 8 . Please see our website for video results. We also provide random examples with random prompt pairs in Fig. 9 with the last two rows as failure cases. For the failure cases, we can see that they either have good audio quality but lose clear visual patterns (mountains/fireworks) or have clear visual appearances but noisy audio (dogs/trains).Vocoder analysis. We show more examples of the vocoder cycle consistency experiment in Fig. 7 .As can be seen, the spectrograms from HiFi-GAN are quite similar to the original ones decoded from latents, indicating that our method does not find adversarial examples against the vocoder, but truly does find spectrograms that look like images.Colorization. We use DeepFloyd IF 6 [26] following Factorized Diffusion [41] for colorizing spectrograms. This technique colorizes a grayscale image by using a pretrained diffusion model zero-shot to generate the color component, and is similar to prior work such as [63, 23, 106, 114] . We use it due to its simplicity. We colorize spectrograms of size 1 × 256 × 1024 by directly feeding these into the diffusion model, which we found produced reasonable results despite the fact that the model was not trained for this size. We use prompts of the form "a colorful photo of [image prompt]" and denoise for 30 steps with a guidance scale of 10. Additionally, we found that starting the denoising at step 7 of 30 gave better results, which we hypothesize works because it gives the model a stronger prior for what the structure of the image is than starting from pure noise.SDS baseline. We follow Diffusion Illusions [11] to implement our SDS baseline with an implicit image representation. We use Fourier Features Networks [109] with a learnable MLP to generate images of size 1 × 256 × 1024. We use stage I of DeepFloyd IF-M to perform image score distillation sampling. We randomly make eight overlapping 256 × 256 crops and resize them to 64 × 64 to compute the averaged image SDS loss with a guidance scale of 80. For the audio modality, we use Auffusion [118] . As Auffusion is a latent diffusion model, we encode the images into 4 × 32 × 128 latents and perform the audio SDS loss with a guidance scale of 10, which we found gave the best performance in the audio-only generation. We set the weight of the image SDS loss λ sds to 0.4 to ensure balanced optimization for both modalities. We use the AdamW optimizer a learning rate of 10 -4 and weight decay of 10 -3 , and optimize the Fourier Feature Network for 40,000 steps. We also apply the warm-start strategy to this method by optimizing the audio SDS loss only for the first 5,000 steps by setting λ sds to zero. We note this method does not require a shared latent codebook between image and audio diffusion models.Algorithm 1 Pseudocode in a PyTorch-like style for the imprint baseline.# get images and specs from LDMs img = stable_diffusion(text_v) spec = auffusion(text_a) # reduce the energy give image masks mask = 1 -rho * (1.0 -img.mean(0)) spec = mask * spec audio = vocoder(spec)Imprint baseline. We begin by generating images, x img , and spectrograms, x spec , of size 256 × 1024 using Stable diffusion and Auffusion, respectively, both with a guidance scale of 7.5. Next, we use the generated images as masks by converting them into inverse grayscale images and scaling them by a factor ρ. This mask is then applied to the generated spectrogram to obtain the final result, given by x spec (1 -ρ gray(1 -x img )). The hyperparameter ρ controls the strength of energy reduction: larger values yield clearer visual patterns but poorer audio quality, and vice versa. To strike a good balance, we set ρ = 0.5. The imprint baseline takes 10 seconds to generate a sample on NVIDIA L40s.Prompt selection. We present the image and audio prompts used for the quantitative evaluation in Tab. 5. We use 10 prompts for each modality, for a total of 100 prompt pairs. Participants for the human study were recruited from Amazon Mechanical Turk (MTurk), and were paid 0.50 USD for a task lasting less than 5 min. We use a total of seven prompt pairs and compare them against two baselines: the SDS baseline and the imprint baseline. For each method and prompt pair, we hand-selected two high-quality samples for a total of 84 videos. Each video is about 10 seconds long, and includes a vertical line moving from left to right, indicating the current temporal position in the spectrogram. All participants were shown 14 pairs of videos-seven pairs comparing our method to the SDS baseline, and seven pairs comparing our method to the imprint baseline, all randomly selected and blinded. The participants are then asked to answer three questions: The first two questions are designed to evaluate the quality of the audio and image generated by the methods, and their alignment with the respective prompts. The third question seeks to understand how well the visual structure or texture of the generated image and spectrogram align. However, note we were not able to guarantee that the participants had prior experience with spectrograms. To mitigate this to an extent, we include the description as a preamble to the third question. Also, note that we use abbreviated versions of the audio and visual prompts to avoid excessively long questions.We provide the prompt pairs we used for human studies in Tab. 6 for reference, and screenshots of our survey including the title block as well as the first video pair and associated questions in Fig. 10 .Table 6 : Text prompts for the human studies. We note that the prompts are paired.Audio prompts a painting of castle towers, grayscale bell ringing a painting of cute dogs, grayscale dog barking a painting of a blooming garden with many birds, grayscale birds singing sweetly a painting of furry kittens, grayscale a kitten meowing for attention a painting of auto racing game, grayscale a race car passing by and disappearing a painting of trains, grayscale train whistling a painting of tigers, grayscale tiger growling "a painting of cute dogs, grayscale" & "dog barking""a painting of a blooming garden with many bird, grayscale" & "birds singing sweetly" "a painting of tigers, grayscale" & "tiger growling""a painting of trains, grayscale" & "train whistling"Figure 8 : More qualitative results. We show more qualitative results of our approach. Please zoom in for better viewing."a painting of kittens, grayscale" & "dog barking""a painting of a farm, grayscale" & "bird chirping, tweeting" "a painting of a beach, grayscale" & "people cheering" "a painting of mountains, grayscale" & "fireworks banging" "a painting of dogs, grayscale" & "train whistling"Figure 9 : Random results. We show random results from our approach, using random audio and visual text prompt pairs. We provide failure cases in the last two rows. Please zoom in for better viewing. NeurIPS Paper ChecklistWe encourage the reader to listen to popular examples of spectrogram art[10].Recent work has called this a conjunction[30], since conceptually the samples are roughly from the intersection of both distributions.Stable Diffusion v1-5 hugging face model cardAuffusion hugging face model cardWe note that perfect cycle consistency is not generally possible since vocoders are fundamentally lossy.https://huggingface.co/DeepFloyd 


Decision: This paper describes a method to generate spectrograms called "images that sound", where the spectrogram of perceptually natural audio looks like a natural corresponding image. This suggests that the distributions of natural images and spectrograms of natural sounds have surprising overlap.

Reviewers note that the paper is very well-written and appreciate the "beautiful idea" that is the core contribution of the paper. The majority of reviewers find the objective and subjective evaluations to be thorough, complete, and convincing. Sufficient details are provided for replication. 

The main weaknesses brought up by reviewers include concern about novelty, since the paper combines two prior latent diffusion models and is perhaps overlay simple; I tend to disagree that this is a major point of weakness, since this is more of an idea paper. Another point of concern is about general applicability of this method, as well as the quality of generated outputs (I tend to think quality of generated output isn't as much of a concern, given this is a challenging task, and more of an idea paper). The more critical reviewers pointed out a number of concerns including lack of clarity in some details, important of shared latents, meaningfulness of quantitative metrics, lack of standard metrics, some lack of clarity in human study, parameters of generative model sampling, and some other miscellaneous details. I think the author rebuttal and discussion provided clarity for these issues, and I encourage the authors to include these clarifications in the camera-ready draft.

Overall, with a consensus of reviewers recommending acceptance, I think this is a paper of interest to many audio, vision, and audio-visual researchers at NeurIPS that proposes and validates an interesting idea, and should thus be accepted.

Reviews: review: paper_summary: paper_summary: This paper proposes a very creative idea, synthesize spectrograms that simultaneously look like natural images and also sound like natural audio, which they call images that sound. The method is rather simple, and leverages two pre-trained diffusion models, one text-to-image and the other text-to-spectrogram. A method is proposed to leverage the shared latent space and generate samples that are likely under both models. Both qualitative and quantitative results are provided, demonstrating the effectivenss of the proposed method.
strength_weakness: strength_weakness: There are no major weaknesses. One can argue that the method is too simple: it's basicially just combining existing techniques on diffusion, leveraging pre-trained diffusion models StableDiffusion and Auffusion, and also getting inspirations from recent image diffusion papers. Having said that, the system just works and it's more an idea paper, so I don't mind it's a simple method.

Another weakness or question is that it would be good to provide some analysis of the shared latent space. Is there way to better interpret it? For example, would it be possible to doing some interporation in the latent space such that we can see some meangingful changes in both the image domain and audio domain.

Apart from just being cool, it would also be nice to include more discussions on some more concrete potential applications of such a system.
questions: questions: See weakness.
limitations: limitations: Yes, the authors have faithfully discussed the limitations of the proposed framework.
paper_summary: paper_summary: The authors propose to leverage pre-trained text-to-image and text-to-spectrogram diffusion models, and de-noise noisy latents with both models in parallel during reverse process. They show that the proposed method can generate spectrogram aligned with audio prompt while having visual appearance of the image prompt with quantitative analysis as well as human study.
strength_weakness: strength_weakness: - Text prompts used for generating both spectrogram and image are mostly templating based, and composed only with simple object nouns. It would be interesting to show some generated examples on more complicated prompts with compositions of objects, this helps to provide a view into how proposed method can be scaled up for future directions.
- There is lacking in depth discussion in the choice of including "grayscale", "lithograph style", "black background", and the effect of excluding these style words. It might help to provide potential future directions in controlling the styles between the two modalities.
questions: questions: - How are text prompts for both modalities (y_v, and y_a) selected, in Table 2, some of the pairs are the same objects, some of them are different, are there any rationale how these selections are made?
limitations: limitations: - The text prompts curated in this work is limited to only a few objects, it would be beneficial to include some methods to curate a richer and diverse prompts.
paper_summary: paper_summary: The paper proposes using pretrained text-to-image and text-to-speech diffusion and leveraging their compositional property to generate spectrograms that look like images and can be also be converted into meaningful sounds. The work is motivated by applications in art. The paper also curates a set of text prompts for conditioning the diffusion models. Besides, it evaluates the generated spectrograms using both automatic and human evaluation. In automatic evaluation, it measures both intra-modal similarity between the generated output and the modality-specific text prompts using CLIP-style models. In human evaluation, human subjects are asked to rate the output compared to baselines' on the basis of both intra- and cross-modal similarity between the generated output and the modality-specific text prompts. Furthermore, to tackle the lack of baselines for the task, the paper introduces two baselines by adapting existing methods, and shows that the proposed method performs better than the baselines across different evaluation types and metrics. Finally, the paper also provides good qualitative examples that show the promise of the idea.
strength_weakness: strength_weakness: 1. Text is unclear / poorly structured at parts:
    i) L21-23, "We hypothesize ... share statistical properties ...readily process": what kind of statistical properties is the paper referring to? Clear examples in the next version could help a reader.
    ii) It's probably better to put the application/motivation para (L29-37) before the para in L21-28. The current order leaves a reader guessing the use of the work for quite a bit (at least that was the case with me)

2. Importance of shared latents (L72 and elsewhere): I am not entirely convinced by the current content of the text why a model that does not share latents won't work. There is no model analysis/ablation for this claim as well. I think that if the model is able to work even with different prompts for the text-to-image and  text-to-audio model, it might work with separate latent space as well. 

3. Quantitative metrics are meaningful? if CLIP and CLAP are given equal weightage in table 1, auffusion is the strongest model, but obviously it isn't. This makes me wonder if these standalone quantitative metrics are even meaningful for this task.

4. Lack of other standard quantiative metrics: the paper does not evaluate the generated outputs using standard metrics [1, 2] like FID, inception score, etc. 

5. Human study setup and results are not entirely clear.
   i) In L208-9, the paper says that most of the time the SDS baseline collapses to either modality. If that is true, how come the win rate against SDS is so much higher than 50% for both audio and visual quality?
   ii) Why are the examples hand-picked (L218) for human evaluation, shouldn't they be randomly sampled instead?
   iii) It's not clear how the cross-modality alignment evaluation (L222-3, and upper and lower last row in table 2) makes sense when the text prompts to the two diffustion models are independent

6. L278, "attractive balance between CLIP and CLAP scores": the choice of weightage on CLIP and CLAP for determining t_a and t_v in table 3 seems arbitrary. If they are given equal weightage, the best values would be t_v = 0.8 and t_a = 1.0?

7. What's the rationale behind limiting the prompts to the ones listed in table 4 given that the individual diffusion models work with a much broader set of prompts?

8. Minor:
   i) L264, "simply found adversarial examples against the vocoder": the meaning of this phrase was unclear to me
   
Refs:
[1] High-resolution image synthesis with latent diffusion models. Rombach et al.  CVPR 2022.
[2] Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation. Arxiv 2024.
questions: questions: Could the rebuttal comment (more) on the following
1. What kind of statistical properties is the paper referring to in L21-23: "We hypothesize ... share statistical properties ...readily process"? 

2. importance of shared latents (L72 and elsewhere) and if it is possible to provide an analysis/ablation to support the claim. See weakness 2 for details. 

3. how meaningful the quantiative metrics are (see weakness 3 for details) and if it is possible to report other standard image and audio generation metrics (see weakness 4 for details)

4. i) why the model wins against SDS by a large margin even when "SDS baseline often fails to optimize both modalities together, producing either spectrogram or image content only" (L208-9). See weakness 5i for details.
    ii) why the samples for human evaluation are cherrypicked. See weakness 5ii for details.
    iii) why the cross-modality alignment metric makes sense when the prompts to the two diffusion models are not shared. See weakness 5iii for details.

5. how the CLIP and CLAP scores are weighted while determining which model is better (details in weakness 6 and also related to Q3)

6. the rationale behind limiting the prompts to the ones listed in table 4 given that the individual diffusion models work with a much broader set of prompts
limitations: limitations: The paper has discussed its limitations and societal impact.
paper_summary: paper_summary: This paper explores the feasibility of synthesizing spectrograms (images that sound) that simultaneously look like natural images and sound like natural audio. This paper proposes a zero-shot approach, which leverages pre-trained text-to-image and text-to-spectrogram diffusion models that operate in a shared latent space, and denoises noisy latents with both the audio and image diffusion models in parallel. This paper shows quantitative evaluations and perceptual studies to claim the proposed method is able to generate spectrograms that align with a desired audio prompt while also matching the corresponding visual appearance.
strength_weakness: strength_weakness: - Technical contribution is limited. The proposed method combines two existing latent diffusion models (audio and visual), weighted integrate noises from these two processes to ensure semantic consistency for both modalities. Additionally, another existing diffusion model is used for colorization. The technical innovation appears incremental, raising concerns about the novelty of the contribution.
- The capability of the proposed model is not high. Despite the inspiring topic, the generated audio and visual quality show a noticeable performance gap compared to state-of-the-art generators. Ensuring consistency of both modalities in one image may result in less diverse outputs for audio and visual generation, potentially failing to accurately reflect prompt details.
questions: questions: Please see the weaknesses section for my major concerns.
limitations: limitations: The authors have explained the current limitations, and potential negative social impact in the paper.

Comments: title: Author Reply
Response to rebuttal 2
Author-reviewer discussion ending soon
Response to rebuttal
comment: We thank the reviewer for recommending acceptance. We appreciate this discussion and will incorporate those changes in our final version.
Thanks for the further clarification. These make sense and improve the interpretability of the proposed model. I suggest the authors could include these interesting observations and conclusions in the analysis. I have increased my rating.
We thank the reviewer for their quick response. We will include all additional results and clarifications in our manuscript as the reviewer suggests. Additionally, we will keep the human study in the paper also as suggested by the reviewer, but make it abundantly clear that we are evaluating the best-case performance of all methods, and clearly explain the rational for doing so as listed above. We agree with the analysis that human studies are the best way to evaluate methods that are artistic in nature, as is the case with our method. We also agree with the reviewer that the cited papers do not explicitly mention hand-picking. We cite them to show that it is common for human evaluations to not be entirely random, in terms of prompts chosen, but also in terms of model inputs in general.
Thanks for the additional table. I would urge the authors to add all additional results and clarifications to the next draft.

Regarding hand-picking outputs for human evaluation, I skimmed all 4 papers referred to in the latest response from the authors, but couldn't find any mention of hand-picking outputs. They do control the inputs, but that's in essence similar to controlling the text prompts in this work. However, the other arguments made by the authors are not very unreasonable. As for pushing the user study to supp., I would advise against it, as human user study is arguably the best way to evaluate these generative models, as shown in other papers in this area, including the ones cited in the paper and the rebuttal responses, as well, especially when their applications are mostly artistic in nature.
We thank the reviewer for recommending acceptance and valuable feedbacks for this work.
Thank the authors for the additional clarifications. I don't have further questions at this point.
Dear authors and reviewers,

Thank you for the robust discussions so far. We only have about a day left in the author-reviewer discussion period, so please prioritize any remaining questions or responses.

In particular, it would be valuable to hear from reviewers mkQc and RECb. Could you please respond to acknowledge that you have read and considered the reviews and author rebuttal, and post any remaining questions that would be useful to discuss with the authors? Have the author responses addressed your concerns?

Thanks in advance.

All the best,
Your AC
We thank the reviewer for recommending acceptance. We appreciate this discussion and will incorporate the suggestions in our final version.
Thank to the authors for your answers. I still think this is an interesting idea and I am keeping my rating.
We thank the reviewer for the further clarification about their reviews. Below are our new responses. 

&nbsp;

**Technical contribution**

Our main method is simple but non-trivial for combining noise estimates from two different modalities' diffusion models, which no prior work has explored. Besides our main method, we also proposed two other methods called *imprint* and *SDS* baseline that generate *images that sound* examples from different perspectives (L174-188). Overall, we believe our work makes significant technical contributions. For clarification, colorization is not one of our claimed contributions. We simply use it to improve the visualization quality of the results.

&nbsp;


**Model capabilities**

The reviewer states that their primary concern is a lack of analysis of the capabilities of our model. We discussed some of them in L257 and L293. Further, we give a detailed explanation below:

First, we want to emphasize that the nature of this task inherently forces the model to produce examples that may not appear as natural as those from specialized models in each modality, as many visual patterns are unlikely to align with audio models. In other words, realistic examples are constrained to the overlapped distribution between both modalities, which inherently limits the quality of the generated results. Furthermore, our experiments show that we outperform many other possible approaches, highlighting the capability of our approach. Our results also qualitatively outperform artist-created examples, which often produce random noise (e.g., Aphex Twin and others [9]).

Through our experiments, we observed that our method generally performs well with "continuous" sound events (e.g., racing cars or train whistles) and simple visual prompts. Continuous sounds typically produce spectrograms with high energy distributed across time and frequencies, resulting in “white” spectrograms. This allows our model to effectively reduce sound energy, creating visual patterns that align with the audio.

Simple visual prompts with object or scene nouns provide the diffusion models with *more flexibility* during denoising, enabling sampling from the overlapping distributions of images and spectrograms. However, more complex prompts could push the models into highly constrained distributions where *images that sound* are less likely to be generated.

Generating discrete sounds (e.g., dog barking or birds chirping) is more challenging due to their sparse energy distribution. In these cases, the models are more constrained, making it difficult to produce visual content with clear edges and structures aligned with sound onsets, leading to less satisfactory results **sometimes**.

Additionally, we emphasize that some prompt pairs may not have overlapping image and spectrogram distributions, making it impossible to create meaningful examples. For instance, combining the visual prompt `starry night` with the audio prompt `playing guitar` leads to a conflict, where the image modality tends toward a dark image, while the audio suggests a brighter one.

Lastly, we note that our use of off-the-shelf Stable Diffusion, trained on $512 \times 512$ RGB images, to generate $256 \times 1024$ grayscale images directly could potentially limit our model’s capabilities. The performance could be possibly improved by having a specific image diffusion model that is suitable for this task. We will include a more detailed analysis in the manuscript. 

&nbsp;

We hope these comments have addressed the reviewer’s questions
We thank the reviewers for their time. Below are our clarifications: 

1. **Quantitative metrics**: we provide the combined score (CLIP+CLAP) for baselines below:

| Method                          |         Modality         |       CLIP + CLAP    |   
|:---------------------------------|:------------------:|:----------:|
| Stable Diffusion | $\mathcal{V}$  | 1.0       | 
| Auffusion | $\mathcal{A}$  | 1.0  |
| Imprint                         | $\mathcal{A}$ \& $\mathcal{V}$ | 1.04  | 
| SDS                             | $\mathcal{A}$ \& $\mathcal{V}$ | 0.70  | 
| Ours                            | $\mathcal{A}$ \& $\mathcal{V}$ | **1.15** | 


2. **Human study**: 

Due to constraints in the method or experiments, it is relatively common for human studies to not be completely random. We provide a short list of citations and explanations below.

In our case, we specifically choose to study the best-case scenario for the following reasons:

- "Images that sound" are quite hard to generate. (In fact, prior to this work, it was not clear at all that they even existed.) Not all text prompt pairs give good results, and some prompt pairs are just impossible. As such, we envisioned that a user would use our method to iteratively sample multiple times, choosing the result that they preferred most. To mimic this use case, we designed our human study to quantify the best-case results.

- We find that it is very hard to evaluate "images that sound" on Amazon Mechanical Turk, primarily due to the fact that almost all participants did not understand the concept of a spectrogram. As a result, it is quite difficult to get precise evaluation metrics when using random results, and financially prohibitive to reduce error bars to reasonable levels. The best-case evaluation circumvents these difficulties.

Moreover, we point out that we do in fact evaluate random results systematically and quantitatively, in a scalable fashion, with CLIP and CLAP metrics in Table 1 and FID and FAD metrics in the general rebuttal response above. We would be willing to move the human study to the appendix if the reviewer believes it would improve the manuscript.

[1] "PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation", Zhang et al. conduct human studies on specifically 7 chosen and captured environments.

[2] "WonderJourney: Going from Anywhere to Everywhere", Yu et al. use a total of 6 hand-designed videos for human evaluation.

[3] "TEXTure: Text-Guided Texturing of 3D Shapes", Richardson et al. specifically pick 10 text prompts to evaluate on.

[4] "DreamGaussian4D: Generative 4D Gaussian Splatting", Ren et al. conduct a human study on 12 specifically chosen images.

&nbsp;

We hope these comments have addressed the reviewer’s questions.
Thanks for the reponses. Could you comment on/clarify the following?

1. "Quantative metrics": how do the baslines perform on the combined metric?

2. "Human study": is the idea of hand-picking for comparative human evaluation of generative models common? I think a better alternative would be to run the models with the same prompts multiple times and then compare all possible pairs, for computing the win/loss rates. It's undoubtedly more expensive but potentially more 'foolproof' .
Thanks for the authors' response. However, I think I need to re-clarify my concerns after reading the response.
1) Technical contribution. I'm a little bit confused about the definition of "narrow, algorithm-centric view" and "unusually short reviews". I agree with the novelty and contribution of bringing such an inspiring and interesting task to the community. The technical details, proposed solution and evaluations are all sound and fair. I don't have concerns or questions on the details of the work except for the overall questions. So I mentioned about my concern only on the "technical contribution" side, which specifically means the novelty in terms of bridging the existed methods is a simple way. Indeed, I agree that simplicity is an advantage. So I list this concern as one of the possible weaknesses for the reviewers and AC to judge thoroughly. Just to clarify, I will only consider this part as one of my concerns but not a critical issue of this work that blocks it from publishing.
2) Colorization. This is not my major concern or even a question. I mentioned this because of Line 151 "we use Factorized Diffusion [37] to colorize", to support my concern on the technical contribution reviews.
3) Model capabilities. The model capability is indeed my major concern. I understand the task is a challenging one, but sometimes we as reviewers can have different standard to balance the capacity and an interesting task for a conference paper. I acknowledge that the authors have shown their capacity and advantages on fair evaluation scope. But after reading the paper and response I still feel it would be a problem for the proposed method to scale on general audio-visual examples. A more strong response on this side with more detailed explanations on what case can generally work and what can't, or potential solutions on that as well as analysis etc. can all help solve my concerns.
Given the consideration above, I gave the borderline reject rating. I look forward to have more fair discussion on these side and will be very open to change my mind. It won't bother me if the paper gets accepted even with my current rating. Sorry for causing the misunderstanding from my previous reviews. I think this can better clarify my points.