LLMs can see and hear without any training
Authors: Kumar Ashutosh
Yossi Gandelsman
Xinlei Chen
Ishan Misra
Rohit Girdhar
Venue: ICML 2025 poster (2025)
Keywords: large language models
LLMs
multimodal
reasoning
captioning
image generation
test-time optimization
gradient-free optimization
Primary area: deep_learning->large_language_models
TL;DR: Text-only LLMs can perform multimodal tasks (eg captioning, text-to-image, stylization) without any training! LLM generates candidates that are scored by an off-the-shelf model, and fed back iteratively, eventually producing the final output.

Abstract: We present MILS: Multimodal Iterative LLM Solver, a surprisingly simple, training-free approach, to imbue multimodal capabilities into
your favorite LLM. Leveraging their innate ability to perform multi-step reasoning, MILS prompts the LLM to generate candidate outputs, each of which are scored and fed back iteratively, eventually generating a solution to the task. This enables various applications that typically require training specialized models on task-specific data. In particular, we establish a new state-of-the-art on emergent zero-shot image, video and audio captioning. MILS seamlessly applies to media generation as well, discovering prompt rewrites to improve text-to-image generation, and even edit prompts for style transfer! Finally, being a gradient-free optimization approach, MILS can invert multimodal embeddings into text, enabling applications like cross-modal arithmetic.

LLMs can see and hear without any training
Kumar Ashutosh?
Yossi Gandelsman 13
Xinlei Chen |
Ishan Misra'
Rohit Girdhar'
Abstract
a Multimodal Iterative LLM Solver, uses LLMs as "GENERATOR" to propose candidate solutions to a given task, and an off-the-shelf multimodal model as "SCOREr" to evaluate the quality of each proposal for the said task. The output of the SCORER is fed back into the GENERATOR to give it feedback, and helps produce the next set of can- didates that are more likely to solve the task, This iterative process is run until convergence, or a certain number of steps, and produces an output for the task. We find this sim- ple approach is surprisingly powerful and versatile, working across a variety of tasks and modalities. Using different com- binations of GENERATORS and SCORERS, MILS is able to tackle tasks including multimodal captioning, generation, editing, and multimodal arithmetic.
MILS:
Multimodal Iterative LLM Solver, a surprisingly simple, training-free ap- proach, imbue multimodal capabilities into your favorite LLM. Leveraging their innate ability to perform multi-step reasoning, MILS prompts the LLM to generate candidate outputs, each of which are scored and fed back iteratively, eventu- ally generating a solution to the task. This enables various applications that typically require training specialized models on task tpsc-ipecific data. In partic- ular, we establish a new state-of-the-ar on emer- gent zero-shot image, video and audio captioning. MILS seamlessly applies to media generation as well, discovering prompt rewrites to improve text- to-image generation, and even edit prompts for style transfer! Finally, being a gradient- free opti- mization approach, MIL.S can invert multimodal embeddings into text, enabling applications like cross-modal arithmetic. The code to reproduce MILS is available at https:/ htt///ggttt/btt////////////tttht com/ facebookrresarcch/MIi
We present MILS:
Most prior work for such tasks uses specialized models, often trained data curated for that task. For instance, zero-shot image captioning models are often still trained on paired image-caption data. MILS, on the other hand, does not need any such training, and exhibits emergent zero-shot capabilities. For instance, for image captioning, MILS uses a standard Llama (Dubey et al., 2024) LLM as the GENERA- TOR, along with CLIP (Radford et al., 2021) as the SCORER. Note that while CLIP is trained on image-text data, it is not trained on clean image-caption data that typical captioning models are trained on. Most vision language models that produce captions leverage CLIP only as an initialization, and require post-training on curated captioning data. Hence, while such models may exhibit zero-shot generalization to a new data distribution at test time, MILS exhibits emergent zero-shot generalization to the new task of captioning.
1. Introduction
Test-time reasoning ability of Large Language Models (LLMs) has emerged as a powerful tool for solving chal- lenging tasks. Recently, OpenAl introduced 01 (OpenAI), model trained using reinforcement learning to leverage test-time compute for progressively better results, especially on complex math and coding tasks. Even without additional training, LLMs have shown impressive improvements by us- ing test time compute in the form Chain-of- Shain-of-Thought (CoT) reasoning, by rolling out an execution plan to respond to a user's query (Wei et al., 2022; Kojima et al., 2022).
Furthermore, while there do exist some captioning ap- proaches that do not leverage captioning data (Salewski et al., 2023; Tewel et al., 2022; Shaharabany et al., 2023; Zeng et al., 2024), they are limited to a specific modality and more importantly, the specific task. These approaches typically leverage gradients from CLIP to guide the next token prediction, limiting them captioning. MILS, on the other hand, seamlessly generalizes to new tasks and modalities by simply swapping out the GENERATOR and SCORER modules. For instance, a GENERATOR constructed by simply chaining an LLM to a Text-to- Image is able to improve over ofate-of-the-ar T2I models by using the LLM as a "prompt rewriter", a capability not afforded by prior work.
In this work, we leverage this innate reasoning ability in LLMs to solve multimodal understanding and genera- tion tasks, without needing any training! Our approach,
'Meta AI 'TT Austin 'UC Berkeley. Correspondence to: Kumar Ashutosh <kumar.ashutoshe @ utexas.edu>>, Rohit Girdhar <rgirdhar met.
(T2I) model,
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

LLMs can see and hear without any training
Image, Video or Audio Captioning
Improved Image Generation
Image of city's tallest, historic landmark with sepia litter.
Baseling
Ours
Baseling
Ours.
An umbrelia on top of spoon.
A zebra to the right of a fire hydrant.
Style Transfer
Cross-Modal Arithmetic
Inpud Amaze
Shyle image
Output image
STOP
Growing Thunder
STOP
Figure Our proposed approach, MILS, enables various applications. from captioning images, video, or audio; improving text-to-image generation; image editing such as style transfer; as well as arithmetic across different modalities by inverting them all into text. It accomplishes all this using purely test-time optimization approach without any task specific training or data curation!
In this work, we show applicability of MILS across three different visual and non-visual modalities: images, videos and audio, and three different tasks: captioning, generation and editing. Additionally, we show that MILS, being a gradient-free approach, can be used to invert multimodal embeddings into discrete text. This is in contrast prior work (Kazemi et al., 2024) that uses gradient-based inver- sion of embeddings into continuous spaces like images. This ability enables novel applications, e.g. multimodal arith- metic, by inverting multimodal samples into text, combin- ing them, and mapping them back using MIL.S's generation ability. We visualize some of these capabilities in Figure 1.
buing multimodal capabilities into an otherwise blind and deaf LLM.
Generative models have recently gained popularity due to their ability to generalize new tasks, often zero-shot. LLMs (Dubey et al., 2024; Jiang et al., 2023; Team et al., 2024) have emerged the model of choice for discrete input, such as text. Owing to their large size and training on massive corpuses, followed by instruction tuning on high quality data often involving human feedback, such models are powerful tools for a variety of tasks. Approaches such as chain-of-thouugh prompting (Wei et al., 2022; Kojima et al., 2022; Menon al., 2024), and more recently, train- ing the LLM for better reasoning (OpenAI), have further improved their performance on complex math and coding tasks. However, instruction tuning involves training or fine- tuning LLMS for the target task and modality. In contrast, MILS is an inference-time method that does not require any training and optimizes for the solution at runtime. Recent work has even leveraged LLM's reasoning ability iteratively, to solve optimization (Yang al., 2023), prompt refine- ment (Liu al., 2024), and generation (Mañas al., 2024) tasks. However, they are either not evaluated on visual tasks (Yang et al., 2023), or shown to have emergent zero- shot capabilities (Mañas et al., 2024). Another class of generative models involve diffusion (Nichol & Dhariwal, 2021) or fow-matching (Lipman et al., 2022), popular for continuous domains such images (Rombach et al., 2022; Betker et al., 2023; Dai et al., 2023; Saharia et al., 2022) and videos (Polyak et al., 2024; Girdhar et al., 2024; Ho et al., 2022; Blattmann et al., 2023). Such models have
2. Related Work
Multimodal embedding spaces are typically learned by collecting large amounts of paired multimodal data from the internet, often images and text, and learning encoders for each modality using a pairwise similarity objective (Radford al., 2021; Ilharco et al., 2021; Zhai et al., 2023; Li al., 2023a). Such models can further be expanded to additional modalities by collecting text-paired data (Wang al., 2023; Guzhov et al., 2022). or data paired to different modality in that embedding space (Girdhar et al., 2023; Gong et al., 2023). These embeddings have enabled various applica- tions, including zero-shot recognition (Radford al., 2021: Girdhar et al., 2023) open- world object detection (Zhou et al., 2022), and even image generation (Ramesh et al., 2022). We leverage them to compute similarity score across modalities, which helps guide the optimization, im-
2

---

LLMs can see and hear without any training
dramatically improved in media generation capability, often leveraging L.MS too, for better training data captioning, and inference-time prompt rewrites (Betker et al., 2023; Polyak et al., 2024). Finally, gradient-free approaches (Prasad et al., 2022; Guo et al., 2024; Yang et al., 2023) use the reason- ing ability of LLMs to iteratively improve the optimization process.
Test sample
Generated Candidate Sct
GENERATOR
SCORER
Scored Candidate Set
Final Output
Zero-shot multimodal understanding is studied in two forms: zero-shot across data distributions, and emergent zero-shot (Girdhar et al., 2023), where model generalizes to completely new tasks and not just new data. Multimodal variants of popular LLMs (Dubey et al., 2024; Agrawal et al., 2024; Li et al., 2023b) fall in the former bucket, as they are typically trained or tuned on the kind of data seen at test time. Our focus in this work is the latter, as we show MILS generalizes to completely new tasks at test time. Prior work (Tewel et al., 2022; Zeng et al., 2023; 2024; Salewski et al., 2023; Shaharabany et al., 2023) has attempted this setting, however for specific modalities using specialized techniques. MILS, on the other hand, seamlessly general- izes to many different modalities, across understanding and generation tasks.
Until convergence, iuaps
Figure 2: MILS leverages two key modules, GENERATOR and SCORER, to solve multimodal tasks. The GENERATOR will generate a number of text candidates, e.g. captions for image captioning and prompts for T2I, each of which will be scored by the SCORER, and passed back into the GENER as feedback to generate the next batch of text candidates, eventually producing the final output for the input test sample.
SCORER. The goal of the SCORER is to compute a scalar score SER, for the candidates C from the GENERATOR. It takes as input the test sample, along with C, and compares them. A SCORER could be implemented in various different ways. For instance, it could be low-level image processing function comparing textures in two images, or it could be learned machine learning model, such as CLIP (Radford et al., 2021; Ilharco et al., 2021). The SCORER sorts all the candidates based on their score, and returns the top- K can- didates along with scores. Depending on the GENER s capacity (context length), the scorer may return the full list of scores, use an e-greedy strategy to include some low- scoring candidates. In initial experiments, we found greedy top- K to perform the best, hence use that in this work. The output is formatted into the text T, and passed back to the GENERATOR.
3. MILS
We now describe our simple approach for solving multi- modal tasks using MILSS Since is training-free, MIL.S only takes the test sample as input. It relies on two key modules, referred as the GENERATOR and the SCORER. As the names suggest, GENERATOR generates candidate solutions for the task, while the SCORER scores each of those candidates, and sends them back to the GENERATOR to generate an improved candidate set. For certain tasks, this process may be bootstrapped with scores on an initial candidate set. This optimization is run until convergence or a fixed number of iterations, and produces the final solution to the task. Figure illustrates the overall approach.
The goal of the GENERATOR is produce candidate outputs C, that solve given task. It takes as input some text, T, which contains description of the task, along with scores& S (if any) from the SCORER for the previous optimization step. It leverages this signal to produce the next set of candidate generations. The GENERATOR is typically modeled using an LLM, given its ability to take text as input and reason over it. The output, however, is not limited to text. The candidate generations can be used to prompt subsequent model to generate other modalities, such as using a text-to-image (T2I) model like Emu (Dai et al., 2023) to generate images. Some GENERATORS could also use the test sample as input, egg. for tasks like image editing or stylization.
GENERATOR.
Optimization process. MILS searches for the optimal generation under the SCORER'S 's cost function. The opti- mization process is run for N steps, until convergence. Convergence can be defined by the similarity of the can- didate set C over successive steps. Depending on the task, the optimization process can be bootstrapped by an initial candidate set of generations and scoring them. For instance, in case of image captioning, it could simply be a large set of possible image captions from the GENERATOR. For other tasks like T21, one does not need such an initial set.
4. Experiments
We now empirically evaluate MILSS and compare it to exist- ing approaches on some of the multimodal understanding and generation tasks enabled by it. For each of the down-
3

---

LLMs can see and hear without any training
Method
Training Data
CDD
METEOR
Nagram al. (Nagrani et al. 2022) Nagrami (Nagaui all., 2022)
HowWTo00M (Miech et al. 2019)
0.5
8:23
VideoCCCM (Nagrasi al. 8.2
MILS
Table 2: Zero-shot video captioning MSR- VTT (Xu et al., 2016). MILS outperforms (Nagrani et al., 2022) when trained on HowTol00M, and is competitive to it when trained on the much cleaner VideoCC3M dataset, outper- forming it on METEOR. We grayed (Nagrami et al., 2022) since it is trained for video captioning. while MILS not.
Baston Terrier's is sharp
Close-ap a bird near a ship's cabin window.
Cluttered desk with a large monitor and window.
teeth firmly gripping rope
toy lightl.
NlaCe MaiCe/eerre ao 2000
The image depicts that dog licking mun who inappropriately close praximity.
The image depicts that observation boot water below us," lie said.
The image depicts that workspace during on Clutered desk with a large monitor and unline career as consulant.
Figure 3: Image Captioning using MILSS compared to existing state-of-the-ard zero-shot approach, MeaCap (Zeng al., 2024). MILS, while being a much simpler approach, produces more accurate and syntactically correct captions to the image.
GENERATOR. We use the Llama 3.1 8B (Dubey al., 2024) LLM as the core generation module. We generate an initial list of 30K prompts that we use to bootstrap the opti- mization process. To ensure diversity in this initial set, we prompt the LLM with different object categories to generate a list of prompts, and combine them, similar to (Gandels- man et al., 2024). Then for each optimization step, we keep the top-50 highest scoring generations from the SCORER, and convert them into a textual prompt. The prompt used is described in Section B. We run the optimization process for 10 steps.
Method
BLEU,
CIDEI
METEOR
SPICE
ZeroCap (Tewel al., 2022)
2.6
14.6
11.5
5.5
ConZIC (Zenged et al., 2023)
1.3
13.3
11.2
5.0
CLIPRe (Li et al., 2023c)
4.6
25.6
13.3
9.2
McaCapyr (Zeng et al., 2024)
7.1
42.5
16.6
11.8
MeuCaphy (Zeng et al,, 2024)
4.5
26.0
14.1
9.4
SCORER. We score the candidate captions against the test image using an image-text similarity model, SigL.P (Zhai et al., 2023). Note that unlike image captioning models that leverage curated image-text pairs (Karpathy & Fei-Fei, 2015), SigL.P, by itself is not capable of captioning (Shen et al., 2022). Nevertheless, combined with MILS, it can serve as an effective captioner, as shown next.
MILS
8.0
33,3
15.0
9.6
Table I: Zero-shot image captioning MSCOCO (Karpa- thy & Fei-Fei, 2015). Despite being far simpler than existing approaches, MILS performs competitively on all automatic metrics, and especially METEOR and SPICE which take into account the semantic similarity. "refers to results we obtained by running the provided code.
Benchmarks and Metrics. We evaluate MILS on the MSCOCO captioning test set (Karpathy Fei-Fei, 2015). It consists of 5,000 images sampled from the MSCOCO dataset (Lin et al., 2014). We use the standard suite of cap- tioning evaluation metrics, including BLEU (Papineni et al., 2002), METEOR (Banerjee & Lavie, 2005), CIDEr (Vedan- tam et al., 2015) and SPICE (Anderson et al., 2016). We focus our attention on the METEOR and SPICE metrics, as those take into account semantic similarity rather than exact word match, and are better correlated with human prefer- ence (Anderson et al., 2016). This is particularly important for emergent zero-shot approaches like MILS, which are not trained to learn the vocabulary used in given benchmark or modality.
stream applications, we describe the GENERATOR, SCORER, benchmarks and evaluation setup, followed by the key re- sults. Finally in Section 4.7 we ablate the various design choices in MILS.
Note that MILS is a test-time optimization method and ex- hibits emergent zero-shot behavior (Girdhar et al., 2023), generalizing not only to a new test data distribution, but to the new task and modality itself. This is contrast most existing zero-shot work that typically needs task/modality- specific data curation or training. Since most prior work is of the latter type, it is hard to perform fair comparisons. Never- theless, we compare to the closest zero-shot approaches and show that MILS is competitive or better, even compared to methods tuned for that specific task or modality.
Results. We compare MILS to existing baselines in Ta- ble 1. Some of the basclines, such as ZeroCap (Tewel et al., 2022), also leverage language models in conjunction with a CLIP-like model. However, they propose a gradient based optimization process to search for the optimal next token given a current generation. Other approaches like Mea- Cap (Zeng et al., 2024) filter key concepts from a memory module, and leverage a number of text and multimodal en- coders in a multi-step process to produce the caption. In
4.1. Image Captioning
We start with the fundamental image understanding task of producing a textual caption for given image.
4

---

LLMs can see and hear without any training
Method
BLEU,
ROUGEL
METEOR
SPICE
Quality
Text Faithfulness
ZerAuCapp (Salewski et al., 2023)
2.9
25.4
9.4
5.3
LDM + MILS
75.0
LDM
MILS
2.7
23.1
12.4
7.6
FLUX MILS
65.8
FLUX
02.0
Table 3: Zero-shot audio captioning on Clotho (Drossos ct al., 2020) dataset. MILS performs competitively to the existing zero-shot audio captioning approach ZerAuCap, even outperforming it on semantics-aware metries like ME- THOR and SPICE, while being simpler and applicable to many other modalities and tasks.
0
25
50
75
100
MILS Win Rate Percentage
Figure 4: Improved text-to-image (T2I) generation using MILS. We apply MILS to two of the latest, state-of-the-art T2I models, a latent diffusion model (LDM), and FLUX.1 [schnell] (FLUX). We compare MILS's outputs the gen- erations from the initial models using human annotators. Evaluated over the 200 prompt DrawBench dataset, the annotators clearly preferred MILS's generations on both overall quality and text faithfulness, across both models.
contrast, MILS is simpler conceptually and to implement, while obtaining better results. We also show examples of the captions generated using MILS, and compare them to McaCap in Figure 3. MILS, without ever having seen any captioning data or having captioning-specifn training is able to generate faithful and syntactically correct captions. This result also showcases the strong reasoning capabilities of the GENERATOR, which is able to correctly modify the captions for future iterations.
4.3. Audio Captioning
Similar to videos, MILS transfers seamlessly to audio cap- tioning as well. We use the same GENERATOR as in Sec- tion 4.1, along with 50K initial audio prompts generated using an LLM (cf. Section B). As the SCORER, we use the ImageBind (Girdhar et al., 2023) model, that maps multiple modalities, including audio and text, to a shared embed- ding space. We evaluate our approach on a popular audio captioning dataset, Clotho (Drossos et al., 2020). We use automatic captioning metrics as used in prior work, and described in Section 4.1. We report our performance in Ta- ble 3. MIL.S obtains strong performance vs. comparable zero-shot approach, ZerAuCaps (Salewski et al., 2023), out- performing it especially on semantics- aware metrics like METEOR and SPICE. While other approaches for audio captioning have been proposed, they require training on audio-caption data (Kong et al., 2024). See Section C for qualitative results.
4.2. Video Captioning
Owing to its simplicity and versatility, MILS seamlessly transfers to videos without major changes. We use the same GENERATOR as described for image captioning in Sec- tion 4.1, along with the same initial prompt set. For SCORER, we use a ViCLIP (Wang et al., 2023) ViT-L/14 model that operates on 8 frames from the video, and re- turns similarity score between the video and the caption. We experiment the MSR-VTT (Xu et al., 2016) test set, which contains 2,990 videos, each between 10 to 30 seconds long. We report our results in Table 2. Since most prior work in vidco captioning leverages video-caption training data, we compare MILS to (Nagrani et al., 2022), which learns a languael model on the HowTo100k (Miech et al., 2019) or the VideoCC3M (Nagrani al., 2022) datasets, and reports performance on MSR-VTT. We use the CIDEr (Vedantam et al., 2015) and METEOR (Banerjee & Lavie, 2005) metrics as reported in the prior work. We find that MILS, in spite of never having been trained for video captioning, outperforms (Nagrani et al., 2022) trained on HowTo100M on both metries. Even when compared to the same model trained on the much cleaner VideoCC3M data, MILS outperforms it on the semantics-aware METEOR metric. This difference in performance of the baseline in the two settings shows the importance of training data for video captioning models. MILS, being competitive with these without needing any video captioning training, is very promising. We show qualitative results in the Section C.
4.4. High-Quality Image Generation
As mentioned earlier, MILS not limited to multimodal understanding tasks discussed so far. We now describe how MILS can be used for multimodal generation tasks, starting with improving text-to-image (T2I) generation models.
GENERATOR. To generate high quality images, we chain an LLM to a '21 model. Specifically, we experiment with two state-of-the-eeeeeee the-rrt models, Latent Diffusion Model (LDM) (Rombach et al., 2022) and FLUX. [schnell] (Labs). The goal of the LLM to "rewrite" the prompt passed into the T2I model, such that the final generated image improves in quality, while maintaining or improving the faithfulness to the original text prompt. Note that this CennerATOR does not need an initial prompt set to bootstrap from.
We score the generations using PickScore (Kirstain et al., 2023). PickScore is a
SCORER.
5

---

LLMs can see and hear without any training
Pronee
trianguler purple flnner por. A purple flover por in the shape of triangle
Frov
Applionce comportuent which artiicially kept cool store food drink.
Birds
chirping
People
talking
1 MIIL
Figure 7: MILS enables mross- arithmetic by in- verting modalities into text, combining them, and mapping them back to an image.
Figure 5: Improving image generation using MILS. Ap- plying MILS to GENERATOR using the same base model, a Latent Diffusion Model (LDM) in this case, leads to much higher quality images. We show the original input prompt, the generation from the base model, and from MILS.
2024), we evaluate the generations two axes: quality or visual appeal, and the faithfulness to the input text. For each axis, we set up a web interface where an annotator is shown two images, generated either by a baseline model, or en- hanced using MILS. When evaluating for text faithfulness, the annotator is also shown the original text prompt. The annotator has to pick which image they prefer. Each image is annotated by three annotators and we the majority vote over the annotators to compute win for each model. See Section A for full human evaluation details.
Test sample
Style Image
Output
Results. We summarize the results of our human study in Figure 4. As seen by the win-rates, human annotators clearly prefer generations enhanced by MILS over the gen- erations from the base model itself. We also show qualitative comparisons in Figure 5, where the improvement in acs- thetic quality using MILS is clearly apparent. We find that MILS is able to simplify complex prompts and add aesthetic details, that improve the overall quality and faithfulness of the generations. Recent work has shown that LLM-based prompt rewrites (Betker et al., 2023; Polyak et al., 2024) can improve media generation performance. However, they require laborious process of manually trying various differ- ent rewrite prompts, until one finds rewrite that works well with that model, across all test cases. MILS can automate and complement that process, either by prompt-engineering each generation, or proposing candidate rewrites for an ex- pert prompt engineer to improve upon.
Figure 6: Style Transfer. Using Gram Matrix dis- tance (Gatys, 2015) the SCORER, MILS can discover the edit prompt required to apply a given style to an image.
CLIP-style model takes as input an image and a text prompt, and predicts the likelihood of that image be preferred by humans for that prompt. We score each of the GENERATOR outputs using PickScore along with the input prompt, and return the scores for each of the generations. Rest of the process proceeds the same as before.
We use the DrawBench prompt set from Imagen (Saharia al., 2022), that contains 200 textual prompts, evaluate our generations. Due to the noisiness of automatic metrics for media generation (Gird- har et al., 2024; Ge et al., 2024; Jayasumana et al., 2024), we perform our evaluations using human annotators from Amazon Mechanical Turk. We follow the JUICE frame- work (Girdhar et al., 2024). In-keeping with the standard practice in media generation (Dai al., 2023; Girdhar et al.,
Benchmarks and Metrics.
Note that this capability is not easily afforded by prior work compared to on earlier tasks (Tewel et al., 2022: Salewski et al., 2023). Those methods would require computing gradients through a multi-step diffusion process to estimate which tokens the LLM should produce next. MILS, being a gradient- free optimization approach, easily enables such diverse applications within a simple framework.
6

---

LLMs can see and hear without any training
the test sample, or the style image!
4.6. Cross-Modal Arithmetic
Finally, we explore an interesting application enabled by MILS. Unlike prior work (Kazemi al., 2024) that maps embeddings to continuous image space, gradient-free approach in MILS enables inverting such embeddings into the discrete text space instead. This is also exemplified by our results in Sections 4.1 to 4.3. This enables an interesting application cross-modee arithmetic. We take inspiration from ImageBind (Girdhar et al., 2023), which mapped mul- tiple different modalities into the image embedding space. Using this shared embedding, authors were able combine modalities, and generate retrieve images given that com- bination. MILS, in fact, is even more flexible, as inversion to text enables interfacing with many more models. For instance, ImageBind showed results on audio to image gen- eration by leveraging a DALLE-2-like T2I model (Ramesh et al., 2022). This was possible as ImageBind happened to be aligned to the CLIP embedding space, same what was used in DALLE-2. As such, ImageBind was not compat- ible with any other T21 model, such model (Rombach et al., 2022). A textual representation, on the other hand, would work with any T21 model, includ- ing those that do not represent textual input as a point on an embedding space. In Figure 7, we show examples of combining image and audio modalities. We first invert both image and audio into text using Section 4.1 and Section 4.3 respectively, combine the two outputs using an LLM (details in the Section B), and finally convert the prompt into high quality image as described in Section 4.4. The resulting generated image combines the semantic concepts from both those modalities.
Photo taken with a computer the background.
Picture taken in a country setting.
4 Photograph of cat sitting on a monitor.
Photo of a cat in a farm setting with flowers. Photo of a cat in a farm
Cat's face centered on a monitor with a nearby
setting with a garden
keyboard.
bench.
10
Feline sitting on a keyboard with a nearby monitor and laptop.
Photo of cat in garden
bench with a farm backdrop.
Figure 8: Image captioning over steps. This shows the captions generated by MIL.S different steps (#) of opti- mization process, getting progressively more accurate.
latent diffusion
4.5. Style Transfer
Going beyond image generation, MILS can also be applied image editing tasks. Here we specifically consider the task of style transfer, where given test image and style image, the goal is to generate an image that contains the content from the test image, in the style of the style image.
GENERATOR. Similar to Section 4.4, we implement the GENERATOR by chaining the output of an LLM to an image generation model. Different from Section 4.4, since we want generate the same content as in the test sample, the GENERATOR also takes the test sample as an input. Hence, we use an image editing model (Sheynin et al., 2024) as the image generation module. It produces the stylized image given the test sample and the edit prompt from the LLM.
4.7. Ablations
We now ablate some key design choices in MILS. We pri- marily focus on the image captioning task for most of the analysis, and improved image generation for some of the ablations. For computational ease, randomly sample 1000 images from MSCOCO for captioning, and use the 200 prompt DrawBench set for image generation, as the test set for this analysis. We report all metrics including CLIP similarity and PickScore averaged over these sets.
SCORER. To measure the quality of the style transfer, we simple approach to estimate the similarity of colors and textures in the generated image compared to the style image. We use the distance between Gram matrices of the image features, as proposed in (Gatys, 2015). We com- pute this distance over features from different layers of a VGG19 (Simonyan & Zisserman, 2015) CNN, where the lower layers ensure stylistic faithfulness, and the higher lay- ers ensure content faithfulness. We use MILS to minimize both the style and content losses.
Performance over optimization steps. We evaluate this for both the tasks in Figure 9. We report both the SCORER out- put, which can be thought of a "training loss" in our setup, as well as a downstream metric. We report SPICE (Ander- son et al., 2016) for image captioning, and human evaluation against the original prompt's generation for T2I. For the lat- ter, we also show #4 point error bars, which we found to be the typical random variance margins in human evaluations. As Figure 9 shows, both the SCORER outputs and the down-
Results. Figure 6 shows some sample style transfer re- sults. MILS generalizes to this novel task completely zero- shot and produces accurately stylized images. Note that it achieves such edits not only without any training. but also without the LLM actually seeing any features from either

---

LLMs can see and hear without any training
CLIP CLIPsinilan similarity
PickScore
0.23
Human Eval
CLIP*
0.15
0.1
SPICI
0.22
70
0.1
CLIP Similurity
PickScore Human Eval
SPICE
0.22
50
1
2
3
4
9
10
2
4
6
8
10
12 14
16
18
10a
10a
104
Steps
Steps
Initial set size
Figure 9: Performance with number of optimization steps for captioning (left) and generation (right). We show the optimization metric, i.e. SCORER's output (CLIP Similarity and PickScore), and the downstream metric (SPICE (Anderson al., 2016) and human evaluation win% for quality), respectively. Both tend improve over optimization steps, and correlate with each other.
Figure 10: Performance improves with size of initial set, suggesting is crucial bootstrapping the GEN-
FRATOR. CLIP": CLIP similarity at convergence.
sheep stumning, natural landscape with a few mountoins
car white sheep standing stanning, rural landscape red barn, a trees, and birds.
Prompx rewrite
sheep.
sheep standing
sheep in stunning. natural landscape with few mountains, few wildflowers, and lake.
sheep stumning. natural landscape with few mountains, a wildflowers, and lake.
stanning, rural landscape with barn, few trees, and few birds.
Generated Imuge
Figure 11: Improved image generation optimization steps. The quality of the output improves the optimization
steps (#f). We also show the prompt being produced by the LLM in the GENERATOR, which is passed to the T21 model.
9.5
9.2
performance. This suggests that the initial bootstrapping set critical to ensure the GENERATOR can produce suffi- ciently diverse candidates and avoid local minimas.
SPICE
SPICE
8.6
8.5
Size of the GENERATOR and SCORER. We evaluate the effect of the size (in parameters) of the GENERATOR (Llama 3) and SCORER (MetaCLIP (Xu et al., 2024)) in Figure 12, for image captioning. We find larger models generally performed better, with LLM scaling exhibiting the most promising gains. We also experiment with different kinds of GENERATORS and SCORERS the Section C.
3
8B
GENERATOR Parameters
SCORER Parameters
Figure 12: Impact of GENERATOR (left) and SCORER (right) size. We report the downstream metric SPICE for both these ablations since CLIP similarity may not be comparable across model sizes. For the GENERATOR and SCORER we use different sized Llama or MetaCLIP models. As evident from the graphs, larger models perform better.
5. Conclusion and Future Work
We have presented MILS, simple approach for solving multimodal tasks without requiring any task specific data curation or training. MILS exhitbits emergent zero-shot generalization to various different tasks and modalities. No- tably, we show MIL.S obtains strong performance on cap- tioning across three modalities: images, videos and audio, showing that LLMs This further leads to improving and enabling various media generation tasks, such as image generation, image editing (style transfer) and cross-modal arithmetic.
stream metrics improve over optimization steps, converging after 10 to 20 steps. We also note that the optimization objective (ScORER output) correlates well with downstream performance. Lastly, we visualize this result qualitatively in Figures and 11 for captioning and generation respec- tively. In both cases, the quality of the output improves over the steps, showing the effectiveness of MIL.S.
and hear without any training!
Impact of the initial candidate set. We evaluate this in Fig- ure 10. We subsample the initial set to different sizes, and see strong positive correlation between that and the final
8

---

LLMs can see and hear without any training
While promising, MILS has some limitations that future work can attempt address. Its performance is bounded by the ability of the GENERATOR to generate diverse candi- dates, and the SCORER to provide accurate feedback to the GENERATOR. For instance, style transfer performance is limited by the resolution of Gram matrix distance in detect- ing fine-grained texture similarities, and the LLM's ability to describe potential styles. As the LLMs and the multi- modal models continue to improve (OpenAI; Fang al., 2024), MILS would improve with it. Another limitation is the speed of the optimization process. This would improve as the core LLMs become faster and more efficient, and as their context length (Munkhdalai et al., 2024) and reasoning abilities (OpenAI) improve, requiring fewer optimization steps. It would also be interesting apply MILS other modalities and tasks, such as for spatial and 3D tasks.
S. W., Fidler, S., and Kreis, K. Align your latents: High- resolution video synthesis with latent diffusion models. In CVPR, 2023.
Dai, X., Hou, J., Ma, C.-Y., Tsai, S., Wang. J., Wang. R., Zhang, P., Vandenhende, S., Wang, X., Dubey, A., et al. Emu: Enhancing image generation models us- ing photogenic needles a haystack. arXiv preprint arXiv:2309. 15807, 2023.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li. K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Drossos, K., Lipping, S., and Virtanen, T. Clotho: An audio captioning dataset. In ICASSP. 2020.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle,
Impact Statement
A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint 2407 2024.
Our work builds on top of LLMs and other multimodal models, and as such, the strengths and weaknesses of such models would be reflected when used with MILS. This in- cludes the biases present in such models. However, MILS being training-free approach, does not learn any new bi- and can easily be upgraded with better and less- LLMs or multimodal models as they become available, with- out needing any re-training. While we do not foresee any negative societal impact directly enabled by MILS, any real-world applications should consider additional safety evaluations before deployment.
Fang, A., Jose, A. M., Jain, A., Schmidt, L., Toshev, A., and Shankar, Data filtering networks. In ICLR, 2024.
Gandelsman, Y., Efros, A. A., and Steinhardt, J. Interpreting the second-order effects of neurons in clip. arXiv preprint arXiv:2406.0434 2024.
Gatys, L. A. A neural algorithm of artistic style. arXiv preprint arXiv: 555.00007666 2015.
Acknowledgements: Authors would like to thank Xi Yin, Saketh Rambhatla, and the entire Meta Al team for many helpful discussions.
Ge, S., Mahapatra, A., Parmar, G., Zhu, J.-Y., and Huang,
J.-B. On the content bias in fréchet video distance. In CVPR, 2024.
Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, 2017.
References
Agrawal, P., Antoniak, S., Hanna, E. B., Chaplot, D., Chud- novsky, J., Garg, S., Gervet, T,, Ghosh, S., Héliou, A., Jacob, P. et al. Pixtral 12b. arXiv preprint arXiv:2410.07003, 2024.
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Misra, ImageBind: One embedding space to bind them all. In CVPR, 2023.
Anderson, P., Fernando, B., Johnson, M., and Gould, S. SPICE: Semantic propositional image caption evaluation. In ECCV, 2016.
Girdhar, R., Singh, M., Brown, A., Duval, Q. Azadi, S., Rambhatla. S.. Shah, A., Yin, X., Parikh, D., and Misra, I. Emu video: Factorizing text-to-video generation by explicit image conditioning. In ECCV, 2024.
Banerjee, and Lavie, A. METEOR: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL, 2005.
Gong, X., Mohan, S., Dhingra, N., Bazin, J.-C., Li, Y., Wang, Z., and Ranjan, R. Mmg-ego4d: Multimodal generalization in egocentric action recognition. In CVPR, 2023.
Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., Manassra, W., Dhariwal, P., Chu, C., and Jiao, Y. Improving image generation with better captions. 2023.
Guo, P.-F., Chen, Y.-H., Tsai, Y.D.,, and Lin, S.-D. Towards optimizing with large language models, 2024.
Blatmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim,

---

LLMs can see and hear without any training
Guzhov, A., Raue, F., Hees, J., and Dengel, A. Audioclip: Extending clip to image, text and audio. In ICASSP, 2022.
Li, w., Zhu, L., Wen, L.,, and Yang, Y. DeCap: Decod- ing CLIP latents for zero-shot captioning via text-only training. In ICLR, 2023c.
Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300, 2022.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In ECCV, 2014.
Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv:2210002747. 2022.
Iharrco, G., Wortsman, M., Wightman, R., Gordon, C.,, Car- lini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H.,
Miller, J. Hajishirzi, H., Farhadi, A., and Schmidt, L. Openclip. 2021.
Liu, S., Yu, S., Lin, Z., Pathak, D., and Ramanan, D. Language models as black-box optimizers for vision- language models. In CVPR, 2024.
Jayasumana, S., Ramalingam, S., Veit, A., Glasner, D.,
Mañas, O., Astolfi, P., Hall, M., Ross, C., Urbanek, J., William,, A., Agrawal, A., Romero-Soriano, A., and Drozdzal, M. Improving text-to-image consisteney via automatic prompt optimization. arXiv preprint 1.Xi::2403.17044 2024.
Chakrabarti, A., and Kumar, S. Rethinking fid: Towards better evaluation metric for image generation. In CVPR, 2024.
Jiang, A., Sablayrolles, A., Mensch, A., Bamford, C., Singh Chaplot, D., de las Casas, D., Bressand, F., Lengyel, G., Lample. G., Saulnier, L., et al. Mistral 7b, arxiv. arXiv 2023.
Menon, S., Zemel, R., and Vondrick, C. Whiteboard-o thought: Thinking step-by-step across modalities. arXiv preprint arXiv:2406. 000.406.4445000 /4562, 2024.
preprint 23
Karpathy, A. and Fei-Fei. Deep visual-semantic align- ments for generating image descriptions. In CVPR, 2015.
Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., and Sivic, HowTo Learning a Text-Video Em- bedding by Watching Hundred Million Narrated Video Clips. In ICCV, 2019.
Kazemi, H., Chegini, A., Geiping, J., Feizi, S., and Gold- stein, T. What do learn from inverting clip models? arXiv preprint ar 4rX:v:203..02580, 2024.
Munkhdalai, T., Faruqui, M., and Gopal, S. Leave no con- text behind: Efficient infinite context transformers with infini-attention. arXiv preprint arXiv:2404.07145. 2024.
Kirstain, Y., Polyak, A., Singer, U., Matiana, S., Penna, J., and Levy, O. Pick-a-pic: An open dataset of user preferences for text-to-image generation. NeurlPS, 2023.
Nagrani, A., Seo, P. H., Seybold, B., Hauth, A., Manen, S., Sun, C., and Schmid, C. Learning audio-video modalities from image captions. In ECCV, 2022.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models zero-shot reasoners. NeurlPS, 2022.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In ICML, 2021.
Kong, 7., Goel, A., Badlani, R., Ping, W., Valle, R., and Catanzaro, B. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. In ICML, 2024.
OpenAI.
Introducing openai ol-preview.
https://openaa cwm/indewe introducing -ewpeiiiiino--nni---eee oedenii-o-i www-oooleppe...ooooe cessed: 2024-10-01.
Ac-
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. BLEU:
Labs, B. F. Flux. https://huggingeee co htt/bback-foosst--eeeee Accessed: 2024-10-01.
A method for automatic evaluation of machine translation. ACL, 2002.
FT:FPPLXX.---1111 1-schnell.
Polyak, A., Zohar, A., Brown, A., Tjandra, A., Sinha, A., Lee, A.. Vyas, A., Shi, B., Ma, C.-Y.. Chuang, C.-Y., et al. Movie gen: A cast of media foundation models. arXiv 2024.
J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023a.
preprint arXiv:2410.
Li, L., Yin, Y., Li, S., Chen, L., Wang, P., Ren, S., Li, M., Yang, Y., Xu, J., Sun, X., et al. A large-scale dataset to- wards multi modal multilingual instruction tuning. arXiv preprint Xiv::2306. 2306.043877 2023b.
Prasad, A., Hase, P., Zhou, X., and Bansal, M. Grips: Gradient-free edit-based instruction search for prompting large language models. arXiv preprint arXiv:2003.07281, 2022.
10

---

LLMs can see and hear without any training
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, Learning transferable visual models from natural language supervision. In ICML,, 2021.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F, Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022.
Xu, H., Xie, S., Tan, X. E., Huang, P.-Y,, Howes, R., Sharma, V., Li, S.-W., Ghosh, G., Zettlemoyer, I.., and Feichten- hofer, C. Demystifying clip data. In ICLR, 2024.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C.,, and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint ar arXiv:2004.06122 2022.
Xu, J., Mei, T., Yao, T., and Rui, MSR-VTT: A large video description dataset for bridging video and language. In CVPR, 2016.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models as optimizers. In ICLR, 2023.
Saharia, C., Chan, W., Saxena, S., Li, L.,, Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M. Photorealistic texi-to-image diffusion models with deep language understanding. Neur/PS, 2022.
Zeng, Z., Zhang, H., Lu, R., Wang, D., Chen, B., and Wang, Z. ConZIC: Controllable zero-shot image captioning by sampling-base polishing. CVPR, 2023.
Salewski, L., Fauth, S., Koepke, A., and Akata, Zero-shot audio captioning with audio-language model guidance and audio context keywords. In NeurIPS Workshops, 2023.
Zeng, Z.., Xie, Y., Zhang, H., Chen, C., Chen, B., and Wang, Z. Meacap: augmened zero-shot image cap- tioning. In CVPR, 2024.
Shaharabany, T., Shaulov, A., and Wolf, L.. Zero-shot au- dio captioning via audibility guidance. arXiv preprint 230.0004409 2023.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In ICCV, 2023.
Shen, S., Li, L. H., Tan, H., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., and Keutzer, K. How much can clip benefit vision-and- lannaaugde tasks? In ICLR, 2022.
Zhou, X., Girdhar, R., Joulin, A., Krähenbühh, P., and Misra, I. Detecting twenty-thousand thousand classes using image-level supervision. In ECCV, 2022.
Sheynin, S., Polyak, A., Singer, U., Kirstain, Y., Zohar, A., Ashual, O., Parikh, D., and Taigman, Y. Emu edit: Precise image editing via recognition and generation tasks. In CVPR, 2024.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprini arXiv:2403.08290 2024.
Tewel, Y., Shalev, Y., Schwartz, I., and Wolf, L. Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic. In CVPR, 2022.
Vedantam, R., Lawrence Zitnick, C., and Parikh, D. CIDEr: Consensus- based image description evaluation, In CVPR, 2015.
Wang, Y., L. Yu, J., Ma, X., Li, X., Chen, G., Chen, X., Wang, Y., et al. Internvid: A large- scale video-text dataset for multimodal understanding and generation, In ICLR, 2023.

---

LLMs can see and hear without any training
A. Human evaluation Setup
why, and don't combine two different concepts (with 'or' or 'and"), and keep it short 15-20 words.
We use Amazon Mechanical Turk (AMT) for human eval- uation, with a setup similar to (Girdhar et al., 2024). As discussed in Section 4.4, we evaluate the enhanced image generation two axes: quality and faithfulness the text prompt. Screenshots of the evaluation interface are shown in Figure 13. The annotators are also given elaborate in- structions with examples, as shown in Figures 14 and 15 for the faithfulness and quality evaluations respectively. We give each sample to three annotators and use majority vote to choose the winning model.
where Jabel] is the AudioSet class name. The above prompt similar to the prompt used in image and video captioning, adapted to mention the need for focus on audio. This process yields candidate set of about 50, 000 captions.
B.2. GENERATOR prompts
We use the following prompts in the GENERATOR. Here, (descriptions) denote the list of previous generations, along with the score for each of the generation from the SCORER. The scores are sorted, and we provide one score and text per row. In high-quality image generation, we also provide an additional {init description) which is the original text prompt. (requested number) is the number of new genera- tions that we ask from the GENERATOR.
B. LLM prompts
B.1. Generating the initial candidate set
We bootstrap captioning tasks with an initial candidate set. We use the following strategies to create this initial candidate set.
Image Captioning:
Image and video captioning: We use the initial candidate set as generated in (Gandelsman et al., 2024). See Table in their Appendix for the exact prompt. The overall idea is to take class labels from ImageNet (Deng et al., 2009) and prompt the LLM to generate 40 candidate captions based on the concept of the chosen class label. This process yields a set of around 30, 000 captions. This same set is used for the video captioning task.
You need to provide a short image description. am providing to you a list of short image descrip- tions and scores. Higher score means that the image description characterizes the image better: [descriptions]
Generate additional (requested number) short im- age descriptions that you think that will maximize the score and fully capture the image. Be concrete and try to find elements that are unique to this image. You can introduce new elements to the descriptions, combine unique elements and objects from provided descriptions to form new descriptions, rephrase in- dividual descriptions, drop elements, or simplify de- scriptions. Be creative and don't be afraid to come up with erroneous descriptions. Put each description in a different line, with a counter at the beginning (e.g. "1. "I....", and try to keep them very short (up to 10 words).
Audio captioning. Since audio captions typically describe audible information, opposed to visual information, we recreate the candidate set based on the above protocol with small modifications. We use class labels from Au- dioSet (Gemmeke et al., 2017) and generate 50 captions per audio label. We use Llama 3.1 and provide the following prompt:
Generate 50 diverse descriptive captions for an audio clip that features the sound of {class label]. Write concise and vivid description of what can be heard in the clip, using complete sentences. For example:
1. car drives by with its horn honking.
Video captioning:
2. Children are playing and laughing in park.
3. Heavy rain falls on pavement and roofs.
You need provide a short video description. I am providing to you a list of short video descrip- tions and scores. Higher score means that the video description characterizes the video better: {descriptions]
4. A crowd cheers and applauds at a sports event.
Write the generation as if a person would write that after listening to the audio clip. Do not mention concepts that cannot be heard, like sunshine, star, any color or taste. Try to capture the main sounds and any background or accompanying noises in your caption, without referencing the fact that you're 're lis- tening to audio clip. Simply describe what can be heard. Put each description in a different line, with counter at the beginning (e.g. '1. ...), don't 't explain
Generate additional (requested_ deseriptions that you think that will maximize the score and fully capture the video. Be conerete and try to find elements that are unique to this video. You can introduce new elements to the descriptions, combine unique elements and objects from provided
short video

---

LLMs can see and hear without any training
PLEASE ANSWER BOTH QUESTIONS BELOW.
PLEASE ANSWER BOTH QUESTIONS BELOW.
Watch the tollowing two images and select the one that you think ALIGNS more with the given text
Thesk witu dudeo mnseno may Now inge Watch the following sharpness, objoct/scane
prompt according text haitifulness:
select the according (pixel
Text: An Illustration small green elephant standing behind large red mouse.
concapta):
Image 1;
Image 2:
Image
Image
Text to Image
Question Which
Image
Image O Image
Qusation 2.. Can us think this mage better? Plaas select factios that considercd cd making dodiion.
Question 2
you tell why you Image
considered makiing your dacsioo.
combinatioo))
Plae Shargness
Object Scans Condistency
Recogeizable Objects, Conceptie and Scones
Figure 13: Human evaluation question form for faithfulness (left) and quality (right). We ask two questions. The first question asks which of the two images the rater prefers. The next question elaborates on the first response by asking the rater to the reason for their preference, based on JUICE (Girdhar et al., 2024) evaluation protocol.
Audio captioning:
test: bear drisieg string pood bed
bott drtinng can diiiigggg eac (tadacas
You need to provide a short audio description. I am providing to you a list of audio descriptions and scores. Higher score means that the audio descrip- tion characterizes the audio better: {descriptions)
DUXXS Ul calle or hetthiuteee the mmage
that remaliatiin
Generate additional {requested number) short au- dio descriptions that you think that will maximize the score and fully capture the audio. Be concrete and try to find elements that are unique to this audio. You can introduce new elements to the descriptions, combine unique elements and objects from provided descriptions form new descriptions, rephrase in- dividual descriptions, drop elements, simplify de- scriptions. Be creative and don't be afraid come up with erroneous descriptions. Put each description in a different line, with a counter at the beginning (e.g. "I....", and try to keep them short (under 20 words).
test: flancr imag fancy and
tosts femoy green rot
Figure 14: High-qualite image generation faithfulness examples. We provide these examples to the annotators to judge the faithfulness. The generated image should have the correct object, color, texture or scene layout.
High-quality image generation:
descriptions form new descriptions, rephrase in- dividual descriptions, drop elements, simplify de- scriptions. Be creative and don't be afraid to come up with erroneous descriptions. Put each description in different line, with counter at the beginning (c.g."1. "I.,,, and try to keep them short up to 20-25 words.
You need to expand and rephrase the provided de- scription for image generation to make the best im- age, by maximizing the image score: The description is: desecritsciiiinn
Here are some example rephrases and the corre- sponding image scores:
2

---

LLMs can see and hear without any training
Style transfer:
Aspoct
Explanation
Good Exampec
Exompie
Viayaese hight uargared
Compured la the odter
You need to generate instructions for image editing that minimize pair of scores: am providing you a list of example editing instructions and their pairs of scores.
sharpnese with on raggl.
Sheraeens
[descriptions}
Generate additional (requested editing in- structions. Be concrete and come up with different instructions with various guesses for the possible edits that will minimize both of the scores. You can introduce new styles the instructions, combine unique styles and textures from provided instruc- tions to form new instructions, rephrase individual instructions, drop instruction parts, or simplify them. Be creative and don't t be afraid to come up with er- roncous editing instructions. Put each instruction a different line, with a counter at the beginning (e.g. "1, ".....") and keep them short (less than 50 words).
soodeed shaps:
loya
Onjeet/
the lmags, gred conailency
Condimmancy appairenes, teds lncedit or onler
beample where the vitu uncentard one on the luft
concgg urcenteed hanes
Recrggstrato Vieaal
oujecis
bith mat and vibo the compared shew ciage
Scanes
Cross-modal arithmetic:
Figure 15: High-quality image generation quality exam- ples. We provide these examples to the annotators to judge quality. The resulting image should have high pixel sharpness, object and scene consistency, and the concepts should be recognizable.
have image description and an audio description that I want to combine together into a text descrip- tion that will help an Al imagine that scene. As an example, if the caption says "Crane on grass" and the audio says "An ocean with the waves crashing on shore" then you need generate a text description like "Crane beside the shore with waves coming". The combinations can be imaginative and not neces- sarily true in real world. Here are the captions and the audio description:
Model
Training data
SPICE
CLIP
WIT (Radford et al., 2021)
8.5
SigLIP
WebLI (Zhai et al., 2023)
9.7
MetaCLIP
MetaCLIP 400M (Xu et al., 2024)
9.2
DFN
DFN2B (Fang et al., 2024)
9.3
Image caption: {image_caption)
Table 4: Type of SCORER. All the models are based on ViT-L/14 with different training data.
Audio caption: faudio caation))
Generate the combined caption in single sentence.
(descriptions)
C. Additional Results and A Ablations
Generate additional drequested_number descrip- tions that will maximize the score. Be concrete and come up with different descriptions with various guesses for the possible way to rephrase and expand it, in a way that will maximize the score. You can introduce new elements to the descriptions, combine unique elements and phrasings from provided de- scriptions to form new ones, drop description parts, simplify them. Be creative and don't be afraid come up with erroneous descriptions. Put each instruction in different line, with a counter at the beginning (e.g. "1. ..."), and keep them short (less than 77 words).
C.1. Image Captioning
In Fig. 17, show examples of image captioning on fantas- tical images from an image generation model. We see that the MILSS model is able to generate captions that are more descriptive and accurate than the baseline model, even for images that are not realistic. This example also showcases the rigidity of memory-based models like McaCap (Zeng et al., 2024).
C.2. Video Captioning
Please see the attached supplementary video for video cap- tioning results.
3

---

LLMs can see and hear without any training
10
CLIP similarity
9
SPICE
0.1
8
Gemma2 9B
Mistral 8B
Llama 3.1 8B
6
0
6
1
2
3
4
5
6
7
8
9
10
0
2
3
4
5
6
8
9
10
Steps
Steps
Figure 16: Performance comparison with various LLMs GENERATOR- Gemma2 9B, Mistral 8B, and Llama 3.1 8B, on CLIP similarity (left) and SPICE (right). This trend shows that the performance improves over optimization steps, regardless of the choice the L.M.. In particular, LJama 3.1 8B gives the best performance, and used primarily as the GENERATOR in our experiments.
C.6. Ablations
Different types of GENERATOR and SCORER We exper- iment with different open source LLMs, and CLIP models. We present results in Figure 16 and Table 4.
First, we observe that while we see the same trend of perfor- mance improvement over steps for all LLMs we considered, the Llama model performed the best, with Mistral a close second. This shows that MILS generalizes to various dif-- ferent LLMs, and can expect to improve as LLMs advance further.
Cosmic artwork foaturing space explome's dreamy, ethereal, surreal, otherwor ldly journey. image depicts space vihrant poster using digital imag film and graphics software.
in Table that SigLIP is the strongest scorer for this task, when compared with models of the same pa- rameter size. Again, the performance remains high for all scorers, showcasing the flexibility of MILS.
Second,
Figure 17: Image Captioning using MILS, compared to existing state-of-the-an cero-shot approach, MeaCap (Zeng al., 2024) in captioning fantastical image. We see MeaCap is not able to generate the correct caption since it uses memory that only contains captions for real- real-images.
Investigating convergence to the global maximum As discussed Section we explore an e-greedy strategy to encourage exploration. This process includes low-score generations, account for noise in the scoring process. However, we do not observe any gain using this idea. Re- gardless, MILS' overall use of a stable score is similar to other applications of optimization in machine learning.
C.3. Audio Captioning
Please see the attached supplementary video for audio cap-
tioning results.
C.4. Cross-Modal Arithmetic
Please see the attached supplementary video for cross-modal arithmetic results.
C.5. Inference-time analysis
MILS' inference time is proportional the number of iter- ations. As shown in Fig. 9, MILS performs well even with few iterations, and the runtime will continue decrease more efficient and higher quality LLMs become available, that can reason in fewer steps. Moreover, the initial set size does not affect compute time much, as we can compute dot product similarity with the media features very fast on modern GPUs.


Reviews:
[{"review_id": "IJdCRD6y6X", "reviewer": "Reviewer_934y", "summary": "This paper presents an iterative solver to enable pure language LLMs to \"perceive\" visual or audio signals through trial and error using discriminative scorers. Specifically, the authors construct a feedback loop between generators (LLMs) and discriminative ranker (e.g., SigLIP for image captions): first, the LLM first output initial random guess lists by prompting; next, the score ranks guess lists with regard to images/audio, and feed guess lists with scores back into the LLM; then, the LLM analyze the guess lists with scores and iteratively refine new guess lists into the \"right\" direction. The authors also generalize this framework to audio caption, image generation and style transfer tasks, and explore other cross-modal tasks with the help of ImageBind.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful review and willingness to accept the paper. We are glad you found our work novel, and claims supported by clear and convincing empirical evidence. We address all the remaining concerns below:\n\n\n## For multimodal understanding, what is the strength or potential of MILS rather than proposing yet another way to do it?\n\nGreat point! We agree -- one can train specific multimodal models for tasks that have enough training data that are hyper-optimized for that specific task. For online production use cases that require millisecond-level optimization, training a specific model for that task might indeed be the way to go.\n\nMILS, on the other hand, can supercharge research exploration or applications that may not require such a high level of runtime optimization. For example, MILS can quickly provide a strong multimodal model for a new task, without any training, completely zero-shot. This could be useful to researchers as an initial baseline for any new task they might want to solve. It could also serve as an easy and effective approach to interpret new models a researcher develops. Consider the use case of interpreting the strengths/weaknesses of a new multimodal embedding model and comparing it to CLIP/SigLIP etc. MILS can enable that interpretability by helping invert each model’s embedding space into text (without any training that might bias this inversion). The produced image captions for different embedding models could be very helpful for a researcher to better understand the behaviors of these models. Finally, MILS can be used as an offline data generator, which can then be used to train or distill optimized multimodal models. Consider for instance the enhanced image generation task – MILS can be used to generate large amounts of <text, improved image> pairs, which can be used to tune a text-to-image model for improved quality.\n\n## Q1. Inference time analysis\n\nMILS’ inference time is proportional to the number of iterations. As shown in Fig. 9, MILS performs well even with a few iterations (eg 3), and the runtime will continue to decrease as more efficient and higher quality LLMs become available, that can reason in fewer steps. Moreover, the initial set size does not affect compute time much, as we can compute dot product similarity with the media features very fast on modern GPUs. We will add additional runtime analysis and comparisons with similarly sized task-specific multimodal understanding models in the final paper.\n\n\n## Q2. Can MILS solve tasks like spatial referring?\n\nCertainly! We can generate box proposals and referring expressions (e.g. a class label or a short description) with the LLM, compute box-level CLIP similarity score by computing CLIP features on the cropped box, and use that as the score to give feedback to LLM. The LLM will then produce new box proposals and referring expressions. We can then repeat this iterative process as proposed in MILS. The scoring can further be made more efficient by pooling part of the full image’s spatial CLIP features that correspond to the box, and matching that to text.\n\nOverall, as long as we design a scorer for a task, which can evaluate the correctness of the LLM generation with respect to the input (can be any modality), we can integrate that task into MILS. It can be similarly extended to segmentation, object detection in 3D, etc.\n\n## Discuss more gradient-free optimization papers\n\n\nThat is a great suggestion! We will add more references and discussions to gradient-free optimization works to the final paper.", "claims_and_evidence": "The claims, LLMs can see and hear without any training, can be supported by clear and convincing empirical evidence presented in this paper. Indeed, LLMs can see and hear by correctly prompting based on feedbacks in a loop style. We can say this paper presents a gradient-free optimization to the score by prompting candidates.", "methods_and_evaluation_criteria": "This method present a novel approach to make pure LLMs to see and hear without any training on image/audio-language pair samples. That said, I have concerns whether proposed MILS have real application potentials compared to other multimodal LLMs. While loop methods between generators and scores being under-explored before, MILS performance seems relatively lagging behind common pipelines to multimodal understanding. I understand that authors' goal is not achieve new state-of-the-art performance, but the looping mechanism between generators and scores seem time-costing and highly inefficient. Indeed, I find MILS do have potentials in generating high-quality image generation/style transfer samples in Sec 4.4 and 4.5 to serve as post-training samples for further fine-tuning nowadays pipelines. Now back to my question, for multimodal understanding, what is the strength or potential of MILS rather than proposing yet another way to do it?", "theoretical_claims": "There are no theoretical claims nor proofs in this submission.", "experimental_designs_or_analyses": "Most experimental designs and analyses are sound and valid in my opinion. A minor concern is the absence of inference time analysis concerning the number of iterations and the initial set size, which is crucial for real-world applications.", "relation_to_broader_scientific_literature": "In my view, this paper presents a gradient-free optimization for scores by the scorer by prompting candidate sets.", "essential_references_not_discussed": "I do not find any specific references missing in this submission. But it would make this paper more sound to discuss more gradient-free optimization.", "other_strengths_and_weaknesses": "Overall I think this paper presents a \"yet-another-approach\" to leverage pure LLM to understanding multimodal signals without any training/fine-tuning on paired samples. This paper is indeed novel but with some minor concerns in real application potentials (See Methods And Evaluation Criteria).", "other_comments_or_suggestions": "NA", "questions_for_authors": "Q.1. As mentioned in Experimental Designs Or Analyse, the authors might provide inference time analysis and its comparison with standard multimodal understanding pipelines.\n\nQ.2. I feel MLIS quite limited in \"caption\" tasks. Actually, it seems that the capability of MLIS as a gradient-free optimization is \"bounded\" by the scorer capability. In the paper case, the scorer, SigLIP, provide caption and visual similarity. My concern is, whether MLIS can solve other multimodal questions beyond captioning, like spatial referring? No experiments needed but I'd like to see authors' insights into it.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7VWnTO7eaU", "reviewer": "Reviewer_qksR", "summary": "The authors propose MILS: Multimodal Iterative LLM Solver, a simple method that claims to use the reasoning abilities of textual LLMs to have impressive zero-shot performance on multimodal tasks. At a high level, this involves a Generator model, which is either a text-LLM or a text-LLM chained to another system such as an image generation model and a Scorer, which evaluates the quality of the generations. By optimizing the generator to produce outputs that are preferred by the scorer, it is found that the method is competive against purpose-built models for image, video and audio captioning, and also improves generative capabilities for text-to-image generation. The authors also discuss cross-modal arithmetic where they are able to combine different modalities using MILS", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful review and their willingness to accept our paper. We are glad they found our method generalizable, claims well justified, and improved performance on a variety of different tasks. We address all concerns next:\n\n## Investigation into the reasoning capabilities of the LLM that cause this to work\nThis is a great suggestion and we’ll attempt this for the final paper. Understanding the reasoning capabilities of LLMs is indeed a very fast evolving field of research, and many of the discoveries there would also help better understand MILS. For instance, deeper insights into MILS can be obtained by taking into account the reasoning traces available in recent LLMs such as DeepSeek-R1. Another interesting exploration could be in the LLM-scorer interface, for instance experimenting with the sensitivity of the LLM to the resolution of the scorer outputs. In general, LLMs as optimizers are relatively easier to probe compared to more opaque optimizers, given their ability to explain decisions in plain language. We believe this ability will help MILS expose new avenues for research both in LLM reasoning and multimodal understanding.\n\n\n## Novelty statement compared to other ML literature\n\nOurs is the first work to leverage test-time reasoning capabilities of LLMs to solve multimodal tasks, obtaining state-of-the-art results across a wide range of computer vision/multimodal benchmarks: image, video and audio captioning, enhanced image generation (outperforming very strong models like FLUX), style transfer and cross-modal arithmetic. As reviewer 934y also notes on the novelty, this approach enables pure text-only LLMs to see and hear without needing any training on that kind of data. Our novelty also lies in the simplicity of our framework that does not need any task-specific modifications. We do take inspiration from standard ML literature where the concept of scoring functions and optimization is used extensively, however to our knowledge, it hasn’t been used in this way for test-time optimization, using LLMs, to solve multimodal tasks.\n\n\nIf there is specific prior work the reviewer would like us to compare to, please let us know and we would be happy to provide more comparisons to that.", "claims_and_evidence": "The authors claim that their method is extremely generalizable and only requires a generator and scorer, which can be basically any system that generates output for a task and one that can evaluate them. This is well-justified by examples of image/audio/video captioningn and generative tasks such as image generation and style transfer.", "methods_and_evaluation_criteria": "The authors compare to appropriate models using standard metrics on standard benchmark datasets.", "theoretical_claims": "The paper does not make any theoretical claims of note", "experimental_designs_or_analyses": "The experiments are set up in an appropriate manner, with detailed explanation of bootstrapping, set up and output generation provided across a variety of tasks. The experiments are mainly just examples of the method working well, rather than trying to question and/or investigate the method itself, though.", "relation_to_broader_scientific_literature": "The fundamentally plu-gan-play nature of this method implies that it is both deeply embedded in the general generative deep learning literature while also standing somewhat apart from it by being a gradient-free method that is plug and play.", "essential_references_not_discussed": "I did not find any major references missing", "other_strengths_and_weaknesses": "Strengths:\n1. The method is simple, plug-and-play and effective, showing improved performance of the method on a variety of tasks.\n2. The method also generalizes across different textual LLMs, and scales to their intelligence/reasoning abilities. \n\nWeaknesses:\n1. There is little, if any, deep investigation into the reasoning capabilities of the LLM that cause this to work. I would like to see a more detailed deep dive into the optimization process and it's potential brittleness when coming up against harder tasks.\n2. I am not sure if this is necessarily novel and therefore appropriate for ICML. The novelty is very much at the system-level and the idea of a scorer giving feedback to a generator, while applied well, is not new to the ML literature in general.", "other_comments_or_suggestions": "None", "questions_for_authors": "I do not have any major questions for the authors. This is a well-written paper and shows a practical method that enables us to make existing systems better without expensive training runs.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0uJLTkrG9p", "reviewer": "Reviewer_XRE1", "summary": "This paper introduces MILS (Multimodal Iterative LLM Solver), which enables large language models (LLMs) to perform various multimodal tasks without any additional training. Through test-time optimization, it incrementally improves its outputs by leveraging two key modules (GENERATOR and the SCORER), by generating a set of outputs and feedback with scores to iteratively refine the outputs. Experimental studies show that MILS can handle image, video, and audio captioning as well as generative tasks such as text-to-image creation and style transfer, all without requiring task-specific pretraining.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback. We address all concerns below. In particular, we highlight the benefits of our iterative approach vs MeaCap, which is designed specifically for image captioning. We also clarify a possible misunderstanding regarding the use of large 405B LLM–except the qualitative examples shown for cross-modal arithmetic, all other results/examples in the main paper+supplementary use Llama 8B.\n\n## Benefits compared to MeaCap/DeCap\n\na) **Iterative optimization generalizes to more tasks**\n\nMeaCap/DeCap are specific to image captioning. They leverage a corpus of image captions from real-world datasets (e.g. CC3M). In contrast, MILS’ initial set does not contain captions from any dataset. Consider the task of captioning fantastical images from an image generation model. Retrieval based methods like MeaCap/DeCap would struggle since the CC3M captions will be very out-of-distribution and need significant modifications. To illustrate this, we ran both on [this image](https://www.freepik.com/free-ai-image/spaceman-riding-horse-outer-space_222195825.htm), and got the following outputs:\n```\nMeaCap: The image depicts that space in a vibrant poster using raster or digital images, film and graphics software.\nMILS: Cosmic artwork featuring a space explorer's dreamy, ethereal, surreal, otherworldly journey.\n```\nClearly, MILS captures the essence of the image much better than MeaCap. We will add more such comparisons to the final paper.\n\nb) **MILS avoids hand-crafted design and embraces end-to-end optimization**\n\n\nMeaCap relies on a hand-designed subject-predicate-object based refinement, followed by iterative CBART sentence generation. MILS does not enforce any such constraints, and end-to-end optimizes the LLM generations, directly using feedback from the pre-trained multimodal embedding model. In this sense it can be thought of as a simplification and generalization of MeaCap, making it applicable to five more tasks than just image captioning.\n\n\nc) **Experimental comparison with MeaCap and Decap (Tab. 1)**\n\nWe report comparison with *training-free versions* of all methods: 42.1 and 50.6 CIDEr in DeCap require training the LM on captions from image/video-captioning datasets. In contrast, MILS is completely training-free (TF). Moreover, MILS’ 33.3 CIDEr, outperforms MeaCap results that we could reproduce from the published code, which is 26.0 for MeaCap-TF (vs 42.0 reported in the paper)–we confirmed with the authors this variance was expected for the TF version. \n\n\n## “Quality of the captions is inherently capped by the LLM’s linguistic generation capabilities”\n\nLLMs have very strong linguistic generation capabilities, as evident from their ability to generate stories, rhymes, etc; even conversing with humans such that many consider it to [have passed the turing test](https://www.nature.com/articles/d41586-023-02361-7). The external corpus in MeaCap–CC3M–contains 3M captions, which one could argue is much smaller than the linguistic ability and world knowledge stored in the LLM, if measured by their training data or parameters.\n\nIn fact, using CC3M gives MeaCap an advantage over MILS since it leverages captioning-specific knowledge, while MILS relies only on a general-purpose LLM.\n\n## MeaCap improves performance by increasing memory, “MILS appears to have intrinsic performance ceilings”\n\nMILS scales well too! We obtain clear improvements using LLMs/Scorers that are bigger (Fig 12), better (Appendix Fig 16, Tab 4), or using a larger initial set (Fig 10), with no ceilings observed thus far.\n\n\n## Error accumulation analysis\n\nMILS performs iterative refinement, i.e. optimization, as opposed to iterative generation—the error will reduce over iterations as shown in our results over optimization steps (Fig. 9 and Fig. 8, 11). Errors would accumulate if we were fixing some part of our output at a given iteration and generating the remainder based on that—we are globally improving the output until it converges.\n\n##  Computational efficiency/“seems too expensive with 10 steps for optimization”\n\nThe inference time is proportional to the number of iterations. The model performs well even with few iterations–please refer to the performance curves in Fig. 9, which shows even in 3 steps the performance is within 90% of the best. The iteration knob further allows MILS to trade-off quality vs speed, for time-sensitive applications. We expect the speed to improve and number of steps required to reduce as LLMs become more efficient and powerful.\n\n## “Use of extremely large pretrained LLMs such as Llama 405B”\n\nThis is a possible misunderstanding. We only use 8B LLM models for all experiments except multi-modal arithmetic (Fig. 7). All captioning tasks use 8B LLM. In fact, we only used 405B for multi-modal arithmetic since we could easily use API calls for the qualitative results; using the 8B variant also yields similar results. We will update the paper with 8B for arithmetic for consistency.", "claims_and_evidence": "The paper claims that MILS (Multimodal Iterative LLM Solver) demonstrates emergent zero-shot multimodal capabilities through an iterative optimization framework leveraging pre-trained LLMs and CLIP-based multimodal scorers. However, I feel it is inadequately supported by the presented evidence. To convincingly establish this emergent reasoning capability, the authors must explicitly analyze and experimentally distinguish the incremental performance benefits provided specifically by iterative reasoning via LLM-generated candidates compared to existing methods that already effectively leverage pre-trained multimodal embedding representations.\n\n- First, the paper does not clearly distinguish how much iterative reasoning via the LLM contributes to the observed zero-shot performance beyond what has already been demonstrated by existing memory-based methods such as MeaCap. Recent works like MeaCap have already shown strong zero-shot multimodal capabilities by directly leveraging pre-trained multimodal embeddings (e.g., CLIP, SigLIP, ImageBind). Given this prior work, it is essential for MILS to provide explicit evidence demonstrating that iterative candidate generation through LLM reasoning meaningfully enhances performance beyond what pre-trained multimodal embeddings alone achieve. However, the authors do not provide a clear comparison of the additional benefits specifically attributable to iterative optimization via LLM-generated candidates.\n\n- Additionally, in the presented experimental results (e.g., Table 1), MILS does not achieve clear superiority over memory-based approaches such as MeaCap, particularly regarding critical evaluation metrics like CIDEr and BLEU. For instance, MILS attains a CIDEr score of 33.3, which is substantially lower than MeaCap’s 42.5 (and 51.7 when using high-quality SS1M memory). Higher scores (42.1 and 50.6) of DeCap  (which is only cited) are also missing. While the paper selectively emphasizes metrics such as METEOR and SPICE, even in these semantic metrics, MILS either achieves performance similar to or only marginally better than MeaCap.", "methods_and_evaluation_criteria": "The methodological approach of MILS is based on iterative optimization. It leverages a process in which a large language model (LLM) generates multiple candidates and a SCORER provides repeated evaluations to address multimodal tasks. However, the iterative optimization employed by MILS is fundamentally restricted by the candidate generation capacity of the LLM. \n- During each iteration, MILS generates tens to hundreds of candidates. This process implies that the ultimate quality of the captions is inherently capped by the LLM’s linguistic generation capabilities. In contrast, the memory-based approach adopted by MeaCap capitalizes on a vast external corpus that stores extensive world knowledge, which can be retrieved and filtered during caption generation. \n- Consequently, an increase in both the size and quality of the memory can directly and substantially enhance caption performance. For example, according to MeaCap, when employing a large-scale, high-quality memory such as SS1M, the CIDEr score can rise to 42.5. By comparison, MILS, even when limited to around 10 iterations, converges at a CIDEr score of 33.3, demonstrating significantly lower performance relative to SS1M. \n- This discrepancy shows a fundamental drawback: the iterative candidate approach in MILS appears to have intrinsic performance ceilings that cannot be overcome merely by increasing the number of iterations or candidates. However, the authors do not sufficiently discuss or analyze this limitation.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I have several concerns about the experimental design and analyses.\n\n1. Inadequate Comparison with MeaCap\n- The paper’s experimental comparisons with MeaCap are insufficient. MeaCap employs a memory retrieval-based approach, which offers fundamentally different scaling properties and cost structures compared to the iterative methodology of MILS. While MeaCap’s performance can significantly improve with larger and higher-quality memories (as evidenced by the high CIDEr score using SS1M), MILS is constrained by the inherent limitations of LLM’s candidate generation. Even increasing the number of iterations or candidates does not seem to surpass these fundamental limitations. This disparity between the two approaches is not adequately addressed or discussed in the experimental design.\n\n2. Lack of Analysis on Error Accumulation\n- The authors do not provide an analysis of error accumulation—a critical issue in iterative methods. With each iteration, there exists a risk that erroneous candidates may be generated and, through the feedback process, inadvertently reinforced. Such errors can be caused by either Generator or Scorer, and can be worse during iteration adversarial to CLIP scores.\n\n3. Missing Computational Efficiency Evaluation\n- Given that MILS necessitates repeated calls to an LLM, I have concerns regarding inference time and computational cost. However, the paper does not present any rigorous evaluation or discussion on computational efficiency. Due to the fundamental limitation of LLM capability, I suspect MILS has to scale up LLM to achieve fine-quality outputs. I also found the video provided in the supplementary material provided by Llama 405B model. \n- When we consider MILS requires around 10 (or more) steps for optimization, it seems too expensive compared to the existing memory-style training-free approaches.", "relation_to_broader_scientific_literature": "The key contributions of this paper align closely with recent works exploring training-free and zero-shot multimodal approaches such as MeaCap and DeDap, specifically those relying on pre-trained large multimodal embedding models and LLMs. \n- For example, although MILS does not employ external memory directly, MeaCap similarly avoids task-specific training and relies heavily on pre-trained multimodal embedding models (e.g., CLIP) for evaluation. The key difference lies in MILS’s iterative candidate generation approach, where it repeatedly generates and scores multiple candidate captions, refining them iteratively. In contrast, MeaCap utilizes a single-step retrieval-and-generation pipeline, relying solely on high-quality external memory to enhance caption quality. So, I think MILS's K-candidates play a similar role in MeaCap's memory. It can be useful when we even do not have any external texts to build such memory, but  I think high-quality memory can perform much better than the proposed method.", "essential_references_not_discussed": "The paper explicitly cites DeCap (ICLR, 2023) in its references but omits DeCap from main experimental comparisons (such as Table 1). \n- Reported numbers of DeCap correspond to Table 1 are 42.1 and 50.6 CIDEr, which are significantly higher than 33.3 of MILS. Including DeCap in experimental comparisons is essential for clearly validating the effectiveness of MILS’s iterative candidate optimization.", "other_strengths_and_weaknesses": "Strengths\n- This paper clearly reveals the performance capabilities and limitations of pre-trained LLM-based training-free methods across various multimodal tasks. By evaluating emergent zero-shot capabilities in multimodal tasks such as video captioning and image generation, MILS provides strong baseline performance achievable purely through iterative optimization of LLM-generated candidates and multimodal embeddings. \n\nWeakness\n- MILS inherently requires extremely large-scale pre-trained LLMs to achieve competitive performance. As explicitly shown in the supplemental experiments on video captioning tasks, MILS utilizes exceptionally large-scale LLMs such as LLaMA 400B. Such huge model sizes impose considerable computational costs and resource constraints, severely limiting practical feasibility and scalability. Specifically, the iterative optimization approach of MILS inherently restricts meaningful performance gains to the expensive strategy of continually scaling up the size of the underlying LLM. In contrast, memory-based approaches (e.g., MeaCap) inherently offer a much more scalable and cost-effective means of improving performance by simply increasing the size and quality of external memory. Given these considerations, while MILS can indeed serve as a useful baseline reference for multimodal tasks beyond standard captioning benchmarks, it appears clear that, in practice, memory-based methods would offer a far more effective solution due to their superior scalability and cost-efficiency.", "other_comments_or_suggestions": "see the above sections.", "questions_for_authors": "see the above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sMeLXVEdYD", "reviewer": "Reviewer_Deut", "summary": "The paper introduces Multimodal Iterative LLM Solver (MILS), a novel framework designed to provide multimodal capabilities, such as image, video, and audio captioning, without the need for task-specific training. The central innovation of MILS is its iterative optimization strategy, leveraging the inherent reasoning skills of LLMs. It operates by repeatedly using an LLM as a GENERATOR that proposes candidate solutions, which are then assessed by a SCORER, typically a pretrained multimodal model. The scored feedback informs subsequent generations in a loop until convergence. Key contributions include achieving emergent zero-shot generalization across different modalities, performing competitively with specialized models. MILS simplifies those multimodalities tasks through a straightforward approach..", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your insightful review and helpful comments. We address all concerns below:\n\n## Clarification of the text representation in cross modal arithmetic\n\nThe representation is indeed the raw text. The novelty is in the inversion of the image into the caption itself. Using a captioning model would require training it on image-caption data. MILS can invert multimodal embedding spaces like CLIP into text *without needing any captioning training* and is completely gradient free, allowing for the inversion to discrete text as opposed to a continuous embedding space. Combining that with generation, it is able to perform cross-modal arithmetic, similar to ImageBind (Girdhar et al. 2023). However, by using text-space instead of the image embeddings in ImageBind, MILS is a lot more flexible–making it work with various image generation models, allows for user control etc (as we mention in L384 left).\n\n## Generalizability of MILS\n\nWhile MILS’ performance does depend on the choice of LLM/scorer, it works well with a wide range of LLMs and scorers. Please see Fig. 16 and Tab 4 in appendix – while better scorer/generator give better results, MILS works reasonably well across all the combinations of LLMs (eg Llama, Mistral, Gemma) and scorers (CLIP, SigLIP, DFN), showing its generalizability.\n\n## Convergence to the global maximum\n\nGreat point! To avoid local maximas, we tried an experiment similar to noise injection in gradients from ML optimization.  As we mention in L134 (right column), we chose a portion of the candidates in an optimization step randomly, without considering their score. The idea is that, if a caption has a low score in the current round, a variant of it can still have a better score at a later round, effectively escaping the local maxima. However in practice, we did not notice a significant improvement. \n\nRegardless, MILS’ overall use of a stable score is similar to other applications of optimization in ML (e.g. neural network training) -- it could lead to a local optima but still be useful. As we show, MILS’ optimization leads to many useful applications. We will add more details and a discussion on this point in the appendix.\n\n## Using FID as a metric\n\nIn the image generation experiments, our goal is to generate images preferred by humans. Thus, human evaluation is the best fit for our setup.\n\nFID, on the other hand, has been shown to not be a reliable measure for human preference, as discussed in relevant papers ([Jayasumana et al. 2024 ‘Rethinking FIDs…’](https://arxiv.org/abs/2401.09603), [Parma et al. 2022, ‘On…GAN evaluation’](https://arxiv.org/abs/2104.11222), [Chong et al. 2020, ‘...unbiased FID…’](https://arxiv.org/pdf/1911.07023), [Borji et al. 2018, ‘Pros and Cons…’](https://arxiv.org/abs/1802.03446)). Large-scale image/video generation efforts (eg [SDXL](https://arxiv.org/abs/2307.01952), [Emu Video](https://arxiv.org/abs/2311.10709),  [MovieGen](https://arxiv.org/abs/2410.13720), etc) have found automatic metrics like FID do not correlate well with human judgment, and also primarily focus on human evaluation. SDXL mentions “we find these classical quantitative scores not to be suitable for evaluating the performance of foundational (text-to-image) DMs.” Consequently, we do not use FID as a metric in this work.\n\n## It is unclear how different LLMs as GENERATOR impact the results….and other ablation studies\n\nWe do provide experiments studying the effect of different kinds of  LLMs and Scorers in Appendix Fig. 16 and Tab. 4, respectively. We also study performance over different sized LLMs and Scorers in Fig. 12 (main paper). The performance is proportional to the relative strength of the LLM being used.\n\nPlease let us know what other ablation studies are missing and we will clarify and add those.\n\n## It lacks a clear distinction from standard prompting techniques or direct zero-shot inference using existing multi-models.\n\nWe would love to get more details on specific models that the reviewer is comparing novelty to, and will be happy to compare to that. \n\nIn general, multimodal vision-language models models are not *emergent zero-shot* (as our approach is), since they are trained for tasks like captioning. Hence, they generalize ‘zero-shot’ to a different data distribution. Our approach, on the other hand, generalizes to the completely new task of captioning itself – and captions images without ever being trained for it. We mention this in L071 left.\n\n## How is it different from choosing the best generation using a scorer?\n\nThis is exactly the first step of our optimization process, and the performance remains low without iterative refinement using the generator and the scorer (Fig. 9, 0th step).\n\nIn short, our method consists of creating an initial set, followed by using the scorer to find top-K candidates, and iteratively feeding it back to the generator to create more candidates based on the score. This process is described in Sec. 3.", "claims_and_evidence": "Overall, the paper presents strong experimental evidence supporting many of its claims. However, some claims, particularly those regarding Cross-Modal Arithmetic and Generalization Ability, require further clarification and justification.\n\n1. The paper claims that MILS allows arithmetic across modalities by converting them into text representations, combining them, and mapping them back. However, the authors do not clearly explain what these text representations are. Are they simply raw text formats? If so, what differentiates MILS from a pipeline consisting of a captioning model followed by a generation model? Without a clear distinction, the novelty and effectiveness of MILS in cross-modal arithmetic remain unclear.\n\n2. The paper claims that MILS is conceptually simple and generalizable to many tasks. While the framework itself is modular, its success is heavily dependent on the choice of the GENERATOR and SCORER models. The effectiveness of MILS may vary significantly depending on the underlying models used, making it less generalizable than implied.", "methods_and_evaluation_criteria": "1. The paper uses score similarity to determine convergence, assuming that a stable score indicates an optimal solution. However, this approach may be insufficient and potentially misleading. A lack of score change does not necessarily imply the best possible result, it could indicate stagnation in a local minimum rather than true convergence.\n\n2. The paper primarily evaluates image generation and image editing using PickScore and human preferences, but these alone do not fully assess generation quality. Metrics like FID could offer quantitative insights into visual realism and text alignment, while diversity metrics would help assess mode collapse.", "theoretical_claims": "No theoretical claims in the paper.", "experimental_designs_or_analyses": "1. The paper determines convergence by monitoring score similarity over iterations. However, this method assumes that a stable score directly correlates with optimal performance, which may not be valid.\n\n2. It lacks established quantitative measures such as FID, or other diversity metrics, which would provide a more comprehensive analysis of generation quality and text-image alignment.\n\n3. The effectiveness of MILS depends on the GENERATOR and SCORER models, but the paper does not provide enough ablation studies to analyze the contribution of different components. For example, it is unclear how different LLMs as GENERATOR impact the results.", "relation_to_broader_scientific_literature": "The paper extends test-time reasoning in LLMs by applying iterative refinement beyond text tasks. In zero-shot multimodal learning, prior captioning methods rely on gradient-based optimization, whereas MILS achieves zero-shot generalization purely at inference by combining LLMs with multimodal scoring models. The paper also contributes to multimodal representation learning, similar to ImageBind, by inverting multimodal embeddings into text, enabling cross-modal arithmetic and generation.", "essential_references_not_discussed": "References are essential.", "other_strengths_and_weaknesses": "The paper presents a straightforward approach, but its novelty is limited. The proposed method essentially combines an LLM with a pre-trained multimodal model in an iterative optimization loop, which does not fundamentally differ from existing approaches that use LLMs and multimodal models together. While MILS formalizes this process as a simple framework, it lacks a clear distinction from standard prompting techniques or direct zero-shot inference using existing multi-models.", "other_comments_or_suggestions": "The paper is well-written and organized, making it easy to follow the methodology and experimental setup.", "questions_for_authors": "The paper presents MILS as a zero-shot multimodal optimization framework, but it appears to function similarly to direct LLM prompting with a multimodal model as a filter. What distinguishes MILS from simply iterating over LLM-generated candidates and selecting the best one using a scoring model? A clearer articulation of its novelty is expected.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a training-free approach to extend LLM for multi-modal tasks such as image captioning, text-to-image generation, etc. The basic idea is to use LLM to generate prompts that are fed to to an additional multi-modal scorer. The paper receives mixed scores. In general, most reviewers consider the problem and proposed approach to be novel, the presentation is clear, the gradient free approach to extend LLM with multi-modal ability is interesting, and the experiments are extensive. At the same time, reviewers also raised multiple concerns. For example, inference time complexity seems to be high, the underlying working mechanism of the proposed method is not very clear, the reliance of the performance on specific generator and scores, as well as the convergence issue of the proposed optimization method. While most of the concerns have been addressed in the rebuttal, some reviewers remain unconvinced on several aspects. Particularly, some reviewer believe one critical issue of the proposed method is the scalability. At the current stage, it is not clear whether the method can be scaled to outperform existing baselines such as MeaCap. I share a similar concern with this, and also have a little question on the reasonability of the proposed method since it seems to have limited application scenarios in real problems.

Overall, I consider this paper to be borderline. It is acceptable depending on the room in the program.
Decision: Accept (poster)