An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching
Authors: Hugo Malard
Michel Olvera
Stéphane Lathuilière
Slim Essid
Venue: NeurIPS 2024 poster (2024)
Keywords: Multimodal representation learning
Audio Captioning
Image Captioning
Audio-Visual
Large Language Model
Primary area: speech_and_audio
TL;DR: We propose a novel method for aligning audio and image tokens to enable zero-shot audio captioning throught MMD and Optimal Transport leveraging a large vision language models, achieving superior performance in unsupervised settings.

Abstract: Multimodal large language models have fueled progress in image captioning. These models, fine-tuned on vast image datasets, exhibit a deep understanding of semantic concepts.
In this work, we show that this ability can be re-purposed for audio captioning, where the joint image-language decoder can be leveraged to describe auditory content associated with image sequences within videos featuring audiovisual content. This can be achieved via multimodal alignment.
Yet, this multimodal alignment task is non-trivial due to the inherent disparity between audible and visible elements in real-world videos. Moreover, multimodal representation learning often relies on contrastive learning, facing the challenge of the so-called modality gap which hinders smooth integration between modalities. In this work, we introduce a novel methodology for bridging the audiovisual modality gap by matching the distributions of tokens produced by an audio backbone and those of an image captioner. Our approach aligns the audio token distribution with that of the image tokens, enabling the model to perform zero-shot audio captioning in an unsupervised fashion. This alignment allows for the use of either audio or audiovisual input by combining or substituting the image encoder with the aligned audio encoder. Our method achieves significantly improved performances in zero-shot audio captioning, compared to existing approaches.

An Eye for an Ear:
Zero-shot Audio Description Leveraging an Image
Captioner using Audiovisual Distribution Alignment
Hugo Malard Michel Olvera"
Stéphane Lathuiliere'
Slim Essid"
LLCII Télécom Paris, Institut Polytechnique de Paris
michel aichell-ovveraadddeeeeomopprppr
Abstract
Multimodal large language models have fueled progress in image captioning.
These models, fine-tuned vast image datasets, exhibit deep understanding of semantic concepts. In this work, we show that this ability can be re-purposed for audio captioning, where the joint image-language decoder be leveraged describe auditory content associated with image sequences within videos featuring audiovisual content. This can be achieved via multimodal alignment. Yet, this multimodal alignment task is non-trivial due to the inherent disparity between audible and visible elements in real-world videos. Moreover, multimodal repre- sentation learning often relies on contrastive learning, facing the challenge of the so-called modality gap which hinders smooth integration between modalities. this work, we introduce novel methodology for bridging the audiovisual modality gap by matching the distributions of tokens produced by an audio backhone and those of an image captioner. Our approach aligns the audio token distribution with that of the image tokens, enabling the model to perform zero-shot audio captioning in unsupervised fashion while keeping the initial image captioning component unaltered. This alignment allows for the use of either audio or audiovi- sual input by combining substituting the image encoder with the aligned audio encoder. Our method achieves significantly improved performances zero-shot audio captioning, compared to existing approaches
Introduction
Recent progress image captioning has been driven by methods integrating Large Language Models (LLMs) with vision encoders I 2 3). The impressive capabilities of Vision Language Models (VLMs) stem from supervised training on large image-text collections and extensive parameterization. Even the smallest VLMs exceed more than billion parameters 4 5). These models excel in generating human-like text descriptions for images and understanding complex semantic relationships. be potentially extended other tasks, beyond image analysis.
Such capabilities
One such task audio captioning, in which large-scale audio-text collections, with precise descrip- tions of the audio content, are lacking. Recent works have proposed diverse solutions to address this limitation. Solutions include augmenting class labels with phrases such as "This is sound of ( prompting LLMs to generate natural language descriptions directly from class labels (8). Further, datasets from related domains like Speech (9) or Music (10, IT), have been considered to enlarge training data. While such methods show impressive results, scaling them further is challenging due to the rarity and low quality such labels-captin captioss.. In particular, these approaches cannot cope with the annotation noise encountered in existing audio datasets, especially in the largely adopted resource: audio tracks of AudioSet (12). which is only partially and weakly labeled.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Previous
Encoder
taliiigg .
Encoder
LLM
The sound of rangsing.
MODALITY
*iee acrund oi
Lunge al.mm
mage
chainean's
Fiky mymmem
ronz thoooagh socd.
Audio
Joint audio-visual space
*Th cound chachease and (1ET besing movved.
Ours
Figure 1: Conventional audiovisual alignment through contrastive learning leads to gap between modalities. Our proposed distribution alignment method matches closely both distributions leading to better joint representations for audio captioning.
We hypothesize that VLM-based image captioning models, inherently possess the knowledge to perform audio captioning. Thus, joint image-language representations of VLMs can be used to describe auditory content associated with image sequences from videos featuring audiovisual content, eliminating the need for handcrafted audio-caption pairs. To our knowledge, this is the first attempt to leverage VLMs for this purpose. Extending VLMs to audio captioning is non-trivial due to the inherent disparity between audible and visible elements in real-world videos. For instance, sound- producing objects in images may occluded or out view, while visible objects might not generate sound. This mismatch hinders connecting visual and auditory modalities seamlessly.
Multimodal representation learning [33, crucial for tasks like image captioning, is often accomplished through contrastive learning (14 6). This paradigm aligns features across modalities by maximizing agreement between similar samples and minimizing agreement between dissimilar ones. Despite its effectiveness, contrastive learning faces the so-called "modality gap". where embedded data are represented distinct, non-overlapping regions of the embedding space (55 16) (the distance between red and green manifolds in Figure D. limiting flexibility when conditioning modalities.
We propose novel methodology to bridge the audiovisual modality gap by matching the distribu- tions of tokens produced by an audio backbone and those from the encoder of an image captioner (represented the green and blue manifolds in Figure T. enabling the image captioner to perform zero-shot audio captioning in an unsupervised way. We define zero-shot audio captioning as the task of generating audio captions without training on manually assembled audio-caption pairs. We propose and compare two token distribution alignment methods: one based on Maximum Mean Discrepancy (MMD) (17 and a more flexible alternative using Optimal Transport (OT) (8). possibly enhanced with cross-attentio mechanism. This mechanism refines distribution matching by attending to each token within modality and determining its match with a token from the other modality based on semantic similarity. We find this approach to result in more accurate and coherent joint audiovisual representations.
Acknowledging the challenges of zero-shot audio captioning, we enhance caption quality using "prefix tuning", a technique commonly employed across domains adapt LLMs diverse tasks in a few-shot manner (19; 20; 21). This tuning process conditions the model with few image-audio caption pairs, prompting it to generate audio-centric captions and thereby boosting performance
across standard audio captioning metrics (22 23 24
25
26
27).
Notably, methodology allows for the use of either audio or audiovisual input by combining or substituting the image encoder with the aligned audio encoder. As result, our approach extends the ability of the VLM audio captioning without compromising its image performance. Moreover, the method achieves significantly improved performances in zero-shot audio captioning, compared to existing approaches.
In summary, our contributions are as follows:
We introduce a novel methodology for unsupervised zero-shot audio captioning through Distribution ALIgnment (DALI), leveraging advanced image captioning models, and suc-

---

cessfully instantiate it with a particular image captioner (namely Llava 1.5), as we obtain state-of-the-arr pero-shot audio captioning results without any supervision from annotated audio data (see Section 5).
To bridge the "modality gap". we propose and study two multimodal token distribution alignment methods exploiting MMD or Optimal Transport (OT). The ntter-never used before in this context-is enhanced with a cross-attention mechanism for greater robustness.
We introduce the use of prefix tuning to guide the image captioner towards audio captioning, adapting the model with a few shots of image and audio-caption pairs (only using the textual descriptions of sound and not the audio signals).
Finally, our method supports both audio and audiovisual inputs, extending the image captioner's capabilities without compromising its performance in image analysis tasks.
2
Related works
Vision Language models Large Vision Language Models (VLMs) typically use a pre-trained LLM that processes both visual and textual information. The image encoder usually consists of a ViT (28) trained via multimodal contrastive learning using millions of image-text pairs such as the ones used in CLIP (4). Integrating visual information into the LLM is usually done by concatenating image tokens extracted by vision encoder and then further transformed by an MLP, with textual tokens from a prompt. This straightforward architecture can be easily adapted to other modalities by simply replacing the input tokens to accommodate the new input modality.
Audio Language models Recently, similar paradigm emerged in the audio application domain. Various works rely on both I.MMs and audio encoders trained jointly on millions of audio-text pairs, similarly to CLAP (6). Listen Think and Understand (LTU) (2)). Pengi (L)) QwenAudio (g) and AudioFlamingo (0) rely on a language model fed with tokens from both audio and text modalities. These large models require substantial volumes of training data. For instance, AudioFlamingo has been trained on nearly all publicly available labeled audio data. One limitation is that the quality of the audio-text data is generally not comparable to that of image-text data. This situation made worse by the fact that audio data labelling is inherently difficult due to the potential overlap of multiple sound sources at different acoustic levels, which sometimes makes it hard to perceive some of the audio classes in presence. Despite this limitation, AudioFlaminge and similar models exhibit remarkable performance. To further enhance the capabilities of languaae models of paramount importance to explore alternative paradigms beyond strong reliance on (often unreliable) annotations and supervised learning. This is the goal pursued in this work.
Audio captioning Training large audio captioner models, as mentioned previously, requires large volumes of audio and textual descriptions and substantial computational resources. To address this limitation, some approaches leverage existing robust backbones to enable zero-shot audio captioning. ZerAuCaps (30) uses CLAP (an 80.8M 4MM-arramttr model trained on 3.4M audio-text pairs) to infer potential classes from predefined word bank (45 words) before using them to prompt an LLM (OPT (31) 1.3B), generating multiple plausible captions. The final caption is the closest to the audio in the CLAP space. However, this method is constrained by the finite set of classes in the word bank, limiting its application to predefined scenarios. The most closely related work to ours is that of Shaharabany et al. (32) who also performed unsupervised zero-shot audio captioning. The authors adopted the audio encoder of ImageBind (33) (86M parameters trained on AudioSet), a multimodal model trained align various modalities in the image space via contrastive loss. They fine-tuned a GPT-2 (44) model (117M parameters) fed with ImageBind audio tokens using two distinct loss functions for the captioning task: an audibility score ensures the generation of audio-centric captions without visual artifacts. while an ImageBind score measures the similarity between the generated caption and the audio. Despite its innovative approach, its performance falls short compared to Zer primarily due to ImageBind's 's less effective handling of audio.
Audiovisual Alignment Multimodal alignment aims to encode information from different modali- ties such as text, image, or audio into shared semantic space, enabling cross-modal relationships and multimodal retrieval (e.g., retrieve the image closest to a caption) (3). Despite the abundance of web video datasets, aligning the audio and image/video modalities remains challenge. The conventional contrastive loss (35) often fails to effectively co-train the encoders. Unlike the relatively clean associations in image-text pairs, audio-image image pairs present greater degree of variability, with
3

---

Training
a
Audio captianing via visual inguls and prefix luning
Toxen distribution alignment
Audio captinnngg via loken distr distriuuiion alignment
grest.
* dog 18, barking
LLM
Stage
Image Encoder
Audio Encoder
Audio Encoder
Encedder
Training
Inference
d
Audio-iisual dist lation
e
Audio captioning via audio or audio visual Inguts
dogg
backing.
NILL
Stage 2
LLM
LLM
LUM
Image
Audio takens
Prompt tokere
MLP
MLP
Learnable
Image Encoder
Audio Encoder
Audio Encoodr
Audio Encoder
Repisca
Encoder
Figure 2: Overview of the proposed approach. In the first stage, prefix tuning is performed using a few (image, audio-caption) pairs (1-a). Additionally, the audio backbone is aligned with the image backbone (1-b) through distribution alignment. Audio captioning can then be performed by switching the image backbone with the audio backbone and adding the prefix tokens (I-c). In second stage, visually-informee audio captions are generated using both audio, image, and prefix tokens. The MLP mapping the audio encoder to the language model is then fine-tuned with these pseudo captions (2-d). The final inference for audio captioning, using audio or audio visual inputs, is performed by forwarding the aligned audio backhone's s output through the trained MLP to obtain the LLM input (2-e).
the content described in the audio often differing from that depicted in the corresponding image or video. Consequently, stable pre- pre-training needs augmenting the contrastive loss with reconstruction loss (36 37 38) reducing the similarity of the representations between the two modalities.
3
Method
Current Large Vision Language Models (VLMs) exhibit a sophisticated understanding of concepts related to real-world objects in the visual domain. This capability arises from optimizing the language model, already containing lot of intrinsic knowledge, as extract additional information from image features and learn how to describe the (visual) world. We hypothesize that the knowledge gained through this process sufficiently general be transferable across different modalities. To test this hypothesis, we developed methodology that employs an VLM's s image captioner to generate audio captions from images, audios and audiovisual inputs, while preserving the original performance of the image captioner. Our methodology stands out for its applicability to any modality requiring integration with image data. Here, we focus on audiovisual alignment for audio captioning, benefiting from the abundance of audiovisual content in videos from web data.
3.1 Proposed Framework Overview
An image captioning system outputs descriptions of objects in an image, such as "A dog in the grass," providing visual cues. To adapt this system for audio captioning, we need it to focus on audible information instead of visuals. For example, given an image of a dog, the audio caption should instead be "A dog barking.". Our methodology enables audio captioning from image captioner by keeping both the image encoder and LLM frozen, ensuring that the image captioner original

---

performance is preserved. In our work, we choose Llava ① (Apache License 2.0) as an instance of an image captioner, for its proven effectiveness. We use the CAV-MAE (36) (BSD 2-Clause License) audio model as our audio encoder, since it is already roughly aligned with an image backhone, which expected to help reach faster convergence.
As seen in Figure 2 we follow a 2-stage process, First, we use prefix tuning to start re-tasking the system for audio captioning, learning tokens from a few image-caption pairs, ensuring that the textual descriptions target the audio content (stage I, step (a)). We also align the token distribution of an audio encoder with that of the image encoder (stage 1, step (b)), then replace the image encoder with the aligned audio encoder for direct audio captioning from audio files (step (c)). We can also combine both encoders for visually informed audio captioning, using both audio and visual inputs. This setup generates audio captions to fine-tune the MLP that maps audio tokens to the language
model embedding space, a process we call audiovisual distillation (stage 2, step (d)). The fine-tuned MLP then replaces the original MIP, generating robust audio captions from audio or audiovisual inputs (step (e)). In the following, we zoom on specific aspects of the proposed methodology.
Prefix-tuning with image-caption pairs Image captioners, while effective for image analysis tasks, struggle produce accurate audio-centric descriptions when prompted to describe sounds from images. For example, when asked, "What sounds can the objects in this image generate?", the descriptions provided heavily reflect the visual characteristics of the objects. Instead of focusing on sound-related details, their outputs are dominated by visual descriptions, which are considered artifacts in the context of audio captioning. To address this problem, we adopted a prompt tuning strategy (39), aiming to guide the model to generate captions that exclusively describe audible events as depicted in Figure a. Prompt tuning involves learning a portion of the prompt tokens: the prefix, that guides the model perform specified task (39). This method allows the rest of the model to remain frozen, and by simply removing the learned tokens, the model is able to perform its original task with the same performance. This approach requires only small number of parameters to be trained and can be effectively trained using a few examples, a process akin to few-shot adaptation. We employed this technique to re-task the model to perform audio captioning. We trained the additional tokens using image-text pairs, with the text describing the sounds associated with the images i.e., audio captions. Specifically, used images from AudioSet vidcos and corresponding captions from AudioCaps (40) (MIT). We used negative log-likelihood as the loss function for the prefix tuning:
N Σ
Eprefix =
log po PC (C Ch ,T,... C9-1):
with n and j being respectively the image and caption token indices, N the number of elements in the batch, the number of predicted tokens, the index referring to the vision tokens (from 1 to n,), Cj the j-th caption token and the parameters of the model. Using the prefix tokens, the image captioner can now output more plausible sound descriptions.
Audio captioning via token distribution alignment Aligning the distribution of the audio encoder with that of the image one allows for substituting one encoder with another and therefore conditioning the generation either on the audio or image modality. We align the distributions (Figure [ step (b)) using either Maximum Mean Discrepancy or Optimal Transport. More details are provided in Section 3.2. Using the aligned audio backbone, as represented in Figure 2 step (c), we feed the language model with the audio tokens along with the prefix, which enables the model to perform audio captioning. Additionally, we incorporate both the image and audio tokens, to give the ability to the model to perform audio captioning fusing visual and audible cues. This setup allows to infer and generate, visually informed, audio captions for AudioSet balanced (l2), a standard subset of AudioSet, containing ~ 20k videos, curated such that they involve the same number of samples for each class.
Audiovisual distillation The captions generated from audiovisual inputs serve as pseudo-labels to supervise the fine-tuning process of an audio captioner with an audio-only setup. This process refines only the MIP that maps the audio tokens to the LLM embedding space. Training employs negative log-likel likeliiel loss, as per equation ( albeit without conditioning on image tokens. Instead of relying on limited number of real captions, pscudo-labels are employed. This procedure is
5

---

illustrated in Figure
step (d), and optimized through the following loss:
N
Ldisil
log Pogpo(Ce (CT)A A (0.,,,y-11
(2)
with A referring to the audio tokens (from 1 to na))
Audio captioning via audio or audiovisual inputs With all components trained, our model can perform audio captioning using the aligned audio encoder and fine-tuned audio MLP, or visualy- informed audio captioning by feeding to the LLM the image tokens in addition to the audio tokens (shown in Figure step (e)). Notably, the original image captioning task can still be performed at its original performance level by using only the image tokens, without the audio.
3.2 Token distribution alignment
Many studies on multimodal alignment typically use contrastive loss on the averaged output tokens (14 41). This method merely aligns token mean-statistics, potentially leading information loss, when the full token distribution is needed. Llava (I) for instance, makes use of the full token distribution as input, naturally creating the need for full token distribution alignment, to allow for swapping the pretrained encoder with new one targeting new modality. Moreover, contrastive learning faces the so-called "modality gap" problem where each modality is encoded distinct sub-regions of the embedding space (see Figure 4 For all these reasons, replacing image tokens with audio tokens obtained by alignment through a contrastive learning approach may yield undesirable responses from the language model. In light of these issues, we propose to learn the full image token distributions, rather than relying solely on contrastive learning, so as to facilitate knowledge transfer. We refer distribution alignment framework as DALI which stands for Distribution ALIgnment. We study two variants of distribution matching.
Modality alignment through MMD Maximum Mean Discrepancy (MMD) serves robust measure of dissimilarity-or more specifically, discrepancy- two probability distributions. MMD has been shown to be effective in distribution alignment, notably, but not exclusively, for domain adaptation (4)45) 4%). It computes the distance between the expected values of two distributions, denoted by P and Q, within feature space characterized by mapping & X H where H. is reproducing kernel Hilbert space (RKHS):
MMD(P. Q) = ||xx [x()) - 1 (Ey=ol@(Y)))ll
In practice the MMD is computed using the kernel trick (44). In our work, we consider the Gaussian Radial Basis Function (RBF) kernel and use MMD as a loss function to directly align the audio and image token distributions. We term this distribution alignment through MMD: DALIMMD-
Modality alignment through Optimal Transport Optimal Transport (OT) stands out as powerful method for aligning probability distributions (45). Despite its successful application across various domains such as unsupervised domain adaptation (46). image generation (47). and style transfer (48), our context,
its application to multimodal representation alignment remains relatively unexplored. where the alignment of audiovisual modalities is paramount, OT emerges as natural solution for its intrinsic capacity to integrate the geometry of the underlying space, enabling optimal distribution matching. Discrete optimal transport considers two distributions z E R and y I where Nand M represent the number of samples in each distribution. These distributions are represented by their empirical measures: a = Σm OuoOn; and 3 = Σ Bzōny, and OT seeks a coupling E IT(, between them that minimizes a transportation cost. The problem can be formalized as:
OT(a, min (7, D)F:
(4)
7ElI(a,,))
where D NxM is the cost matrix, (r denotes the Frobenius dot product, and N represents the transportation coupling matrix. The minimum of the optimization problem can be interpreted distance (45). When considering the square of the la norm, the distance is known as the Earth Mover Distance (EMD). In this work, we align the audio and image token distributions using EMD and refer to this approach Distribution ALIgnment through Optimal Transport (DALIor).

---

Improving cross-modal correspondence with attentive distribution alignment Optimal transport typically applied assuming uniform distributions of the weights {a, and [3, However, in audiovisual content, objects producing sound are often occluded, with the visual information serving to complement the auditory content. The classic formulation of OT enforces strict mass preservation, needing that all mass from the source distribution is transported to the target distribution. This constraint becomes problematic in audiovisual alignment, where not all tokens from one modality may find match in the other. While alternative methods exist to circumvent these constraints, such as Unbalanced Optimal Transport (UOT) (49), which replaces this rigid requirement with a soft penalization term, the tuning of such hyper-parameter poses significant challenges (50). Alternatively, we propose learning the weights Og and By via a cross- cooss-tttnntion mechanism which offers as well a robust approach. This mechanism, attending to both modalities, enables learning which objects and/or sound-related tokens are present in both modalities. Tokens encoding modality-specific information, e.g., an occluded source, this approach promotes the alignment of modality-shared content. We provide the details of this cross-attention mechanism in Appendix F. It is important to note that to avoid the trivial solution that would put all the weights to 0 except the one of the closest tokens, we regularize the entropy of the weights.
object apparently not generating a sound, receive lower weights. Thus,
The final loss consists of a weighted sum of the optimal transport distance and the entropy of distribution weights:
(5) where H(-) denotes the Shannon entropy and OT y, a<"1, BAat) represents the optimal transport between All and [AA,, the weights at the output of the attention mechanism, with the cost matrix computed from x and y. An intuitive depiction of the method is illustrated in Figure 3 Note that the cross-attention layers are only used for training and discarded afterward. We refer to this alignment Att method as DF oT-
C(z, y, a A" ,a = OT
(a. y, a At ,Ath) X(H(a (1)(a>")) + H(B*")),
Token distribution alignment
Token distribution alignment with cross attention
Att
Aadi silrn
Image Encoder
Audio Encoder
Image
Audio Encoder
Encoder
Att
Figure 3: Multimodal distribution alignment through optimal transport. The audio and image tokens are used to compute the cost matrix, while two separate cross-attention layers estimate the weights All and BAu
4 Experiments
Our experiments focus on AudioSet, a widely explored large-scale collection of videos from YouTube. We evaluate our proposed approach two popular and publicly available human- annonttdd audio captioning datasets: AudioCaps and Clotho (Tampere University licence), using conventional audio captioning metrics. We also provide details of our training and fine-tuning procedures.
Datasets AudioSet (12) is a dataset (license CC BY 4.0) composed of 2 million YouTube videos with annotations indicating the presence of environmental sound events at the clip level (without precise time location), spanning 527 classes. The dataset includes reliability scores for annotations, some below 50%. Despite diverse range of classes, significant portion predominantly features music or speech content. Many videos show notable disparities between their visual and auditory content, thus attempting align audio and image representations without any pre-filtering of the content is challenging. Similarly to Nagrani et al. (51). who filtered the dataset by balancing the classes, we train our models on a subset of 500k videos from AudioSet. We chose them by computing the similarity between the noisy audio labels and an image caption generated by (L) (chosen for the conciseness of the captions), to remove the biggest discrepancies between images and audios. More details on this filtering process are given in Appendix C. For each video, we extracted 10 frames (the first of each second), and we randomly chose, during training, one of these 10 frames for each video.
7

---

Alignment
METEOR1
ROUGE-LT
SPIDErt
Image only"
0.1324
0.3008
0.1499
Ours (Audio-Only)
DA
0.1277
0,3106
0.1592
Contrastive
0,1215
0.2914
0,1524
DALlor
0.1332
0.2923
0.1422
DALIMMD
0.1346
0.3025
0.1360
Ours (Audio+Image)
DALI
0.1257
0.3061
0.1946
Shaharabany et al. Salewski et al.
ImageBind
n.a
0.082
CLAPP*
0.123
0.331
0.183
Ablation of audiovisual distillation (Stage 2-d)
DALLOT
0.11
0.2901
0.1443
DALLMMD
0.1330
0.3018
0.1385
DALLoT
0.1062
0.2709
0.1183
Contrastive
0.0728
0.1838
0.0871
Table I: Audio captioning performance on AudioCaps test set. Our results are obtained using 16
Limage,audio-caption pairs for the prefix tuning phase. (*): No alignment. (**): Trained
supervised fashion using audio-caption pairs. Results ordered by SPIDEr score.
Evaluation metries We evaluated our system on all known audio captioning datasets which are publicly available: AudioCaps (40) which contains AudioSet audio samples captioned by humans, and Clotho (53), which is composed of audio samples from FreeSound, also annotated by humans. Performance measured using 3 standard captioning metrics: METEOR (24), ROUGE-L (23), and SPIDEr (27), which combines both the CIDEr (25) and SPICE (26) scores. Additional evaluation metrics are available in Appendix G. We evaluate our model in the context of zero-shot audio captioning, comparing it against other models in the field. All the training details and hyper parameters are given Appendix E.
5 Results and discussion
Audio Captioning performance Table presents the audio captioning performance of our methods on the AudioCaps test set, using 16 image captions to train the prefix tokens. We show the relatively small importance of the number of pairs in the audio captioning learning process in Appendix A. Our study compares the alignment variants considered our zero-shot audio captioning framework against the current satte-of-the-are art methods, which employ audio encoders trained in either supervised or an unsupervised manner. Our results indicate that the alignment through contrastive learning without audiovisual distillation does not fully match distributions, compared to DALI. Indeed, even without audiovisual distillation, DALIMMD and DALL Au significantly outperform Shaharabany et al. (32), this system being our closest competitor, as relies on a backbone trained an unsupervised fashion.
Without audiovisual distillation, DALIMMD performs comparably to DALI however, the latter improves with audiovisual distillation. We hypothesize that this occurs because DALIMMD is trained to learn the complete image token distribution, while DALI only learns a part of it (due to the attentive optimal transport mechanism). Therefore DALI features can take more advantage of the combination with image tokens. We verify qualitatively this hypothesis in the Discussion paragraph below. Interestingly, both DALIor and the contrastive backbone benefit from the audiovisual distil- lation. Given the relatively oow-perfomaance scores prior the distillation process, that they primarily Iearn from the image captioner (as the audiovisual pseudo captions tend to rely solely on visual features due the inadequate audio representations) which shown to be already performing well for audio captioning (first line of results). We hypothesize that the second stage is effective even for poorly aligned backbones because the distillation occurs on subset of AudioSet where the images align well with the audio. If this distillation were applied to the entire dataset, the performance of these backbones might decline substantially. These hypotheses will require further investigation in future work. Lastly, it is important to note that feeding the image tokens in addition to the DALI ones, increases even more the performances, leaving the door open for further improvements.
plausible
Aul
8

---

Discussion Unlike standard contrastive meth- ods, we do not average the output of the back- bone, which allows us to employ token-level distribution alignment methods that do not rely on negative samples. Both Optimal Transport (OT) and Maximum Mean Discrepancy (MMD) focus on directly aligning the embedding's ' dis- tributions. OT minimizes the cost of transporting one distribution to another, aligning them with- out the need for negative samples, and avoiding the issue of pushing different modalities into separate subspaces. Similarly, MMD measures the distance between kernel-projected mean em- beddings, aligning the distributions without neg- ative sampling. These methods promote more cohesive embedding space, closely aligning dif- ferent modalities and thus avoiding the modality gap inherent in contrastive learning approaches.
PCA of the multimodal embedding space
13-
CLIP Image
DALJOT
DALJAMD
20-
Contrastive
Dimension
DAIJor
5
-15
-10
Dimension
10 15
Figure 4: AudioCaps average tokens distribution. While contrastive learning maps the audio in a space separate from the image ones, MMD and optimal transport project in the same part of the space. The model trained using attentive optimal transport projects the audios in a space closer to the image, with marginal overlap.
Indeed, Figure shows a Principal Component Analysis (PCA) of the global representation (the average of the output tokens) of audio and im- ages from various backbones on a subset of the AudioCaps test set. In our audiovisual experiment, we observe a phenomenon akin to the one demonstrated by Liang et al. for image-text contrastive learning. Specifically, models encode each modality in a narrow cone, resulting in the so-called modality gap where the embeddings of each encoder are confined to subregion of the embedding space. However, our backbones, which are trained using distribution alignment, do not exhibit this issue. The embeddings generated by both the DALIMMD and DALLoT encoders closely resemble those of the image modality. In contrast, the DALIØT ones, while located in the same region of the embedding space, remain distinct from the image tokens and exhibit minimal overlap. This indicates DA and image tokens are complementary and provides an explanation for why the former benefits more from audiovisual distillation than the DALIMMD model.
The root cause of this discrepancy lies the training process: the cross-attention-bass DALLOT model is trained using only to the matching audiovisual tokens, whereas DALI model is trained on the complete image distribution, which inherently includes a degree of image bias.
To verify this claim, we examined whether the aligned backbone with DALIMMD (after Stage I) exhibits image bias. We prompted the captioner with "What can you see? Answer concisely" along with the audio tokens without the tuned prefix to avoid constraining it to generate only audio-related captions. Table presents multiple captions generated using the DALI and DALIMMD backbones. As shown, the encoder trained with MMD often produces captions with notable visual biases, which are absent in the outputs of D Aul These results confirm our hypothesis and align with the visualizations in Figure 4 Note that the captions may seem incomplete as we ask the model to answer concisely, focusing only on the main element of the scene Complete generated captions by different models are provided in Appendix D.
Ground Truch
DALhan
DALI
woman is speaking from a microphone
women in black shir and a and scarf speaking into a microphone.
A woman
A make voise and muchine buxting
A man is using a drill to muke a hole is piece of wood. A person and a dog standing in frent of lange machine.
A persoe's hand holding tnol. Bell
A bell is ringing
A long burp ends sigh A choinson cutling wond emiking A speaking min lighity falls followed by chander
A man with beard and mustache.
A man
A man cutting
trink with a chaisssow.
A lawn newer
A man wenring black shitt and grey suit.
A Ainn
vehicle engine rewving and squealing tines
car meing track.
car:
Table 2: Ground-truth audio captions, and captions generated by our audio models (after stage I, without prefix tokens) by asking "What can you see answer concisely".
Assessing generalization ability The Clotho dataset presents a greater challenge than AudioCaps, primarily due to the complexity of the captions, which is reflected in the relatively lower performance of all the methods. The audio tracks in AudioCaps are sourced from AudioSet, thus the distribution of audio events resembles that of the data on which our models are trained. In contrast, Clotho's o's audios

---

Alignment
METEORT
ROUGE-L1
SPIDErt
Ours
DALIMMD
0.1067
0.2993
0.0655
DALLOT
0.1008
0.2850
0,0625
Contrastive
0.0918
0.2490
0,0620
DALlor
0.1049
0.2765
0.0574
Salewski et al.
CLAP
0.094
0.254
0.097
Ablation of audiovisual distillation (Stage 2-d)
DALIMMD
0.1058
0,2768
0,0640
Contrastive
0.0630
0.1777
0.0377
DALI
0.0666
0.2101
0.0355
DALlor 0.0617
0.1825
0,0299
Table 3: Clotho audio captioning performance. Similarly AudioCaps, DALLOT is performing. however, DALIwMD gives slightly better results. The bias learned by matching the complete image distribution seems to be beneficial for out-of-domain samples. (*): Trained in supervised fashion using audio-caption pairs.
are sourced from FreeSound, a dataset distinct from AudioSet. Consequently, our model, having been trained solely on AudioSet's audio excerpts and images, may encounter unfamiliar concepts in Clotho, thereby testing its capacity for generalization. pabset] the models' performance on Clotho's test set. The observed trends are consistent with those on AudioCaps where DALIMMD does benefit from audiovisual distillation, while DALLOT does. However, the absolute results differ, as DALI marginally outperforms DALI even after the second training stage. We posit that the image bias inherent in DALIMMD becomes beneficial when confronted with out-of-distribution data. Indeed what may be regarded as outoo--distribution for the audio backbone might have been seen by the image one (due to its bigger training set). Consequently, the learning of the image backbone's 's bias could potentially be beneficial. Notably, our method outperforms Saleswki et al., despite the fact that the latter employs backbone trained using millions of audio-text pairs (CLAP). It is important to note that our method differs from Saleswki et al. in that we perform unsupervised instead of supervised zero-shot audio captioning.
6 Conclusion
We presented novel methodology for unsupervised zero-shot audio captioning, effectively leveraging advanced image captioning models, such as Llava. We adapted it to the audio captioning task by making use of prefix tuning and using innovative token distribution alignment techniques (using MMD or optimal transport with cross-attention mechanism) that successfully bridged the modality gap between audio and visual inputs. Our comprehensive evaluation, both quantitative and qualitative, demonstrates that method achieves state-of-the-an results in unsupervised zero-shot audio captioning. Remarkably, our approach, not using any audio recordings, not only matches but even sometimes, exceeds the performance of existing methods that rely on backbone trained with extensive audio-text data such as CLAP. Moreover, our method requires only raw videos without any audio annotations, significantly enhancing its potential scalability. This positions our approach promising direction as a leading direction for the future of audio captioning.
Limitations Although capable of addressing partial audiovisual mismatches, our method remains insufficiently robust to complete cross-modal mismatches, such as instances where the image is entirely unrelated the audio. We reserve addressing this limitation for future work. Moreover, to achieve even higher performance in audio captioning, our method would benefit from degree of supervision. The requirement for an external supervisory source (e.g., audio-text pairs) arises due to the method's dependence on audiovisual alignments, which makes it challenging to learn certain sounds because they cannot be seen. For example, almost impossible to generate a caption for the sound of the wind by merely using images.
This work was supported by the Audible project, funded by French BPI. This work was performed using Al resources from GENCI-IDRIS (Grant 2023-AD01 10023-A00111488880 1014885.
Acknowledgement
10

---

References
[1] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee, "Visual instruction tuning." Advances in neural information processing systems, vol. 36, 2024.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, al., "Flamingo: visual language model for few-shot learning," Advances in neural information processing systems, vol. 35, pp. 23716-23736, 2022.
[3] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al., "Obelies: An open web-scale filtered dataset of interleaved image-text documents," Advances in Neural Information Processing Systems, vol. 36, 2024.
[4] Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang, "Tinyllava: A framework of small-scale large multimodal models," arXiv preprint arXiv:2402. 14289, 2024.
[5] Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang, "Llava-phi: Efficient multi-modal assistant with small language model," arXiv preprint (tti::2001023300 2024.
[6] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang, "Clap learning audio concepts from natural language supervision," in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1-5.
[7] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Berg-Kirkpattick, and Shlomo Dubnov, "Large-scale contrastive language-audin pretraining with feature fusion and keyword- to-caption augmentation." " in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1-5.
[8] Xinhao Mei, Chutong Meng. Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. "Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research," " arXiv preprint 233.:22303. 2023.
[9] Yunfei Chu. Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou, "Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models," arXiv preprint arXiv:2311.0799 2023.
[10] Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro, "Audio flamingo: novel audio language model with few-shot learning and dialogue abilities." arXiv preprint arXiv:2402.011 00831, 2024.
[11] Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang, "Pengi: An audio language model for audio tasks," Advances in Neural Information Processing Systems, vol. 36, pp. 10190018108,..11 2023.
[12] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter, "Audio set: An ontology and human labeled dataset for audio events," in 2017 IEEE international conference acoustics, speech and signal processing (ICASSP). IEEE, 2017, pp. 776-780.
[13] Amnb Barua, Mobyen Uddin Ahmed, and Shahina Begum, "A systematic literature review on multimodal machine learning: Applications, challenges, gaps and future directions," JEEE Access, 2023.
[14] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al., "Learning transferable visual models from natural language supervision," International conference on machine learning. PMLR, 2021, pp. 8748-8763.

---

[15] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou, "Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning," Advances in Neural Information Processing Systems, vol. 35, pp. 17612-17625, 2022.
[16] Peiyang Shi, Michael C Welle, Märten Björkman, and Danica Kragic, "Towards understanding the modality gap clip." " ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023.
[17] Sijie Mai, Ying Zeng, and Haifeng Hu, "Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations"" IEEE Transactions on Multimedia, 2022.
[18] Shraman Pramanick, Aniket Roy, and Vishal M Patel, "Multimodal learning using optimal transport for sarcasm and humor detection," in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 3930-3940.
[19] Zonghai Yao, Ahmed Jaafar, Beining Wang. Zhichao Yang, and Hong Yu, "Do physicians know how to prompt? the need for automatic prompt optimization help in clinical note generation," 2024.
[20] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu, "Learning to prompt for vision-language models," International Journal of Computer Vision, vol. 130, no. 9, pp. 2337 2348, 2022.
[21] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu, "Conditional prompt learning for Dision-language models," in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2022, pp. 16816-16825.
[22] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, "Bleu: a method for automatic evaluation of machine translation." in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311-318.
[23] Chin- Yew Lin and FJ Och, "Looking for a few good metrics: Rouge and its evaluation," in Ntcir workshop, 2004.
[24] Satanjeev Banerjee and Alon Lavie, "Meteor: An automatic metric for mt evaluation with improved correlation with human judgments," in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 2005, pp.
65-72.
[25] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh, "Cider: Consensus-based image description evaluation." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 4566-4575
[26] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould, "Spice: Semantic propositional image caption evaluation," in Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14. Springer, 2016, pp. 382-398.
[27] Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy, "Improved image captioning via policy gradient optimization of spider," in Proceedings of the IEEE international conference on computer vision, 2017, pp. 873-881.
[28] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al., "An image is worth 16x16 words: Transformers for image recognition at scale," in International Conference on Learning Representations, 2020.
[29] Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James R Glass, "Listen, think, and understand," The Twelfth International Conference on Learning Representations, 2023.
[30] Leonard Salewski, Stefan Fauth, A Koepke, and Zeynep Akata, "Zero-shot audio captioning with audio-language model guidance and audio context keywords," arXiv preprint arXiv:231 1,0836, 2023.
12

---

[31] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, al.. "Opt: Open pre-trained transformer language models," arXiv preprint arXiv:2205.010m. 2022.
[32] Leonard Salewski, Stefan Fauth, A Koepke, and Zeynep Akata, "Zero-shot audio captioning with audio-language model guidance and audio context keywords," arXiv preprint arXiv:2311.08396, 2023.
[33] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra, "Imagebind: One embedding space to bind them all," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023,
pp. 15180-15190.
[34] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei. Ilya Sutskever, Jeffrey Dean, and Sanjay Ghemawat, "Language models are unsupervised multitask learners." " in OSD/04: Sixth Symposium on Operating System Design and Implementation, pp. 137-150.
[35] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. "Representation learning with contrastive 2018.
predictive coding." arXiv preprint
[36] Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R Glass, "Contrastive audio-visual masked autoencoder," in The Eleventh International Conference on Learning Representations, 2022.
[37] Mariana-luliana Georgescu, Eduardo Fonseca, Radu Tudor lonescu, Mario Lucic, Cordelia Schmid, and Anurag Arnab, "Audiovisual masked autoencoders," in Proceedings of the IEEF/CVF International Conference on Computer Vision, 2023, pp. 16144-16154.
[38] PaoYao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Yanghao Li, Shang-Wen Li, Gargi Ghosh, Jitendra Malik, Christoph Feichtenhofer, et al., "Mavil: Masked audio-video learners," Advances in Neural Information Processing Systems, vol. 36, 2024.
[39] Xiang Lisa Li and Percy Liang. "Prefix-tuning Optimizing continuous prompts for generation." in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 4582 4597.
[40] Chris Dongjoo Kim, Bycongchang Kim, Hyummin Lee, and Gunhee Kim, "Audiocaps: Gen- erating captions for audios in the wild," in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), 2019, pp. 119-132.
[41] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer, "Sigmoid loss for language image pre-training," " in Pruceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 11975-11986.
[42] SiYa Yao, Qi Kang, MengChu Zhou, Muhyaddin Rawa, and Aiiad Albeshri, "Discriminative manifold distribution alignment for domain adaptation." " IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 53, no. 2, pp. 1183-1197, 2022.
[43] Jindong Wang, Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, and Philip S Yu, "Visual domain adaptation with manifold embedded distribution alignment," in Proceedings of the 26th ACM international conference on Multimedia, 2018, pp. 402-410.
[44] Bernhard Schölkopf, "The kernel trick for distances." " Advances in neural information processinnn systems, vol. 13, 2000.
[45] Gabriel Peyré, Marco Cuturi, et al., "Computational optimal transport: With applications to data science," Foundations and Trendy@ in Machine Learning, vol. 11, no. 5-6, pp. 355-607, 2019.
[46] Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty, "Deepjdot: Deep joint distribution optimal transport for unsupervised domain adap- tation." in Proceedings of the European conference on computer vision (ECCV), 2018, pp- 447-463.
13

---

[47] Martin Arjovsky, Soumith Chintala, and Léon Bottou, "Wasserstein generative adversarial networks," International conference on machine learning. PMLR, 2017, pp. 214-223.
[48] Pan Li, Lei Zhao, Duanqing Xu, and Dongming Lu, "Optimal transport of deep feature for image style transfer," in Proceedings of the 2019 4th International Conference on Mullimedia Systems and Signal Processing, 2019, pp. 167-171.
[49] Thibault Séjoumé,, Gabriel Peyré, and François-Xave Raviier Vialard, "Unbalanced optimal transport, from theory to numerics," Handbook of Numerical Analysis, vol. 24, pp. 407-471, 2023.
[50] Kilian Fatras, Thibault Séjourné, Rémi Flamary, and Nicolas Courty, "Unbalanced minibatch optimal transport; applications to domain adaptation." in International Conference on Machine Learning. PMLR, 2021, pp. 3186-3197.
[51] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun, "Attention bottlenecks for multimodal fusion," Advances in neural information processing systems, vol. 34, pp- 14200-14213, 2021.
[52] Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi, "Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models," in International conference on machine learning. PMLR, 2023, pp. 19730-19742.
[53] Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen, "Clotho: An audio captioning dataset," in ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). IEFE, 2020, pp. 736-740.
[54] Yujia Li, Kevin Swersky, and Rich Zemel, "Generative moment matching networks," in International conference on machine learning. PMLR, 2015, pp. 1718 -1777.
[55] Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélic Boisbunon, Stanislas Chambon, Lactitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. "Pot: Python optimal transport," Journal of Machine Learning Research, vol. 22, no. 78, pp. 1-8, 2021.
14

---

The appendix is organized as follows: The first part presents additional experiments that aid in understanding the behavior of our encoders, followed by a detailed explanation of the pre-filtering process performed on AudioSet and the training parameter details.
A Impact of the number of image captions
To determine whether the tuned prefixes only specify the task and help avoid visual artifacts in the captions, we analyzed the performance of the method by varying the number of image- audio caption pairs. Figure (eguureSjllune illustrates Filusrass that only 16 captions are required to achieve good performance, and increasing this number does not lead to further improvement. In fact, while the SPIDER, CIDEr, and ROUGE metrics remain stable, the SPICE, METEOR, and BLEU scores even show slight decline. This suggests that prefix tokens only serve to specify the task to the language model and do not contribute learning the captioning task itself. We hypothesize that this prefix tuning could potentially be replaced by carefully chosen hand-crafted prompt.
Spider Scores
Cider . Scores
Houge I
Scores
0.16
0.54
0.12
0.00
024
0.88
D4Ulao
DAU/O
0.2
DAUWE
0.99
Irage
0.10
mage
0.2
Image
Spice Scores
Neteor Scores
Bieu Scores
Diu
0:355
0:300
0.759
0.723
09290
0.200
DAUioo
Dille
8.15
mage
Image
Figure 5: Captioning performances according to the number of image captions. After 16, the performance does not improve, indicating that the prefix tuning does not play an important role the learning process, it just specifies the task.
B Deeper analysis of the learned distributions
[ a 2-D projection through PCA of CLIP image tokens along with the audio tokens learned by DALI DALIMMD, and contrastive learning for the audiovisual pair shown in the figure ("A sound of a guitar playing")) The contrastive learning method maps the audio tokens to a very restricted subset of the space, without any overlapping with the image distribution. Interestingly, while DALIMMD only learns to align the expected value of the distribution, it appears to fit the full image distribution quite well. A exhibits similar behavior but is less noisy: the tokens are either very close to the image tokens or much further away. This behavior is inherited from the attentive optimal transport, which assigns low weights points considered mismatches.
15

---

FCA the contrastive
cantrath rodd
Dimension 8
Dimension of ambedding
embedting
CuP mmage Deuff
Drmonston 2
Dimersion
Figure 6: Audio and image tokens distribution. While contrastive learning encodes tokens in completely separated subspace, the MMD learns a distribution similar to the image one, noisier. The attentive optimal transport fits more the image distribution, except in some points that are mapped further.
Figure 7 illlustrates the optimal transport weights computed by the cross- cooss-attention mechanism for the given image and its associated audio (an emergency vehicle siren). The size of the circles represents the weights. Although the overall distribution appears uniform, closer inspection of the bottom right section reveals that many tokens have scores close zero. These tokens likely represent the road, which does not produce any sound and thus has corresponding match in the audio tokens. Consequently, these tokens are not considered in the calculation of the transportation coupling
"An emergency vehicle siren"
Image
Figure 7: DALLON All Cross-Attention scores. The size of the dots represents the weights of the transport defined by the cross-attentions. The image and its associated audio show a partial mismatch: the audio only contains the siren sound and the image also shows the road, All the tokens with low weights belong to the same part of the space which might indicate that they represent similar information such the road.
C AudioSet pre-filtering to limit audiovisual discrepancies
As AudioSet primarily focuses audio rather than audiovisual content, it features audio events that may be unrelated to the visual event depicted. For example, many vidcos feature music as an audio event while displaying a static image of the album cover.
To address this issue, we filtered out audiovisual discrepancies. To achieve this, we generated captions for the first frame of each second of video using BLIP2, which we then encoded into GPT2-embedding space. Subsequently, we computed the distance between the embedding of each frame's caption and the embedding of the corresponding AudioSet labels. This process is illustrated Figure 8 The average distance across the video frames was used the distance between the video and the audio. We retained the 500,000 pairs with the smallest distance.
16

---

sounds zaz: produce the objects in the amage?
BLIP-2
Image captions
Image Encoder
Frames
avg(
Diatance
||
AudioSet labels
Image Encoder
Figure 8: Discreptancies filtering process: 10 frames the video are captioned by BLIP2, captions embedded in text space and compared to the embedding of the class labels. The average of the distances of the frame considered as the distance between audio and video.
D
Caption examples
Grund Truth
Contiadive
DAII
Rusting Homming enginee
spesking Wand hony people spakknn
vanity
perae engine
spledhnng
speaking ing misspphnn
Doserile the
hesed
speaking
sound ringing
A
fills followed thunder
The sound of buiddng donut
man talking
Man speaking and clinking dishes
talking
choinssw cutting uned cracking
The sound of chainonss and din being muvel.
The sound
chuiraazu catling througha
A woman speaks and meoes
"[OU0""
A parsoz taltting
A male voice and machine buzzing
The sound of metal being cur or saw
A persoz talkting
Table 4: Example of captions from our method after stage using contrastive backbone and DALI
Au
Table 4 shows multiple examples of captions generated by the model using the contrastive backbone and DALI both after the first stage of training. While the captions from the latter are accurate, using the contrastive backbone generates quite noisy/bad captions. This outlines the fact that contrastive learning is not well suited to swap the encoder a modality with the one from another.
Training details
mixture of K kernels spanning multiple ranges (being the average pairwise squared distance between samples multiplied by with with K As far as optimal transport is concerned, the first epoch of the first stage is performed with uniform transport weights. A decreasing value of À is then used: from 500 to 100 at the end of 100 is then used for the remaining epochs. We observed that the training is quite
For the MMD alignment, following (54)
..., K),
the 5th epoch. robust to small variations X, as long as the first epoch is performed with uniform weights. We used the Python Optimal Transport (POT) (55) library compute efficiently the EMD. In the prefix tuning stage, we learned 16 prefix tokens on different numbers of image and audio caption pairs. We used cosine decay and learning rate of 2c-4 for both the prefix tuning and the MLP tuning in the second stage. The prompt used for all the experiment is: "What sound can be heard in this scene ?". We used version 1.5 of Llava: the image encoder being the ViT-L@336px 336px version of CLIP (14,, and the mapping network between image tokens and the LLM embedding space 2 ppyer-perceppron. As for the audio encoder, we start from a pre-trained CAV-MAE (36) model in order to speed-up convergence. This model consists of 12 transformer encoder layers of dimension 768 and one final linear layer that maps the output of the transformers a 1024-dimensional space (dimension of the CLIP encoder). Note that this encoder is way smaller than the image one (88M 428M parameters),
17

---

because the training set used is 2 orders of magnitude smaller than the CLIP one. All the trainings were performed using 4 Nvidia A100 gpus, and all the steps of the training can be performed within 30 hours of compute on those machines.
F
Attentive Optimal Transport details
For DALLOT- computed the weights of the optimal transport by averaging the feature dimen- sion the tokens resulting from the cross-attention and then feeding them through a softmax with temperature (we used 20).
More formally, the weights of the transports a Att and 8 are computed as follow:
(Wo 1)
Att softmax(softmax
Vd JW 1/T);
(6)
D'WK A)T
A softmax(softmax
WW A/T),
vd
with A and I being the audio and image tokens, W W W.Y; W w WV the image and audio projections, respectively, and a
G Additional captioning metrics
Alignment
BLEU,t
METEORT
ROUGE-LI
CIDErt
SPICE
SPIDErt
Image only"
0,0511
0.1324
0,3008
0.2186
0.0812
0.1499
DALI
0.1277
0.3106
0.2441
0.1592
DALbaa
0.0678
0,1346
0.3025
0.1991
0.0729
0.1360
DAIlor
0.0461
0.1332
0.2923
0.1979
0.0866
0.1422
Contrastive
0.0590
0.1215
0.2914
0.2290
0.0779
0.1524
DALI, +Image
0.0677
0.1257
0,3061
0,3063
0.0828
0.1946
Shaharahany Salewski al.
ImageBind
0,086
0.082
0.092
CLAP*
0.068
0.123
0.331
0.281
0.086
0,183
Ablation audiovisual distillation
DALI 0.0523
0.11
0.2901
0.2227
0.0659
0.1443
DAI Dya
0.0694
0.1330
0,3018
0.1976
0.0094
0.1385
DALlor
0.0476
0.1062
0.2709
0.1769
0.0597
0.1183
Contrastive
0.0268
0.0728
0.1838
0.1287
0.0455
0.0871
Table 5: AudioCaps captioning results using 16 image captions. (*): No alignment.
(**):: Trained in
supervised fashion using audio-caption pairs.
Aligument
BLEU,"
METEORT
ROUGE-LT
CIDErT
SPICET
SPIDErT
Ours
DALI
0.0583
0.1008
0.2850
0.0685
0.0565
0.0625
DALlaMD
0.0587
0.1067
0.2993
0.0713
0.0598
0.0655
DAL
0.0396
0.1049
0.2765
0.0567
0,0582
0.0574
Contrastive
0.0358
0.0918
0.2490
0.0682
0.0562
0.0620
Salewski et
CLAP 0.029
0.094
0.254
0.140
0.053
0.097
Ablation audiovisoal distillation
DALE
0.0193
0.0666
0.2101
0.0408
0.0302
DALIxMD
0.0483
0.1058
0.2768
0.0659
0.0621
0.0640
DAL.oT 0.0109
0.0617
0.1825
0.0321
0.0278
0.0299
Contrastive
0.0167
0.0630
0.1777
0.0460
0.0293
0.0377
Table 6: Clotho audio captioning performance. Similarly AudioCaps. DALI
Or is performing,
however, DALLnMD gives slightly better results. The bias learned by matching the complete image distribution seems to be beneficial for out-of-domain samples. (*): Trained in supervised fashion using audio-caption pairs.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We justify by quantitative and qualitative analysis the claims made in the abstract. This can be found in the Results and discussion section as well as in appendices
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We dedicated paragraph this t ('Limitations') part of the Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: NA
19

---

Justification: NA
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the hyper parameters used for the training are given in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification: Data used are already publicly available, both the model and the training code will be made available Github along with tutorial how use it. The preparation of this repo is in progress.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
ptapps..c/uubllc/gguueeeee////// hute://////ddee//deeuubbbbn Cabb///deeeSeeaaaaaaeeea ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the hyperparameters are given in the Appendix, the prefiltering process used on the training dataset well, and the list of files used for training will be made available on the Github repo
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Experiments were too costly to run multiple times, however, a lot of ablations are available and the model is tested on multiple datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The machine on which we run the experiments, as well as the computing time needed is mentioned in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the htcc/uubbic/Ehhcsuuuueeee iinesssieeeieee
NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our research does not involve human subjects participants. We systemati- cally referred to the licences of the licenced datasets that we used (namely CC BY 4.0). We are not aware of any potential misuse of our algorithm, but there may be some.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [No]
Justification:
We are not competent for such a discussion.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not believe that our model can present any risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite both the paper and the licence of each model and dataset we used.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We did not introduced any assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We did use crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research did not involve experiments humans subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "cieDI0eApX", "reviewer": "Reviewer_mzTo", "summary": "The paper under review looks at the problem of captioning of short audio-video clips, leveraging existing multimodal large-language-models. While many strong image captioners exist due to the abundance of image/caption pairs for training data, for clips of non-speech audio, the amount of data available are comparably much less. To build a performant captioner that can produce accurate captions for clips contained either of both audio and visual modalities, the paper proposes a framework that leverages LVLMs, adapts them to produce more audio-centric descriptions via prefix tuning, and most critically \"aligns\" the distributions of the audio and visual tokens to match making the tokenized output from the visual or audio encoder interchangeable. The main novelty is the use of Optimal Transport to align the visual and audio modalities and modifying the loss with cross attention terms to improve the alignment. While this DALI_OT^Att is not a large improvement over DALI_OT or DALI_MMD, the experimentation of these modality alignment methods and demonstrated improvement over a constrastive loss is a novel contribution. The analysis and plot of the multimodal embedding space of the various approach to align the A-V modalities further demonstrates the effectiveness. Overall the paper does demonstrate how LVLMs can be utilized to build a strong A-V captioning system.", "questions": "Has there been any analysis done in comparison to other large, strong multimodal LLMs such as  GPT-4o, Gemini 2 or Llama 3?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Goal of the paper, contributions, execution are all clear. There are a good set of experiments, visualizations of the expected improvement in alignment of the visual and audio token distributions, and overall the application of MMD or OT learning improves the alignment as demonstrated by the experimental results over the dominant baseline approach of contrastive learning.", "weaknesses": "The paper has framed the problem of A/V captioning in a somewhat narrow way of leveraging LLVMs with the only comparison to a strong supervised baseline in CLAP while there are others such as GPTo, Gemini 2 or Llama 3.\n\nSection 3.1 is rather confusing. It seems Figure 2 was revamped and the references to the figure in section 3.1 was not completely updated. Figure 2 should also include a description of steps 2d and 2e in the descriptive caption (the description of 1c needs updating too I believe. The reference on line 177 for 2-c should probably be 2-b; line 190 2-c should be 2-d; and on line 196 2-d changed to 2-e. Overall, it would be clearer to first indicate the stage and then explain it, e.g. \"1-a) prefix tuning the prefix tokens\" rather than the other way around. Also each of the bolded titles in section 3.1 should correspond with a stage in Figure 2 and include the stage number for better clarity.\n\nIn section 3.2, it would be helpful to the reader to provide concrete equations on how to compute the quantities and eqns 1-5. Having these in an appendix would be fine. While its understood that the code will eventually be released, having the equations that the code is intended to implement is always useful for clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. We’re glad to hear that you found our paper’s **goal, contributions, and execution clear** and appreciated the **good set of experiments and visualizations,** along with the improvement in alignment demonstrated by our **application of MMD or OT learning**. We are now addressing your comments in detail.\n\n> Section 3.1 is rather confusing. It seems Figure 2 was revamped and the references to the figure in section 3.1 was not completely updated. Figure 2 should also include a description of steps 2d and 2e in the descriptive caption (the description of 1c needs updating too I believe. The reference on line 177 for 2-c should probably be 2-b; line 190 2-c should be 2-d; and on line 196 2-d changed to 2-e. Overall, it would be clearer to first indicate the stage and then explain it, e.g. \"1-a) prefix tuning the prefix tokens\" rather than the other way around. Also each of the bolded titles in section 3.1 should correspond with a stage in Figure 2 and include the stage number for better clarity.\n\nWe thank the reviewer for pointing out these corrections. We will ensure to make the following changes:\n- Figure 2 caption: Full training pipeline: a prefix tuning is performed using a few image-captions pairs (1-a). In the meantime, the audio backbone is aligned with the image backbone (1-b) through distribution alignment.\n Audio captioning can then be performed by switching the image backbone with the audio backbone while adding the prefix tokens (1-c). \nVisually-informed audio captions are then generated using both audio, image, and prefix tokens. The MLP that maps the audio encoder to the language model is fine-tuned using these pseudo captions (2-d). \nThe final inference is performed by forwarding the output of the aligned audio backbone to the trained MLP to obtain the LLM input (2-e).\n\n- References to Figure 2: line 177 “We align the distributions (Figure 2:1-b)”; line 190 “This procedure is illustrated in Figure 2: 2-d”; line 196 “The resulting audio captioner is shown in Figure 2: 2-e”\n\n>In section 3.2, it would be helpful to the reader to provide concrete equations on how to compute the quantities and eqns 1-5. Having these in an appendix would be fine. While its understood that the code will eventually be released, having the equations that the code is intended to implement is always useful for clarity.\n\nWe agree with those remarks. Since the details of the $\\alpha$ and $\\beta$ computation are already given in Appendix F, and since the OT does not have a closed-form solution, the title of Appendix F will be changed to “Implementation details” and the following precision will be added to ease the understanding of the practical computations of MMD:\n\nThe MMD distance between the audio token distribution ($X$) and the image token distribution ($Y$) is computed as follows: \n\n $\\text{MMD}(X, Y) = \\frac{1}{m^2} \\sum_{i=1}^{m} \\sum_{j=1}^{m} k(x_i, x_j) + \\frac{1}{n^2} \\sum_{i=1}^{n} \\sum_{j=1}^{n} k(y_i, y_j) - \\frac{2}{mn} \\sum_{i=1}^{m} \\sum_{j=1}^{n} k(x_i, y_j)$\nwhere $k$ is the Gaussian kernel. \n\n>**Question**: Has there been any analysis done in comparison to other large, strong multimodal LLMs such as GPT-4o, Gemini 2 or Llama 3?\n\nIt is important to keep in mind that the purpose of our work is to extend vision captioners so they can **also** perform general audio captioning (i.e., describe an audio event—possibly happening outside of the field of view—, such as ‘a dog barking in the street' or ‘a siren heard at a distance'). \nLarge Multimodal models like the suggested ones are trained to perform various tasks, including video captioning. However, they largely rely on the visual modality for that task, and the use of the audio modality is restricted to the speech present in the video. \n\nTherefore, no fair comparison can be made with them as they are not trained to perform the same task as us (general audio captioning).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4IvEia0jG2", "reviewer": "Reviewer_7jnN", "summary": "The paper proposes a method for how to adapt VLM to perform audio captioning. The pipeline is as follows: 1. perform few-shot prefix tuning on images to caption the audio. 2. Use multi-modal alignment methods, namely MMD or OT, to align token space distribution of audio and visual. 3. Distillation for audio-only model from the visually-informed audio caption model. Experiments show that the proposed method outperforms contrastive-based alignment and also the pretrained CLAP model that is used for audio captioning.", "questions": "See Weaknesees.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The main novelty of the paper lies in its usage of MMD / OT to perform multi-modal alignments in audio-visual space, though MMD / OT is not new and has been extensively explored in cross-domain matching literature.\n\n2. The paper is well-written and easy to follow.", "weaknesses": "1. There is lack of theoretical derivation or enough insights for showing why the OT and MMD can outperform the standard contrastive learning. Figure 4 shows the PCA plots for different methods, however it is unclear why MMD or OT is able to stay closer to CLIP encoder. I think more explanations or theoretical justifications are needed.\n\n2. The experiment results seem mixed, in particular Table 2 for Clotho. Contrastive method is better in several metrics without audio-visual distillation, in particular CIDEr and SPIDEr. Also the SPICE performance is very close to the proposed method. These two metrics, CIDEr and SPIDEr from my point of view is also very important since they are designed specifically for caption task. Why would Contrastive outperform the proposed methods in this dataset? But in AudioCaps, the proposed methods beat the contrastive method by a lot. The inconsistency results across these two datasets make me feel confused.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We’re pleased to hear that you found our paper **well-written and easy to follow** and appreciated the **novelty in our use of MMD/OT for multi-modal alignment in the audio-visual space**. In the following, we'll address your comments in detail. \n\n>There is a lack of insights for showing why the OT and MMD can outperform, more explanations are needed.\n\nFirst, regarding the performance gap between our method and the contrastive baseline, we explain the need for distribution alignment to perform this task instead of contrastive learning in Section 3.2, lines 200-206: “Llava for instance, makes use of the full token distribution as input, naturally creating the need for a full token distribution alignment, to allow for swapping the pretrained encoder with a new one targeting a new modality. Moreover, contrastive learning faces the so-called “modality gap” problem where each modality is encoded in distinct sub-regions of the embedding space (see Figure 4). For all these reasons, replacing image tokens with audio tokens obtained by alignment through a contrastive learning approach may yield undesirable responses from the language model.”\n\nConcerning the modality gap that does not appear in our method, it is important to note that contrastive learning brings associated (positive) samples close together while pushing away unpaired (negative) samples. The cited paper [A] showed that by pushing away the negative samples in a high-dimensional space, combined with multiple false positive pairs, multimodal contrastive learning encodes each modality in a separate subspace, creating a modality gap. A more recent paper [B] goes further and shows that this phenomenon is purely due to the behavior of the contrastive loss itself in high dimensions and is not related to multimodal learning. \nNow, optimal transport and Maximum Mean Discrepancy (MMD) do not suffer from this issue, as they do not rely on negative samples but focus on explicit distribution alignment instead. However, their application in audio-visual alignment has been limited due to the common practice of averaging the final representation into a single token, which does not allow for the computation of distribution distances. Employing these losses for their ability to avoid the modality gap is a core motivation behind our work.\n\nWe will modify the beginning of the Discussion paragraph in Section 5 to make clearer our motivations and why our methods do not suffer from such a modality gap. Specifically, we’ll complement it by adding the following paragraph:\n\"Unlike standard contrastive methods, we do not average tokens, which allows us to employ sample-level distribution alignment methods that do not depend on negative samples. Indeed, both Optimal Transport (OT) and Maximum Mean Discrepancy (MMD) focus on aligning the distributions of the embeddings directly. OT minimizes the cost of transporting one distribution to another, effectively aligning the distributions in a way that does not rely on negative samples. This approach avoids the issue of pushing different modalities into separate subspaces. Similarly, MMD measures the distance between the kernel-projected mean embeddings of the distributions, ensuring that the overall distributions are aligned without the need for negative sampling. By leveraging these methods, we can achieve a more cohesive embedding space where different modalities are aligned more closely, thus avoiding the modality gap problem inherent in contrastive learning approaches.\"\n\n>Why would Contrastive outperform other methods in this Clotho without audiovisual distillation?\n\nFirst, on Clotho, before the audio-visual distillation, the alignment through MMD, which is also part of our method, yields significantly better results (sometimes twice as good) than the contrastive baseline (0.0659, 0.0621, 0.0640 against 0.0460, 0.0293, 0.0377 for CIDEr, SPICE and SPIDEr, respectively). \nAs explained in the paper, the good performance of MMD is likely to be due to the learned image bias: line 343-344: “the image bias inherent in DALI_MMD becomes beneficial when confronted with out-of-distribution data.”\nIt is important to note that Clotho is known to be a more challenging dataset than AudioCaps. Moreover, it contains new audio concepts not seen in AudioSet, representing an out-of-domain scenario for our models (trained only on AudioSet), hence the observed drop in performance, compared to AudioCaps.\nCIDEr and SPICE are indeed important metrics for captioning, and these metrics are roughly equivalent between contrastive and attentive OT (0.0302 0.0355 for attentive OT against 0.0293 0.0377 for contrastive in CIDEr and SPICE respectively). As both scores are fairly low, trying to interpret such small differences would be unreliable. However, those models exhibit significantly different performances when focusing on other metrics.\nIn particular, the ROUGE_l, which measures the longest common subsequence, and takes the order of the words into account (interpreted as the “coherence” of the generated sentence), is significantly higher in our method (0.2101 for attentive OT against 0.177 for the contrastive). A tentative interpretation could be that, while both methods failed to generate accurate enough captions, the attentive OT still encodes the audio in a space that can be \"comprehended\" by the LLM as it stays in the same region as the image tokens, ending up in a more coherent caption, while the contrastive approach encodes something that cannot be interpreted by the LLM, leading to hallucinations or incoherent captions.\n\n[A] Weixin Liang and Yuhui Zhang and Yongchan Kwon and Serena Yeung and James Zou (2022) Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning\n[B] Abrar Fahim and Alex Murphy and Alona Fyshe (2024) It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gap", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9LhtkB9Uyx", "reviewer": "Reviewer_rTvm", "summary": "This paper presents a method to align tokens created by an audio encoder with those from an image encoder to allow zero-shot description of audio and audio-visual recordings. It introduces an attention-based matching to the alignment to be able to account for objects that appear in one modality but not the other (e.g., background music or the visual background). The paper compares two different alignment methods: maximum mean discrepancy and optimal transport, with the attention only applied in the case of optimal transport. And prefix tuning is used to guide the image captioner towards audio captions.\n\nThe model is trained on AudioSet and evaluated on AudioCaps and Clotho using standard captioning metrics. While several metrics are reported, the primary one used by the community is SPIDEr, and by this metric on AudioCaps (in-domain to training) the proposed approach when using audio-visual input (0.1946) outperforms a comparison supervised system (0.1830), an image-only system (0.1499), and other variants of the proposed system such as the audio-only version (0.1592). On the out of domain and audio-only Clotho dataset the supervised baseline performs best (0.097) followed by the MMD variant of the proposed system (0.0655). Note that state of the art performance on Clotho was around 0.3 for systems like QwenAudio and AudioFlamingo and around 0.5 on AudioCaps.", "questions": "What exactly was done in the ablation studies listed in Tables 1 and 2? What do the rows mean? What was taken away?\n\nWhy wasn't the attention weighting used with MMD in addition to OT?\n\nDoes the audio-visual alignment treat each instance as a \"bag of tokens\"? If so, would there be value in considering the sequence of tokens in this alignment as opposed to just their distribution?\n\nWere any categories held out of training so that they could be evaluated in zero-shot category generalization? Or was the focus only on zero-shot modality generalization?\n\nWas the filtering of audiovisual discrepancies described in appendix C applied to both training and test data or just to training data?", "limitations": "The paper has a section focusing on limitations and does an excellent job discussing them, providing true limitations of the approach.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Significance: \n* Cross-modal learning is an important problem, although it is not clear that zero-shot learning is necessary as there is a great deal of audio-visual content available, but if it is possible, then it seems worth exploiting.\n* The ability to leverage visual captioning systems to bootstrap audio captioning systems is valuable, as the former have received much more attention than the latter.\n* The issue of audio-visual correspondence is key in audio-visual processing and the proposed approach is an interesting way to deal with it.\n\nClarity:\n* The paper is generally well written and easy to follow\n* The diagrams are quite helpful in understanding the approach\n* The paper includes a real discussion of its limitations\n* The caption examples included in the appendices are quite informative. It would be nice if there were examples from all of the systems to compare\n\nTechnical correctness:\n* The experiments are well organized and well conducted", "weaknesses": "Clarity:\n* The use of many metrics dilutes the clarity of the results and their interpretation. The metrics should be prioritized and the systems they are comparing discussed accordingly. The discussion of results seems to treat all metrics as somewhat interchangeable and makes broad statements about which systems seem better than others. It would be much clearer and easier to follow if differences in performance on specific metrics were tied to qualitative differences in model behaviors and when there is one main metric (i.e., SPIDEr) to focus on that for overall system comparison.\n* Line 96 states that \"Audio data often lacks the richness and context provided through the visual modality, resulting in less precise and more subjective descriptions of the audio content.\" While I agree that audio captioning datasets typically contain more subjective descriptions of their content, I think this is an issue with the data that is selected to be captioned and the question asked of the captioner as opposed to a shortcoming of audio in general. With the right data and right task, audio captioning could be much better defined. I believe this sentence should be modified to focus on existing datasets as opposed to the general modality.\n* No mention is made of the size of the various models or model pieces involved or how much data they were trained on. This is especially acute for comparing the proposed system to existing baselines from Shaharabany et al and Salewski et al.\n* I can't understand what the ablations are showing, these should be described in the body of the paper, not just shown in tables.\n\nPerformance:\n* Overall, the reported SPIDEr results (0.1-0.2) are fairly low compared to SOTA approaches for Clotho (0.3) and AudioCaps (0.5). It's clear that this zero-shot approach does not completely solve the problem. It would be useful to discuss this in the paper.\n* Additionally, the differences in performance between the proposed variants are not that large, bringing into question the value of including multiple-such approaches.\n\nMinor issues:\n* Figure 2: sub-figures (d) and (e) are not mentioned in the caption\n* Equation (4): please use \\langle and \\rangle as opposed to < and > for brackets\n* Tables 1 and 2: Order the rows within each block intentionally. I would recommend ordering by SPIDEr.\n* Figure 4: It's hard to distinguish between this many colors with this level of transparency. Maybe you could use different symbols instead of different colors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. We’re happy to hear that you found our paper **well written and easy to follow**, appreciated the **helpful diagrams**, and recognized the **value in our approach to addressing audio-visual correspondence and leveraging visual captioning systems**. Below we address your concerns.\n\n> Using too many metrics muddies the results.\n\nWe understand that using many metrics complicates the reading. Here's a simplified analysis:\nCIDEr emphasizes the precision of n-grams (targeting key words), while SPICE focuses more on the semantic content, making it complementary to CIDEr. SPIDEr, an average of both, is therefore used as the primary metric.\n\nOn AudioCaps, all distribution alignment methods outperform the contrastive baseline (SPIDEr: 0.0871) after the first stage. While MMD (0.1385) performs better than standard OT (0.1183), attentive OT improves even further (0.1443).\nAfter distillation, all methods improve except for MMD (0.1360), with attentive OT remaining the best (0.1592) and the contrastive baseline improving significantly (0.1524). These results confirm the intuition that MMD learns a visual bias and therefore does not benefit from the distillation. \nNote that other metrics and especially in ROUGE_l, the contrastive approach remains less effective than other methods (0.2914 against 0.3025 and 0.3106 for MMD alignment and attentive optimal transport respectively).\n\nOn Clotho, MMD’s visual bias is beneficial, outperforming other methods after the first stage (0.0640 vs. 0.0377, 0.0299, and 0.0355 for contrastive, OT, and attentive OT, respectively). Post-distillation, MMD does not improve (0.0655), while other methods do, with attentive OT (0.0625) and MMD remaining slightly ahead of the contrastive approach (0.0620). ROUGE_l shows similar trends to AudioCaps.\n\nThis discussion aligns with the paper’s conclusions. We'll revise the manuscript accordingly.\n\n\n>Line 96 suggests audio data lacks richness. However, this is a data selection and task issue, not an audio modality shortcoming.\n\nWe'll revise Line 96 for clarity: \"Current audio captioning datasets often fall short in capturing the richness and context compared to what the vision datasets offer. This limitation arises from both the selection of data for captioning and the nature of the questions posed to captioners, leading to descriptions that are less precise and more subjective.\"\n\n\n>No mention is made of the size of the various models or \nhow much data they were trained on. \n\nWe agree mentioning this information is important. We will revise as follows:\n\n- Lines 104-107: “ZerAuCaps uses CLAP (an 80.8M-parameter model trained on 3.4M audio-text pairs) (...) to prompt an LLM (OPT 1.3B)”.\n- Lines 109-111: “The authors adopted ImageBind (using an audio encoder of 86M parameters trained on AudioSet) (...) They fine-tuned a GPT-2 model (117M parameters) (...)”\n\n> The reported are low compared to SOTA. What is the value of including multiple such approaches.\n\nWe want to clarify that our approach extends image captioners to audio without compromising their original image performance. While better audio performance might be achievable by sacrificing some image performance (typically by unfreezing the LLM), our focus is on learning from videos for scalability due to their widespread availability. To our knowledge, this is the first attempt at this problem. This is a first step towards large-scale training, and a model trained on 500k videos cannot be compared with others trained on millions of audio-text pairs. Future work could scale this method further to compete with supervised approaches.\nFor now, exploring alternative distribution alignment methods is valuable, but given the minor differences between methods, future work might focus on a single alignment method.\n\n>What was done in the ablation, what was taken away?\n\nTables 1 and 2 present ablations for different training stages. The rows show model performance after the first stage, which includes only distribution alignment and prefix tuning. What was taken away is the role of the second stage: audiovisual distillation (as written in the Table captions: “Ablation of audiovisual distillation”). We will clarify this further in the text.\n\n>Why wasn't the attention weighting used with MMD in addition to OT?\n\nIn attentive OT, cross-attention is applied in the token space to weight each sample of the empirical distributions. MMD, however, computes the expectation of the distribution in the kernel space, so weighting tokens before averaging would significantly alter the distance formulation, which is why we did not initially explore this.\nWe acknowledge this is an interesting experiment and hace since tested it. The results, shown in Table B reveal that while performing well across all metrics,it underperforms in CIDE_r. The model generates overly detailed captions, which affects CIDE_r (measuring n_grams) but not in SPICE. \n\n>Does the audio-visual alignment treat each instance as a \"bag of tokens\"? Would there be value in considering the sequence of tokens ?\n\nAudio-visual alignment does treat each instance as a \"bag of tokens\". However, our preliminary experiments revealed that token order does impact llava's performance in audio captioning with images (on AudioCaps).\nTo support this claim, we shuffled image tokens before feeding them to the LLM and found that the results were similar to those without shuffling (see Table A). This indicates that the sequential order of image tokens is not crucial for this task.\n\n\n>Were any categories held out of training to be evaluated in zero-shot category generalization?\n\nWe only focused on zero-shot modality generalization, but exploring other aspects is worth pursuing in future work.\n\n>Was the filtering applied to both training and test data ?\n\nThe filtering was applied only to the training set to keep test performance comparable with other methods. We will clarify this in Line 271.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2RWi849up1", "reviewer": "Reviewer_aH67", "summary": "This paper proposes to use a well-trained visual LLM (specifically Llava-v1.5) to perform audio captioning in a zero-shot fashion. The authors propose a training framework which comprises 2 stages and 5 sub-steps. The authors also propose to use Maximum Mean Discrepancy (MMD) and optimal transport (OT) as the loss function to align audio and visual representation space.", "questions": "See Weaknesses.", "limitations": "There is a limitations section but no discussion on potential negative social impact.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The two loss functions are novel directions to explore for alignments between different modalities\n2. This paper explores audio-visual LLM application scenarios which is timely.", "weaknesses": "1. The motivation is not clear. The author must specify in which scenario zero-shot audio captioning is needed, with concrete examples that the proposed method will help in practice, such as generating a particular captioning style or targeting a specific type of audio. As far as I can see in the paper, the authors only did experiments on two standard tasks (AudioCaps and Clotho) and both tasks have a reasonably-labelled size, and the performance is far from the state-of-the-art.\n\n2. The method lacks theoretical grounding or insights. The author says they randomly choose one frame from the 10 frames in a video as the image and train to pull the image representation space to be close to the audio space. The author did not analyse how related the image is to the audio. How often does the image reflects what can be heard, and how often it does not? \n\n3. The experimental setup is questionable. \n  - The author says clearly in line 167 that they performed p-tuning with audio captioning data \"We trained the additional tokens using image-text pairs, with the text describing the sounds associated with the images\". This clearly sets this method apart from \"zero-shot learning\" which refers to having no data at all. If the author means \"zero-shot\" for a specific audio caption dataset, they need to be clear about it. \n  - Regarding experimental results: I wonder if the comparison between the proposed method and ImageBind is fair. They also did not say how much data they used for ImageBind p-tuning. If the authors just took the numbers from Shaharabany et al., then putting the numbers in the same table is clearly unfair. I might have missed something here but without a convincing explanation, I do not think the experiments are valid. Why is ImageBind not included in Table 2?\n  - I would be interested to see whether adding audio to the audio-visual description would give better performance or not. It seems that the audio information is all included in the visual information. Getting back to Weakness 1 - what is the point of doing this task then? What extra did you gain?\n\nA few writing problems:\n1. The referencing style makes it difficult to locate various papers. \n2. Fig. 2 captions are incomplete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We’re glad to hear that you found the exploration of our **loss functions as novel directions for alignments between different modalities** and the **timely application scenarios of audio-visual LLM** valuable. Below, please find a point-by-point response to your feedback. \n\n>The motivation is unclear. The author should specify practical scenarios where zero-shot audio captioning is needed.\n  \nWe posit that Zero-shot audio captioning is particularly valuable in scenarios where annotated audio data is scarce or unavailable. Current audio captioners like Audio Flamingo and Pengi, trained on nearly all publicly available audio-text pairs, seem to have reached the limits of the supervised approach. In contrast, the abundance of videos makes unsupervised audiovisual training a viable way to scale further.\nFor example, domain-specific topics (like wildlife sounds) that lack supervised datasets but non-annotated videos are available, which is realistic given that there exist only two \"real\" academic audio captioning datasets.\n\n\n>The author randomly selects one frame from a video without analyzing the relevance of the image to the audio.\n\nWe indeed chose one random frame from the video to match its distribution with the audio and analyzed the image-audio relatedness. The paragraph Datasets of Section 4 as well as the full section C of the Appendix details this process: (line 579-581) “As AudioSet (...) features audio events that may be unrelated to the visual event depicted (...) we filtered out audiovisual discrepancies”. Pairs were chosen “by computing the similarity between the noisy audio labels and an image caption generated by BLIP-2” (line 269-270). This filtering considers the entire video,averaging similarity scores across frames sampled at one-second intervals.\nTo address how often the image reflects the audio, we retained the 500k cleanest pairs, which best retain related pairs (line 269: “we train our models on a subset of 500k videos from AudioSet”). Additionally, since the image and audio can contain partial, non-common information, we used attentive optimal transport to learn only the common parts of the distributions (Section 3.2, paragraph 3).\n\n\n>The author states that they used image-text pairs for p-tuning, which differs from \"zero-shot learning\"\n\nWe acknowledge your concern about potentially breaking the “zero shot” set-up with our handling of data. To clarify this concern, as we explained in line 53 of the submission, and consistent with Salewski et al., we define zero-shot audio captioning as **the model's ability to generate audio captions without training with manually assembled audio-caption pairs**, as they also used textual-only descriptions (AudioSet tags from a \"keyword bank\") as opposed to actual audio files and their corresponding captions. \n\nWhile we did use captions for prefix tuning, we emphasize that we **never** used audio-caption pairs for the targeted audio captioning task, which would indeed compromise the experimental setup. Instead, we used a few image-caption pairs without accessing the corresponding audio files. This use was solely to retask the LLM to create audio-centric captions. \nWe maintain that our approach effectively addresses the zero-shot audio captioning task, allowing direct comparison with Salewski et al. and other methods.\n\n\n>Is the comparison with ImageBind fair? Data used for ImageBind p-tuning isn't specified. Why is ImageBind absent from Table 2?\n\nFirstly, we would like to clarify that our work does not compare directly to ImageBind, as it lacks the capability to perform audio captioning. Instead, it is a model that maps multiple modalities to a shared space. Hence, comparing our method directly to ImageBind is not feasible.\n\nWe instead compare with Shaharabany et al., which, as described in Lines 108-114. This work is the closest to ours in terms of backbone training setup, as it has been trained using audio-image and image-text pairs but **not** audio-text pairs.\n\nSecond, Shaharabany et al. did not perform p-tuning, relying instead on an “audibility score” (part of the training loss, core of their work). In our case, we used p-tuning for the same purpose (using only image-text pairs). Since neither work used audio-text pairs, comparison is feasible and fair.\n\nWe acknowledge that, as ImageBind also contains an image encoder, prefix tuning can be applied. Unfortunately, as Shaharabany et al.’s code is not available, we couldn’t test his interesting idea. \n\nFinally, the work of Shaharabany et al. is not included in Table 2 as they did not report their results on Clotho. We will clarify this by extending Table 2’s caption to ”Shaharabany et al. did not report results on Clotho.”\n\n>Does adding audio to the audio-visual description improves performance ? What was the purpose of this task and what additional value did it provide?\n\nWe would like to recall, that our method is designed to handle multiple modalities, such as image, audio AND audio-visual for the zero-shot audio captioning task. This is achieved by substituting or concatenating modality-specific tokens.\nAs demonstrated in Table 1, our method performs better when both image and audio are used together (6th row, labeled DALI_OT^att+Image) compared to using the image alone (1st row). This highlights the usefulness of including audio, offering details about sound sources occluded or out of view in the image, e.g., a ringing phone inside a bag.\n\nOur results show that using both modalities leads to more informative audio captions. We also show that using audiovisual captions as pseudo-captions to supervise an audio-only model in a second stage (audiovisual distillation), yields a more robust audio captioner. One core motivation was to leverage a powerful image captioner to enhance audio captioning, creating a virtuous cycle where both modalities mutually benefit. Thus, including audio proves significant value beyond what visual data alone can offer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to start by thanking all the reviewers for the time they spent reading carefully our work and their valuable feedback that helped us improve the quality of the submission.

We would like to underline an important contribution of our work: we add audio capability to an LVLM while **keeping its vision-only performances intact**. This is important to keep in mind as this allows for the use of the same model for audio-only, vision-only or audio-visual tasks. This is highlighted in the 4th bullet point of our contributions, line 80-81: “Our method supports both audio and audiovisual inputs, extending the image captioner’s capabilities **without compromising its performance in image analysis tasks**.”

Below, we provide a detailed response to each reviewer's comments.

Meta-review: This paper presents a method to align tokens created by an audio encoder with those from a pre-trained image encoder (Llava-1.5) to allow zero-shot description (captioning) of audio and audio-visual data. It introduces an attention-based distribution matching to the alignment to be able to account for content that appear in one modality but not the other (e.g., background music or the visual background), the so-called modality gap. The paper compares two different alignment methods: maximum mean discrepancy (MMD) and optimal transport (OT), with the attention only applied in the case of optimal transport, over a baseline approach of contrastive learning (widely used). Prefix tuning (p-tuning) is used to guide the image captioner towards audio captions (i.e. mention audio-relevant content), which is also a novel application of this method.

The model is trained on AudioSet and evaluated on AudioCaps and Clotho using standard captioning metrics. While several metrics are reported, the primary one used by the community is SPIDEr, and by this metric on AudioCaps (in-domain to training) the proposed approach when using audio-visual input (0.1946) outperforms a comparison supervised system (0.1830), an image-only system (0.1499), and other variants of the proposed system such as the audio-only version (0.1592). On the out of domain and audio-only Clotho dataset the supervised baseline performs best (0.097) followed by the MMD variant of the proposed system (0.0655). Note that state of the art performance on Clotho was around 0.3 for systems like QwenAudio and AudioFlamingo and around 0.5 on AudioCaps. The proposed model also maintains performance on the image task. 

Reviewers agree that the paper is well written and that the goal of the paper, its contributions, the execution are all clearly described. There are a good set of experiments (even "too many metrics"), visualizations of the expected improvement in alignment of the visual and audio token distributions, and overall the application of MMD or OT learning improves the alignment as demonstrated by the experimental results over the dominant baseline approach of contrastive learning. Reviewers agree that the paper puts together existing components in a novel and interesting way and that the cross-modal learning problem is relevant and deserves more work, since there is a significant gap in investment between image and audio modalities. 3 reviewers recommend a weak accept of this paper after the rebuttal phase (which resolved some initial questions) and the paper was discussed thoroughly, which would allow authors to prepare an improved camera-ready version.

On the flip side, the work may not be very impactful, since:
- the target audience may be small: it's not clear if the approach could be applied to other domains) since the task definition is very narrow and leaves out comparisons with other recent and coming LLMs. though the task is practically relevant (as confirmed by one of the reviewers), given current trends, other LLMs may quickly raise the bar, reducing the potential impact of this paper in practice.
- the performance is good on the "in-domain" task, but drops significantly on out-of-domain tasks (other databases). while the authors' definition of "zero-shot" use is consistent with other previous work (and their evaluations are fair), readers would reasonably expect better performance on Clotho for a "zero-shot" approach than what the method presently delivers
- experimental results are somewhat mixed and require a lot of attention to detail in order to fully understand. authors were able to increase reviewers' appreciation for the work and its theoretical justification during rebuttal, but it seems that readers (even of an improved camera-ready version) may still have other questions which they would not be able to resolve without access to the authors explanations
Decision: Accept (poster)