An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching

Authors: hugo  malard
michel  olvera
stéphane  lathuiliere
slim  essid
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:05:59

Abstract: Multimodal large language models have fueled progress in image captioning. These models, fine-tuned on vast image datasets, exhibit a deep understanding of semantic concepts.
In this work, we show that this ability can be re-purposed for audio captioning, where the joint image-language decoder can be leveraged to describe auditory content associated with image sequences within videos featuring audiovisual content. This can be achieved via multimodal alignment.
Yet, this multimodal alignment task is non-trivial due to the inherent disparity between audible and visible elements in real-world videos. Moreover, multimodal representation learning often relies on contrastive learning, facing the challenge of the so-called modality gap which hinders smooth integration between modalities. In this work, we introduce a novel methodology for bridging the audiovisual modality gap by matching the distributions of tokens produced by an audio backbone and those of an image captioner. Our approach aligns the audio token distribution with that of the image tokens, enabling the model to perform zero-shot audio captioning in an unsupervised fashion. This alignment allows for the use of either audio or audiovisual input by combining or substituting the image encoder with the aligned audio encoder. Our method achieves significantly improved performances in zero-shot audio captioning, compared to existing approaches.

Main paper: 
 introduction: 
 Recent progress in image captioning has been driven by methods integrating Large Language Models (LLMs) with vision encoders (1; 2; 3). The impressive capabilities of Vision Language Models (VLMs) stem from supervised training on large image-text collections and extensive parameterization. Even the smallest VLMs exceed more than a billion parameters (4; 5). These models excel in generating human-like text descriptions for images and understanding complex semantic relationships. Such capabilities can be potentially extended to other tasks, beyond image analysis.One such task is audio captioning, in which large-scale audio-text collections, with precise descriptions of the audio content, are lacking. Recent works have proposed diverse solutions to address this limitation. Solutions include augmenting class labels with phrases such as "This is a sound of " (6; 7) or prompting LLMs to generate natural language descriptions directly from class labels (8) . Further, datasets from related domains like Speech (9) or Music (10; 11) , have been considered to enlarge training data. While such methods show impressive results, scaling them further is challenging due to the rarity and low quality of such labels-captions. In particular, these approaches cannot cope with the annotation noise encountered in existing audio datasets, especially in the largely adopted resource: audio tracks of AudioSet (12) , which is only partially and weakly labeled. We hypothesize that VLM-based image captioning models, inherently possess the knowledge to perform audio captioning. Thus, joint image-language representations of VLMs can be used to describe auditory content associated with image sequences from videos featuring audiovisual content, eliminating the need for handcrafted audio-caption pairs. To our knowledge, this is the first attempt to leverage VLMs for this purpose. Extending VLMs to audio captioning is non-trivial due to the inherent disparity between audible and visible elements in real-world videos. For instance, soundproducing objects in images may be occluded or out of view, while visible objects might not generate sound. This mismatch hinders connecting visual and auditory modalities seamlessly.Multimodal representation learning (13) , crucial for tasks like image captioning, is often accomplished through contrastive learning (14; 6) . This paradigm aligns features across modalities by maximizing agreement between similar samples and minimizing agreement between dissimilar ones. Despite its effectiveness, contrastive learning faces the so-called "modality gap", where embedded data are represented in distinct, non-overlapping regions of the embedding space (15; 16) (the distance between red and green manifolds in Figure 1 ), limiting flexibility when conditioning modalities.We propose a novel methodology to bridge the audiovisual modality gap by matching the distributions of tokens produced by an audio backbone and those from the encoder of an image captioner (represented as the green and blue manifolds in Figure 1 ), enabling the image captioner to perform zero-shot audio captioning in an unsupervised way. We define zero-shot audio captioning as the task of generating audio captions without training on manually assembled audio-caption pairs. We propose and compare two token distribution alignment methods: one based on Maximum Mean Discrepancy (MMD) (17) and a more flexible alternative using Optimal Transport (OT) (18) , possibly enhanced with a cross-attention mechanism. This mechanism refines distribution matching by attending to each token within a modality and determining its match with a token from the other modality based on semantic similarity. We find this approach to result in more accurate and coherent joint audiovisual representations.Acknowledging the challenges of zero-shot audio captioning, we enhance caption quality using "prefix tuning", a technique commonly employed across domains to adapt LLMs to diverse tasks in a few-shot manner (19; 20; 21) . This tuning process conditions the model with few image-audio caption pairs, prompting it to generate audio-centric captions and thereby boosting performance across standard audio captioning metrics (22; 23; 24; 25; 26; 27) . Notably, our methodology allows for the use of either audio or audiovisual input by combining or substituting the image encoder with the aligned audio encoder. As a result, our approach extends the ability of the VLM to audio captioning without compromising its image performance. Moreover, the method achieves significantly improved performances in zero-shot audio captioning, compared to existing approaches. In summary, our contributions are as follows:• We introduce a novel methodology for unsupervised zero-shot audio captioning through Distribution ALIgnment (DALI), leveraging advanced image captioning models, and suc-cessfully instantiate it with a particular image captioner (namely Llava 1.5), as we obtain state-of-the-art zero-shot audio captioning results without any supervision from annotated audio data (see Section 5). • To bridge the "modality gap", we propose and study two multimodal token distribution alignment methods exploiting MMD or Optimal Transport (OT). The latter-never used before in this context-is enhanced with a cross-attention mechanism for greater robustness. • We introduce the use of prefix tuning to guide the image captioner towards audio captioning, adapting the model with a few shots of image and audio-caption pairs (only using the textual descriptions of sound and not the audio signals). • Finally, our method supports both audio and audiovisual inputs, extending the image captioner's capabilities without compromising its performance in image analysis tasks. 
related works: 
 Vision Language models Large Vision Language Models (VLMs) typically use a pre-trained LLM that processes both visual and textual information. The image encoder usually consists of a ViT (28) trained via multimodal contrastive learning using millions of image-text pairs such as the ones used in CLIP (14) . Integrating visual information into the LLM is usually done by concatenating image tokens extracted by a vision encoder and then further transformed by an MLP, with textual tokens from a prompt. This straightforward architecture can be easily adapted to other modalities by simply replacing the input tokens to accommodate the new input modality.Audio Language models Recently, a similar paradigm emerged in the audio application domain.Various works rely on both LLMs and audio encoders trained jointly on millions of audio-text pairs, similarly to CLAP (6) . Listen Think and Understand (LTU) (29) , Pengi (11) , QwenAudio (9) and AudioFlamingo (10) rely on a language model fed with tokens from both audio and text modalities. These large models require substantial volumes of training data. For instance, AudioFlamingo has been trained on nearly all publicly available labeled audio data. One limitation is that the quality of the audio-text data is generally not comparable to that of image-text data. This situation is made worse by the fact that audio data labelling is inherently difficult due to the potential overlap of multiple sound sources at different acoustic levels, which sometimes makes it hard to perceive some of the audio classes in presence. Despite this limitation, AudioFlamingo and similar models exhibit remarkable performance. To further enhance the capabilities of audio-language models it is of paramount importance to explore alternative paradigms beyond strong reliance on (often unreliable) annotations and supervised learning. This is the goal pursued in this work.Training large audio captioner models, as mentioned previously, requires large volumes of audio and textual descriptions and substantial computational resources. To address this limitation, some approaches leverage existing robust backbones to enable zero-shot audio captioning. ZerAuCaps (30) uses CLAP (an 80.8M-parameter model trained on 3.4M audio-text pairs) to infer potential classes from a predefined word bank (45 words) before using them to prompt an LLM (OPT (31) 1.3B), generating multiple plausible captions. The final caption is the closest to the audio in the CLAP space. However, this method is constrained by the finite set of classes in the word bank, limiting its application to predefined scenarios. The most closely related work to ours is that of Shaharabany et al. (32) who also performed unsupervised zero-shot audio captioning. The authors adopted the audio encoder of ImageBind (33) (86M parameters trained on AudioSet), a multimodal model trained to align various modalities in the image space via a contrastive loss. They fine-tuned a GPT-2 (34) model (117M parameters) fed with ImageBind audio tokens using two distinct loss functions for the captioning task: an audibility score ensures the generation of audio-centric captions without visual artifacts, while an ImageBind score measures the similarity between the generated caption and the audio. Despite its innovative approach, its performance falls short compared to ZerAuCaps primarily due to ImageBind's less effective handling of audio.Audiovisual Alignment Multimodal alignment aims to encode information from different modalities such as text, image, or audio into a shared semantic space, enabling cross-modal relationships and multimodal retrieval (e.g., retrieve the image closest to a caption) (13) . Despite the abundance of web video datasets, aligning the audio and image/video modalities remains a challenge. The conventional contrastive loss (35) often fails to effectively co-train the encoders. Unlike the relatively clean associations in image-text pairs, audio-image pairs present a greater degree of variability, with Figure 2 : Overview of the proposed approach. In the first stage, a prefix tuning is performed using a few (image, audio-caption) pairs (1-a). Additionally, the audio backbone is aligned with the image backbone (1-b) through distribution alignment. Audio captioning can then be performed by switching the image backbone with the audio backbone and adding the prefix tokens (1-c). In a second stage, visually-informed audio captions are generated using both audio, image, and prefix tokens. The MLP mapping the audio encoder to the language model is then fine-tuned with these pseudo captions (2-d). The final inference for audio captioning, using audio or audio visual inputs, is performed by forwarding the aligned audio backbone's output through the trained MLP to obtain the LLM input (2-e).the content described in the audio often differing from that depicted in the corresponding image or video. Consequently, stable pre-training needs augmenting the contrastive loss with a reconstruction loss (36; 37; 38) , reducing the similarity of the representations between the two modalities. 
method: 
 Current Large Vision Language Models (VLMs) exhibit a sophisticated understanding of concepts related to real-world objects in the visual domain. This capability arises from optimizing the language model, already containing a lot of intrinsic knowledge, so as to extract additional information from image features and learn how to describe the (visual) world. We hypothesize that the knowledge gained through this process is sufficiently general to be transferable across different modalities.To test this hypothesis, we developed a methodology that employs an VLM's image captioner to generate audio captions from images, audios and audiovisual inputs, while preserving the original performance of the image captioner. Our methodology stands out for its applicability to any modality requiring integration with image data. Here, we focus on audiovisual alignment for audio captioning, benefiting from the abundance of audiovisual content in videos from web data.An image captioning system outputs descriptions of objects in an image, such as "A dog in the grass," providing visual cues. To adapt this system for audio captioning, we need it to focus on audible information instead of visuals. For example, given an image of a dog, the audio caption should instead be "A dog is barking.". Our methodology enables audio captioning from an image captioner by keeping both the image encoder and LLM frozen, ensuring that the image captioner original performance is preserved. In our work, we choose Llava (1) (Apache License 2.0) as an instance of an image captioner, for its proven effectiveness. We use the CAV-MAE (36) (BSD 2-Clause License) audio model as our audio encoder, since it is already roughly aligned with an image backbone, which is expected to help reach faster convergence.As seen in Figure 2 , we follow a 2-stage process. First, we use prefix tuning to start re-tasking the system for audio captioning, learning tokens from a few image-caption pairs, ensuring that the textual descriptions target the audio content (stage 1, step (a)). We also align the token distribution of an audio encoder with that of the image encoder (stage 1, step (b)), then replace the image encoder with the aligned audio encoder for direct audio captioning from audio files (step (c)). We can also combine both encoders for visually informed audio captioning, using both audio and visual inputs. This setup generates audio captions to fine-tune the MLP that maps audio tokens to the language model embedding space, a process we call audiovisual distillation (stage 2, step (d)). The fine-tuned MLP then replaces the original MLP, generating robust audio captions from audio or audiovisual inputs (step (e)). In the following, we zoom on specific aspects of the proposed methodology.Prefix-tuning with image-caption pairs Image captioners, while effective for image analysis tasks, struggle to produce accurate audio-centric descriptions when prompted to describe sounds from images. For example, when asked, "What sounds can the objects in this image generate?", the descriptions provided heavily reflect the visual characteristics of the objects. Instead of focusing on sound-related details, their outputs are dominated by visual descriptions, which are considered artifacts in the context of audio captioning. To address this problem, we adopted a prompt tuning strategy (39) , aiming to guide the model to generate captions that exclusively describe audible events as depicted in Figure 2 -a. Prompt tuning involves learning a portion of the prompt tokens: the prefix, that guides the model to perform a specified task (39) . This method allows the rest of the model to remain frozen, and by simply removing the learned tokens, the model is able to perform its original task with the same performance. This approach requires only a small number of parameters to be trained and can be effectively trained using a few examples, a process akin to few-shot adaptation.We employed this technique to re-task the model to perform audio captioning. We trained the additional tokens using image-text pairs, with the text describing the sounds associated with the images i.e., audio captions. Specifically, we used images from AudioSet videos and corresponding captions from AudioCaps (40) (MIT). We used negative log-likelihood as the loss function for the prefix tuning:EQUATIONwith n and j being respectively the image and caption token indices, N the number of elements in the batch, l the number of predicted tokens, the index I referring to the vision tokens (from 1 to n i ), C j the j-th caption token and θ the parameters of the model. Using the prefix tokens, the image captioner can now output more plausible sound descriptions.Audio captioning via token distribution alignment Aligning the distribution of the audio encoder with that of the image one allows for substituting one encoder with another and therefore conditioning the generation either on the audio or image modality. We align the distributions (Figure 2 : step (b)) using either Maximum Mean Discrepancy or Optimal Transport. More details are provided in Section 3.2. Using the aligned audio backbone, as represented in Figure 2 : step (c), we feed the language model with the audio tokens along with the prefix, which enables the model to perform audio captioning. Additionally, we incorporate both the image and audio tokens, to give the ability to the model to perform audio captioning fusing visual and audible cues. This setup allows us to infer and generate, visually informed, audio captions for AudioSet balanced (12) , a standard subset of AudioSet, containing ∼ 20k videos, curated such that they involve the same number of samples for each class.The captions generated from audiovisual inputs serve as pseudo-labels to supervise the fine-tuning process of an audio captioner with an audio-only setup. This process refines only the MLP that maps the audio tokens to the LLM embedding space. Training employs a negative log-likelihood loss, as per equation ( 1), albeit without conditioning on image tokens. Instead of relying on a limited number of real captions, pseudo-labels are employed. This procedure is illustrated in Figure 2 : step (d), and optimized through the following loss:EQUATIONwith A referring to the audio tokens (from 1 to n a ).Audio captioning via audio or audiovisual inputs With all components trained, our model can perform audio captioning using the aligned audio encoder and fine-tuned audio MLP, or visualyinformed audio captioning by feeding to the LLM the image tokens in addition to the audio tokens (shown in Figure 2 : step (e)). Notably, the original image captioning task can still be performed at its original performance level by using only the image tokens, without the audio.Many studies on multimodal alignment typically use a contrastive loss on the averaged output tokens (14; 41) . This method merely aligns token mean-statistics, potentially leading to information loss, when the full token distribution is needed. Llava (1) for instance, makes use of the full token distribution as input, naturally creating the need for a full token distribution alignment, to allow for swapping the pretrained encoder with a new one targeting a new modality. Moreover, contrastive learning faces the so-called "modality gap" problem where each modality is encoded in distinct sub-regions of the embedding space (see Figure 4 ). For all these reasons, replacing image tokens with audio tokens obtained by alignment through a contrastive learning approach may yield undesirable responses from the language model. In light of these issues, we propose to learn the full image token distributions, rather than relying solely on contrastive learning, so as to facilitate knowledge transfer. We refer to our distribution alignment framework as DALI which stands for Distribution ALIgnment.We study two variants of distribution matching.Modality alignment through MMD Maximum Mean Discrepancy (MMD) serves as a robust measure of dissimilarity-or more specifically, discrepancy-between two probability distributions. MMD has been shown to be effective in distribution alignment, notably, but not exclusively, for domain adaptation (42; 43) . It computes the distance between the expected values of two distributions, denoted by P and Q, within a feature space characterized by a mapping Φ : X → H where H is a reproducing kernel Hilbert space (RKHS):EQUATIONIn practice the MMD is computed using the kernel trick (44) . In our work, we consider the Gaussian Radial Basis Function (RBF) kernel and use MMD as a loss function to directly align the audio and image token distributions. We term this distribution alignment through MMD: DALI MMD .Modality alignment through Optimal Transport Optimal Transport (OT) stands out as a powerful method for aligning probability distributions (45) . Despite its successful application across various domains such as unsupervised domain adaptation (46) , image generation (47) , and style transfer (48) , its application to multimodal representation alignment remains relatively unexplored. In our context, where the alignment of audiovisual modalities is paramount, OT emerges as a natural solution for its intrinsic capacity to integrate the geometry of the underlying space, enabling optimal distribution matching. Discrete optimal transport considers two distributions x ∈ R N and y ∈ R M , where N and M represent the number of samples in each distribution. These distributions are represented by their empirical measures: α = N i=1 α i δ xi and β = M j=1 β j δ yj , and OT seeks a coupling γ ∈ Π(α, β) between them that minimizes a transportation cost. The problem can be formalized as:EQUATIONwhere D ∈ R N ×M is the cost matrix, ⟨•, •⟩ F denotes the Frobenius dot product, and γ ∈ R N ×M represents the transportation coupling matrix. The minimum of the optimization problem can be interpreted as a distance (45) . When considering the square of the l 2 norm, the distance is known as the Earth Mover Distance (EMD). In this work, we align the audio and image token distributions using EMD and refer to this approach as Distribution ALIgnment through Optimal Transport (DALI OT ).Improving cross-modal correspondence with attentive distribution alignment Optimal transport is typically applied assuming uniform distributions of the weights {α i } N i=1 and {β j } M j=1 . However, in audiovisual content, objects producing sound are often occluded, with the visual information serving to complement the auditory content. The classic formulation of OT enforces strict mass preservation, needing that all mass from the source distribution is transported to the target distribution. This constraint becomes problematic in audiovisual alignment, where not all tokens from one modality may find a match in the other. While alternative methods exist to circumvent these constraints, such as Unbalanced Optimal Transport (UOT) (49) , which replaces this rigid requirement with a soft penalization term, the tuning of such hyper-parameter poses significant challenges (50) . Alternatively, we propose learning the weights α i and β j via a cross-attention mechanism which offers as well a robust approach. This mechanism, attending to both modalities, enables learning which objects and/or sound-related tokens are present in both modalities. Tokens encoding modality-specific information, e.g., an occluded source, or an object apparently not generating a sound, receive lower weights. Thus, this approach promotes the alignment of modality-shared content. We provide the details of this cross-attention mechanism in Appendix F. It is important to note that to avoid the trivial solution that would put all the weights to 0 except the one of the closest tokens, we regularize the entropy of the weights.The final loss consists of a weighted sum of the optimal transport distance and the entropy of distribution weights:EQUATIONwhere H(•) denotes the Shannon entropy and OT(x, y, α Att , β Att ) represents the optimal transport between α Att and β Att , the weights at the output of the attention mechanism, with the cost matrix computed from x and y. An intuitive depiction of the method is illustrated in Figure 3 . Note that the cross-attention layers are only used for training and discarded afterward. We refer to this alignment method as DALI Att OT .Figure 3 : Multimodal distribution alignment through optimal transport. The audio and image tokens are used to compute the cost matrix, while two separate cross-attention layers estimate the weights α Att and β Att .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: NA Justification: Our research does not involve human subjects or participants. We systematically referred to the licences of the licenced datasets that we used (namely CC BY 4.0). We are not aware of any potential misuse of our algorithm, but there may be some. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [No] Justification: We are not competent for such a discussion. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 Our experiments focus on AudioSet, a widely explored large-scale collection of videos from YouTube. We evaluate our proposed approach on two popular and publicly available human-annotated audio captioning datasets: AudioCaps and Clotho (Tampere University licence), using conventional audio captioning metrics. We also provide details of our training and fine-tuning procedures.Datasets AudioSet ( 12) is a dataset (license CC BY 4.0) composed of 2 million YouTube videos with annotations indicating the presence of environmental sound events at the clip level (without precise time location), spanning 527 classes. The dataset includes reliability scores for annotations, some below 50%. Despite a diverse range of classes, a significant portion predominantly features music or speech content. Many videos show notable disparities between their visual and auditory content, thus attempting to align audio and image representations without any pre-filtering of the content is challenging. Similarly to Nagrani et al. (51) , who filtered the dataset by balancing the classes, we train our models on a subset of 500k videos from AudioSet. We chose them by computing the similarity between the noisy audio labels and an image caption generated by BLIP-2(52) (chosen for the conciseness of the captions), to remove the biggest discrepancies between images and audios. More details on this filtering process are given in Appendix C. For each video, we extracted 10 frames (the first of each second), and we randomly chose, during training, one of these 10 frames for each video.  
results and discussion: 
 Audio Captioning performance Table 5 presents the audio captioning performance of our methods on the AudioCaps test set, using 16 image captions to train the prefix tokens. We show the relatively small importance of the number of pairs in the audio captioning learning process in Appendix A. Our study compares the alignment variants considered in our zero-shot audio captioning framework against the current state-of-the-art methods, which employ audio encoders trained in either a supervised or an unsupervised manner. Our results indicate that the alignment through contrastive learning without audiovisual distillation does not fully match distributions, compared to DALI. Indeed, even without audiovisual distillation, DALI MMD and DALI Att OT significantly outperform Shaharabany et al. (32) , this system being our closest competitor, as it relies on a backbone trained in an unsupervised fashion.OT , however, the latter improves with audiovisual distillation. We hypothesize that this occurs because DALI MMD is trained to learn the complete image token distribution, while DALI Att OT only learns a part of it (due to the attentive optimal transport mechanism). Therefore DALI Att OT features can take more advantage of the combination with image tokens. We verify qualitatively this hypothesis in the Discussion paragraph below. Interestingly, both DALI OT and the contrastive backbone benefit from the audiovisual distillation. Given the relatively low-performance scores prior to the distillation process, it is plausible that they primarily learn from the image captioner (as the audiovisual pseudo captions tend to rely solely on visual features due to the inadequate audio representations) which is shown to be already performing well for audio captioning (first line of results). We hypothesize that the second stage is effective even for poorly aligned backbones because the distillation occurs on a subset of AudioSet where the images align well with the audio. If this distillation were applied to the entire dataset, the performance of these backbones might decline substantially. These hypotheses will require further investigation in future work. Lastly, it is important to note that feeding the image tokens in addition to the DALI Att OT ones, increases even more the performances, leaving the door open for further improvements. While contrastive learning maps the audio in a space separate from the image ones, MMD and optimal transport project in the same part of the space. The model trained using attentive optimal transport projects the audios in a space closer to the image, with marginal overlap.Discussion Unlike standard contrastive methods, we do not average the output of the backbone, which allows us to employ token-level distribution alignment methods that do not rely on negative samples. Both Optimal Transport (OT) and Maximum Mean Discrepancy (MMD) focus on directly aligning the embedding's distributions.OT minimizes the cost of transporting one distribution to another, aligning them without the need for negative samples, and avoiding the issue of pushing different modalities into separate subspaces. Similarly, MMD measures the distance between kernel-projected mean embeddings, aligning the distributions without negative sampling. These methods promote a more cohesive embedding space, closely aligning different modalities and thus avoiding the modality gap inherent in contrastive learning approaches. Indeed, Figure 4 shows a Principal Component Analysis (PCA) of the global representation (the average of the output tokens) of audio and images from various backbones on a subset of the AudioCaps test set. In our audiovisual experiment, we observe a phenomenon akin to the one demonstrated by Liang et al. (15) for image-text contrastive learning. Specifically, models encode each modality in a narrow cone, resulting in the so-called modality gap where the embeddings of each encoder are confined to a subregion of the embedding space. However, our backbones, which are trained using distribution alignment, do not exhibit this issue. The embeddings generated by both the DALI MMD and DALI OT encoders closely resemble those of the image modality. In contrast, the DALI Att OT ones, while located in the same region of the embedding space, remain distinct from the image tokens and exhibit minimal overlap. This indicates DALI Att OT and image tokens are complementary and provides an explanation for why the former benefits more from audiovisual distillation than the DALI MMD model.The root cause of this discrepancy lies in the training process: the cross-attention-based DALI Att OT model is trained using only to the matching audiovisual tokens, whereas DALI MMD model is trained on the complete image distribution, which inherently includes a degree of image bias.To verify this claim, we examined whether the aligned backbone with DALI MMD (after Stage 1) exhibits image bias. We prompted the captioner with "What can you see? Answer concisely" along with the audio tokens without the tuned prefix to avoid constraining it to generate only audio-related captions. Table 2 presents multiple captions generated using the DALI Att OT and DALI MMD backbones. As shown, the encoder trained with MMD often produces captions with notable visual biases, which are absent in the outputs of DALI Att OT . These results confirm our hypothesis and align with the visualizations in Figure 4 . Note that the captions may seem incomplete as we ask the model to answer concisely, focusing only on the main element of the scene . Complete generated captions by different models are provided in Appendix D.A woman is speaking from a microphone A woman in a black shirt and a red scarf is speaking into a microphone. A woman A male voice and a machine buzzing A man is using a drill to make a hole in a piece of wood. A person's hand holding a tool. A bell is ringing A person and a dog standing in front of a large machine.A man with a beard and mustache.A man cutting a tree trunk with a chainsaw. A lawn mower A man speaking as rain lightly falls followed by thunder A man wearing a black shirt and a grey suit.A car racing on a track. A car. OT is performing, however, DALI MMD gives slightly better results. The bias learned by matching the complete image distribution seems to be beneficial for out-of-domain samples. ( * ): Trained in a supervised fashion using audio-caption pairs. are sourced from FreeSound, a dataset distinct from AudioSet. Consequently, our model, having been trained solely on AudioSet's audio excerpts and images, may encounter unfamiliar concepts in Clotho, thereby testing its capacity for generalization. Table 6 presents the models' performance on Clotho's test set. The observed trends are consistent with those on AudioCaps where DALI MMD does not benefit from audiovisual distillation, while DALI Att OT does. However, the absolute results differ, as DALI MMD marginally outperforms DALI Att OT even after the second training stage. We posit that the image bias inherent in DALI MMD becomes beneficial when confronted with out-of-distribution data. Indeed what may be regarded as out-of-distribution for the audio backbone might have been seen by the image one (due to its bigger training set). Consequently, the learning of the image backbone's bias could potentially be beneficial. Notably, our method outperforms Saleswki et al., despite the fact that the latter employs a backbone trained using millions of audio-text pairs (CLAP). It is important to note that our method differs from Saleswki et al. in that we perform unsupervised instead of supervised zero-shot audio captioning. 
conclusion: 
 We presented a novel methodology for unsupervised zero-shot audio captioning, effectively leveraging advanced image captioning models, such as Llava. We adapted it to the audio captioning task by making use of prefix tuning and using innovative token distribution alignment techniques (using MMD or optimal transport with a cross-attention mechanism) that successfully bridged the modality gap between audio and visual inputs. Our comprehensive evaluation, both quantitative and qualitative, demonstrates that our method achieves state-of-the-art results in unsupervised zero-shot audio captioning. Remarkably, our approach, not using any audio recordings, not only matches but even sometimes, exceeds the performance of existing methods that rely on a backbone trained with extensive audio-text data such as CLAP. Moreover, our method requires only raw videos without any audio annotations, significantly enhancing its potential scalability. This positions our approach as a promising direction as a leading direction for the future of audio captioning.Limitations Although capable of addressing partial audiovisual mismatches, our method remains insufficiently robust to complete cross-modal mismatches, such as instances where the image is entirely unrelated to the audio. We reserve addressing this limitation for future work. Moreover, to achieve even higher performance in audio captioning, our method would benefit from a degree of supervision. The requirement for an external supervisory source (e.g., audio-text pairs) arises due to the method's dependence on audiovisual alignments, which makes it challenging to learn certain sounds because they cannot be seen. For example, it is almost impossible to generate a caption for the sound of the wind by merely using images.The appendix is organized as follows: The first part presents additional experiments that aid in understanding the behavior of our encoders, followed by a detailed explanation of the pre-filtering process performed on AudioSet and the training parameter details. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We do not believe that our model can present any risk. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite both the paper and the licence of each model and dataset we used. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: To determine whether the tuned prefixes only specify the task and help avoid visual artifacts in the captions, we analyzed the performance of the method by varying the number of image-audio caption pairs. Figure 5 illustrates that only 16 captions are required to achieve good performance, and increasing this number does not lead to further improvement. In fact, while the SPIDER, CIDEr, and ROUGE metrics remain stable, the SPICE, METEOR, and BLEU scores even show a slight decline. This suggests that prefix tokens only serve to specify the task to the language model and do not contribute to learning the captioning task itself. We hypothesize that this prefix tuning could potentially be replaced by a carefully chosen hand-crafted prompt.Figure 5 : Captioning performances according to the number of image captions. After 16, the performance does not improve, indicating that the prefix tuning does not play an important role in the learning process, it just specifies the task. OT , DALI MMD , and contrastive learning for the audiovisual pair shown in the figure ("A sound of a guitar playing"). The contrastive learning method maps the audio tokens to a very restricted subset of the space, without any overlapping with the image distribution. Interestingly, while DALI MMD only learns to align the expected value of the distribution, it appears to fit the full image distribution quite well. DALI Att OT exhibits similar behavior but is less noisy: the tokens are either very close to the image tokens or much further away. This behavior is inherited from the attentive optimal transport, which assigns low weights to points considered as mismatches.Figure 6 : Audio and image tokens distribution. While contrastive learning encodes tokens in a completely separated subspace, the MMD learns a distribution similar to the image one, noisier. The attentive optimal transport fits more the image distribution, except in some points that are mapped further.Figure 7 illustrates the optimal transport weights computed by the cross-attention mechanism for the given image and its associated audio (an emergency vehicle siren). The size of the circles represents the weights. Although the overall distribution appears uniform, a closer inspection of the bottom right section reveals that many tokens have scores close to zero. These tokens likely represent the road, which does not produce any sound and thus has no corresponding match in the audio tokens. Consequently, these tokens are not considered in the calculation of the transportation coupling As AudioSet primarily focuses on audio rather than audiovisual content, it features audio events that may be unrelated to the visual event depicted. For example, many videos feature music as an audio event while displaying a static image of the album cover.To address this issue, we filtered out audiovisual discrepancies. To achieve this, we generated captions for the first frame of each second of a video using BLIP2, which we then encoded into GPT2-embedding space. Subsequently, we computed the distance between the embedding of each frame's caption and the embedding of the corresponding AudioSet labels. This process is illustrated in Figure 8 . The average distance across the video frames was used as the distance between the video and the audio. We retained the 500,000 pairs with the smallest distance. Contrastive DALI AttHumming of an engine with a woman and men speaking A man with a vanity A car engine Rustling with some distant banging and people speaking in the distance A loud noise A person is talking Wind is blowing and rain is falling and splashing Rain The sound of a waterfall A woman is speaking from a microphone Describe the sound that can be heard in this scene.The sound of a large stone structure. The sound of a bell ringing A man speaking as rain lightly falls followed by thunderThe sound of a building being built A man talking Man speaking and clinking dishes A person is seen eating a donut A person is talking A chainsaw cutting as wood is crackingThe sound of chainsaws and dirt being moved. The sound of a chainsaw cutting through a tree. A woman speaks and a cat meows '[0.0]' A person is talking A male voice and a machine buzzingThe sound of a metal being cut or a saw A person is talking Table 4 : Example of captions from our method after stage 1, using contrastive backbone and DALI Att OT Table 4 shows multiple examples of captions generated by the model using the contrastive backbone and DALI Att OT , both after the first stage of training. While the captions from the latter are accurate, using the contrastive backbone generates quite noisy/bad captions. This outlines the fact that contrastive learning is not well suited to swap the encoder of a modality with the one from another.For the MMD alignment, following (54) we use a mixture of K kernels spanning multiple ranges (being the average pairwise squared distance between samples multiplied by k with k = {0, ..., K}), with K = 5. As far as optimal transport is concerned, the first epoch of the first stage is performed with uniform transport weights. A decreasing value of λ is then used: from 500 to 100 at the end of the 5th epoch. λ = 100 is then used for the remaining epochs. We observed that the training is quite robust to small variations of λ, as long as the first epoch is performed with uniform weights. We used the Python Optimal Transport (POT)(55) library to compute efficiently the EMD. In the prefix tuning stage, we learned 16 prefix tokens on different numbers of image and audio caption pairs. We used cosine decay and a learning rate of 2e-4 for both the prefix tuning and the MLP tuning in the second stage. The prompt used for all the experiment is: "What sound can be heard in this scene ?". We used version 1.5 of Llava: the image encoder being the ViT-L@336px version of CLIP (14) , and the mapping network between image tokens and the LLM embedding space a 2 layer-perceptron. As for the audio encoder, we start from a pre-trained CAV-MAE (36) model in order to speed-up convergence. This model consists of 12 transformer encoder layers of dimension 768 and one final linear layer that maps the output of the transformers to a 1024-dimensional space (dimension of the CLIP encoder). Note that this encoder is way smaller than the image one (88M vs 428M parameters), because the training set used is 2 orders of magnitude smaller than the CLIP one. All the trainings were performed using 4 Nvidia A100 gpus, and all the steps of the training can be performed within 30 hours of compute on those machines.For DALI Att OT , we computed the weights of the optimal transport by averaging on the feature dimension the tokens resulting from cross-attention and then feeding them through a softmax with temperature (we used τ = 20).More formally, the weights of the transports α Att and β Att are computed as follow:EQUATIONEQUATIONwith A and I being the audio and image tokens,W Q i , W K i , W V i ; W Q a , W K a , W Va the image and audio projections, respectively, and τ a hyper-parameter. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.https://github.com/hugomalard/AnEyeForAnEar.git 38th Conference on Neural Information Processing Systems (NeurIPS2024). 


Decision: This paper presents a method to align tokens created by an audio encoder with those from a pre-trained image encoder (Llava-1.5) to allow zero-shot description (captioning) of audio and audio-visual data. It introduces an attention-based distribution matching to the alignment to be able to account for content that appear in one modality but not the other (e.g., background music or the visual background), the so-called modality gap. The paper compares two different alignment methods: maximum mean discrepancy (MMD) and optimal transport (OT), with the attention only applied in the case of optimal transport, over a baseline approach of contrastive learning (widely used). Prefix tuning (p-tuning) is used to guide the image captioner towards audio captions (i.e. mention audio-relevant content), which is also a novel application of this method.

The model is trained on AudioSet and evaluated on AudioCaps and Clotho using standard captioning metrics. While several metrics are reported, the primary one used by the community is SPIDEr, and by this metric on AudioCaps (in-domain to training) the proposed approach when using audio-visual input (0.1946) outperforms a comparison supervised system (0.1830), an image-only system (0.1499), and other variants of the proposed system such as the audio-only version (0.1592). On the out of domain and audio-only Clotho dataset the supervised baseline performs best (0.097) followed by the MMD variant of the proposed system (0.0655). Note that state of the art performance on Clotho was around 0.3 for systems like QwenAudio and AudioFlamingo and around 0.5 on AudioCaps. The proposed model also maintains performance on the image task. 

Reviewers agree that the paper is well written and that the goal of the paper, its contributions, the execution are all clearly described. There are a good set of experiments (even "too many metrics"), visualizations of the expected improvement in alignment of the visual and audio token distributions, and overall the application of MMD or OT learning improves the alignment as demonstrated by the experimental results over the dominant baseline approach of contrastive learning. Reviewers agree that the paper puts together existing components in a novel and interesting way and that the cross-modal learning problem is relevant and deserves more work, since there is a significant gap in investment between image and audio modalities. 3 reviewers recommend a weak accept of this paper after the rebuttal phase (which resolved some initial questions) and the paper was discussed thoroughly, which would allow authors to prepare an improved camera-ready version.

On the flip side, the work may not be very impactful, since:
- the target audience may be small: it's not clear if the approach could be applied to other domains) since the task definition is very narrow and leaves out comparisons with other recent and coming LLMs. though the task is practically relevant (as confirmed by one of the reviewers), given current trends, other LLMs may quickly raise the bar, reducing the potential impact of this paper in practice.
- the performance is good on the "in-domain" task, but drops significantly on out-of-domain tasks (other databases). while the authors' definition of "zero-shot" use is consistent with other previous work (and their evaluations are fair), readers would reasonably expect better performance on Clotho for a "zero-shot" approach than what the method presently delivers
- experimental results are somewhat mixed and require a lot of attention to detail in order to fully understand. authors were able to increase reviewers' appreciation for the work and its theoretical justification during rebuttal, but it seems that readers (even of an improved camera-ready version) may still have other questions which they would not be able to resolve without access to the authors explanations

Reviews: review: paper_summary: paper_summary: The paper under review looks at the problem of captioning of short audio-video clips, leveraging existing multimodal large-language-models. While many strong image captioners exist due to the abundance of image/caption pairs for training data, for clips of non-speech audio, the amount of data available are comparably much less. To build a performant captioner that can produce accurate captions for clips contained either of both audio and visual modalities, the paper proposes a framework that leverages LVLMs, adapts them to produce more audio-centric descriptions via prefix tuning, and most critically "aligns" the distributions of the audio and visual tokens to match making the tokenized output from the visual or audio encoder interchangeable. The main novelty is the use of Optimal Transport to align the visual and audio modalities and modifying the loss with cross attention terms to improve the alignment. While this DALI_OT^Att is not a large improvement over DALI_OT or DALI_MMD, the experimentation of these modality alignment methods and demonstrated improvement over a constrastive loss is a novel contribution. The analysis and plot of the multimodal embedding space of the various approach to align the A-V modalities further demonstrates the effectiveness. Overall the paper does demonstrate how LVLMs can be utilized to build a strong A-V captioning system.
strength_weakness: strength_weakness: The paper has framed the problem of A/V captioning in a somewhat narrow way of leveraging LLVMs with the only comparison to a strong supervised baseline in CLAP while there are others such as GPTo, Gemini 2 or Llama 3.

Section 3.1 is rather confusing. It seems Figure 2 was revamped and the references to the figure in section 3.1 was not completely updated. Figure 2 should also include a description of steps 2d and 2e in the descriptive caption (the description of 1c needs updating too I believe. The reference on line 177 for 2-c should probably be 2-b; line 190 2-c should be 2-d; and on line 196 2-d changed to 2-e. Overall, it would be clearer to first indicate the stage and then explain it, e.g. "1-a) prefix tuning the prefix tokens" rather than the other way around. Also each of the bolded titles in section 3.1 should correspond with a stage in Figure 2 and include the stage number for better clarity.

In section 3.2, it would be helpful to the reader to provide concrete equations on how to compute the quantities and eqns 1-5. Having these in an appendix would be fine. While its understood that the code will eventually be released, having the equations that the code is intended to implement is always useful for clarity.
questions: questions: Has there been any analysis done in comparison to other large, strong multimodal LLMs such as  GPT-4o, Gemini 2 or Llama 3?
limitations: limitations: Yes
paper_summary: paper_summary: The paper proposes a method for how to adapt VLM to perform audio captioning. The pipeline is as follows: 1. perform few-shot prefix tuning on images to caption the audio. 2. Use multi-modal alignment methods, namely MMD or OT, to align token space distribution of audio and visual. 3. Distillation for audio-only model from the visually-informed audio caption model. Experiments show that the proposed method outperforms contrastive-based alignment and also the pretrained CLAP model that is used for audio captioning.
strength_weakness: strength_weakness: 1. There is lack of theoretical derivation or enough insights for showing why the OT and MMD can outperform the standard contrastive learning. Figure 4 shows the PCA plots for different methods, however it is unclear why MMD or OT is able to stay closer to CLIP encoder. I think more explanations or theoretical justifications are needed.

2. The experiment results seem mixed, in particular Table 2 for Clotho. Contrastive method is better in several metrics without audio-visual distillation, in particular CIDEr and SPIDEr. Also the SPICE performance is very close to the proposed method. These two metrics, CIDEr and SPIDEr from my point of view is also very important since they are designed specifically for caption task. Why would Contrastive outperform the proposed methods in this dataset? But in AudioCaps, the proposed methods beat the contrastive method by a lot. The inconsistency results across these two datasets make me feel confused.
questions: questions: See Weaknesees.
limitations: limitations: N/A
paper_summary: paper_summary: This paper presents a method to align tokens created by an audio encoder with those from an image encoder to allow zero-shot description of audio and audio-visual recordings. It introduces an attention-based matching to the alignment to be able to account for objects that appear in one modality but not the other (e.g., background music or the visual background). The paper compares two different alignment methods: maximum mean discrepancy and optimal transport, with the attention only applied in the case of optimal transport. And prefix tuning is used to guide the image captioner towards audio captions.

The model is trained on AudioSet and evaluated on AudioCaps and Clotho using standard captioning metrics. While several metrics are reported, the primary one used by the community is SPIDEr, and by this metric on AudioCaps (in-domain to training) the proposed approach when using audio-visual input (0.1946) outperforms a comparison supervised system (0.1830), an image-only system (0.1499), and other variants of the proposed system such as the audio-only version (0.1592). On the out of domain and audio-only Clotho dataset the supervised baseline performs best (0.097) followed by the MMD variant of the proposed system (0.0655). Note that state of the art performance on Clotho was around 0.3 for systems like QwenAudio and AudioFlamingo and around 0.5 on AudioCaps.
strength_weakness: strength_weakness: Clarity:
* The use of many metrics dilutes the clarity of the results and their interpretation. The metrics should be prioritized and the systems they are comparing discussed accordingly. The discussion of results seems to treat all metrics as somewhat interchangeable and makes broad statements about which systems seem better than others. It would be much clearer and easier to follow if differences in performance on specific metrics were tied to qualitative differences in model behaviors and when there is one main metric (i.e., SPIDEr) to focus on that for overall system comparison.
* Line 96 states that "Audio data often lacks the richness and context provided through the visual modality, resulting in less precise and more subjective descriptions of the audio content." While I agree that audio captioning datasets typically contain more subjective descriptions of their content, I think this is an issue with the data that is selected to be captioned and the question asked of the captioner as opposed to a shortcoming of audio in general. With the right data and right task, audio captioning could be much better defined. I believe this sentence should be modified to focus on existing datasets as opposed to the general modality.
* No mention is made of the size of the various models or model pieces involved or how much data they were trained on. This is especially acute for comparing the proposed system to existing baselines from Shaharabany et al and Salewski et al.
* I can't understand what the ablations are showing, these should be described in the body of the paper, not just shown in tables.

Performance:
* Overall, the reported SPIDEr results (0.1-0.2) are fairly low compared to SOTA approaches for Clotho (0.3) and AudioCaps (0.5). It's clear that this zero-shot approach does not completely solve the problem. It would be useful to discuss this in the paper.
* Additionally, the differences in performance between the proposed variants are not that large, bringing into question the value of including multiple-such approaches.

Minor issues:
* Figure 2: sub-figures (d) and (e) are not mentioned in the caption
* Equation (4): please use \langle and \rangle as opposed to < and > for brackets
* Tables 1 and 2: Order the rows within each block intentionally. I would recommend ordering by SPIDEr.
* Figure 4: It's hard to distinguish between this many colors with this level of transparency. Maybe you could use different symbols instead of different colors.
questions: questions: What exactly was done in the ablation studies listed in Tables 1 and 2? What do the rows mean? What was taken away?

Why wasn't the attention weighting used with MMD in addition to OT?

Does the audio-visual alignment treat each instance as a "bag of tokens"? If so, would there be value in considering the sequence of tokens in this alignment as opposed to just their distribution?

Were any categories held out of training so that they could be evaluated in zero-shot category generalization? Or was the focus only on zero-shot modality generalization?

Was the filtering of audiovisual discrepancies described in appendix C applied to both training and test data or just to training data?
limitations: limitations: The paper has a section focusing on limitations and does an excellent job discussing them, providing true limitations of the approach.
paper_summary: paper_summary: This paper proposes to use a well-trained visual LLM (specifically Llava-v1.5) to perform audio captioning in a zero-shot fashion. The authors propose a training framework which comprises 2 stages and 5 sub-steps. The authors also propose to use Maximum Mean Discrepancy (MMD) and optimal transport (OT) as the loss function to align audio and visual representation space.
strength_weakness: strength_weakness: 1. The motivation is not clear. The author must specify in which scenario zero-shot audio captioning is needed, with concrete examples that the proposed method will help in practice, such as generating a particular captioning style or targeting a specific type of audio. As far as I can see in the paper, the authors only did experiments on two standard tasks (AudioCaps and Clotho) and both tasks have a reasonably-labelled size, and the performance is far from the state-of-the-art.

2. The method lacks theoretical grounding or insights. The author says they randomly choose one frame from the 10 frames in a video as the image and train to pull the image representation space to be close to the audio space. The author did not analyse how related the image is to the audio. How often does the image reflects what can be heard, and how often it does not? 

3. The experimental setup is questionable. 
  - The author says clearly in line 167 that they performed p-tuning with audio captioning data "We trained the additional tokens using image-text pairs, with the text describing the sounds associated with the images". This clearly sets this method apart from "zero-shot learning" which refers to having no data at all. If the author means "zero-shot" for a specific audio caption dataset, they need to be clear about it. 
  - Regarding experimental results: I wonder if the comparison between the proposed method and ImageBind is fair. They also did not say how much data they used for ImageBind p-tuning. If the authors just took the numbers from Shaharabany et al., then putting the numbers in the same table is clearly unfair. I might have missed something here but without a convincing explanation, I do not think the experiments are valid. Why is ImageBind not included in Table 2?
  - I would be interested to see whether adding audio to the audio-visual description would give better performance or not. It seems that the audio information is all included in the visual information. Getting back to Weakness 1 - what is the point of doing this task then? What extra did you gain?

A few writing problems:
1. The referencing style makes it difficult to locate various papers. 
2. Fig. 2 captions are incomplete.
questions: questions: See Weaknesses.
limitations: limitations: There is a limitations section but no discussion on potential negative social impact.

Comments: title: Rebuttal answer
Response
Answer to further questions
Further Questions
Response to rebuttal
Response to the Authors
comment: Dear Reviewer,

I hope this message finds you well. We wanted to kindly follow up to see if our responses have addressed your concerns regarding our work. As the deadline is approaching in an hour, we would greatly appreciate any feedback you can provide before the end of the discussion period.
Thank you for providing this additional clarification about the original visual model. Given that the proposed system does maintain its visual performance while also gaining the ability to perform audio-only captioning (on Clotho) and improving its performance for audio-visual captioning (AudioCaps), I do think the relatively poor absolute performance can be overlooked. I will raise my rating by one point to 6: Weak Accept.
Thanks for the last answer and those new remarks. However, we do not agree with your points:

1-Concerning the difference between image only and audio only, it's important to notice that our 
method targets AUDIO captioning, which consists in generating a caption while having access to an AUDIO and not an image. Therefore, there is no point in trying to compare audio and image performance.

2-The prefix tuning does not introduce any bias or unbalance between methods as the prefixes are ALSO used with the image-only method, because as previously mentioned, the visual-LLM off-the-shelf is not able to produce audio-centric captions.

We would encourage the reviewer to perform a more careful reading of the paper as he/she seems to have missed a lot of important details which are very clearly stated.
Thank you for providing more details. Please find my following concerns:

1. In those examples given, it seems visual-LLM could generate similar captions and hence achieve similar results since those scores are really in the very low region to draw any useful results (e.g. a SPIDEr value of around 0.19 in comparison to the SOTA which is over 0.50). Looking at Table 1, the comparison to the image-only baseline is actually __mixed__. A significance test needs to be performed here to show real improvements. 

2. I wonder whether the main gain comes from the p-tuning. Since the p-tuning requires some few-shot samples, and hence the selection of those samples is important and I am not sure if that should be counted as "zero-shot" or should be thought of as "few-shot". Given the above two, I strongly think that the authors should study what happens without p-tuning and what p-tuning brings to this training pipeline.

3. Regarding the task itself, it would be interesting to see if the model could capture "audio" that is not included in any image training data. The examples such as "a man speaking" must have appeared multiple times in the image-text training pairs. 

Overall, I am not sure if the results are strong enough and the task still does not feel practical to me. I have to keep my original rating.
Thank you for taking the time to review our rebuttal. Let us address the last concerns which are probably due to a misunderstanding.

On the practical scenarios for zero-shot audio captioning, two points need further clarification here:
 - “zero-shot audio captioning” refers to the scenario where the system is trained without having access to audio-text pairs. It  is useful in diverse use-cases, notably, audio description for the hearing impaired... where footage is largely available, but audio annotations are sparse.
 - While it is standard to evaluate “zero-shot category generalization” as part of classification systems, it is unusual to do so in the captioning application. In general, one would have to first extract “categories” (classes) from the ground-truth captions, which is a problem per se. We agree that the idea is interesting but it is clearly beyond the scope of this work, where the definition of “zero-shot” is different: i.e., no audio-text pairs during training.

On the validity of the experimental setup:
> I disagree with the authors [...] After all, how would one generate an audio description when looking at an image?

We must disagree with the reviewer. It is clearly valuable to use audio description from audiovisual data, since such description correlates to a great extent to both the image and audio modalities.  For example, in audiovisual data, a textual description like "a train passing by" not only describes the visual element of a train but also implicitly refers to the expected sounds of a train—such as the clickety-clack of wheels on tracks and the whistle blowing.  This approach allows the model to shift focus from descriptions with heavy visual cues to auditory cues, akin to style transfer. To mention a few other examples: think of an image of a person holding a phone with the mouth open (“person speaking on the phone”), or one of an inclined bell (“bell tolling”), or a person operating a jackhammer, etc. We'd like to invite the reviewer to do an image search with these descriptions to see the result...

> I do not see the point [...] 

First, please note that this is merely a pre-processing (which only uses 16 image-audio description pairs) and not the central component of the system.
In subsection "Prefix-tuning with image-caption pairs," we explained that LLVMs tend to generate descriptions heavily focused on visual features rather than auditory ones even when asked, "What sounds can the objects in this image generate?" This visual bias negatively impacts audio captioning metrics and calls for a solution to shift descriptions toward auditory content. Importantly, no large image-caption (audio-centric caption) dataset is required, as the audio-centric descriptions **can be easily composed by hand**, given the small number needed. As demonstrated in Appendix A, using more than 16 pairs offers no additional benefit. Without the p-tuning process that reorients the model's descriptive style, the generated captions would continue to exhibit a strong visual bias, undermining the benefits of the audiovisual distribution alignment. This is confirmed by our preliminary experiments. 

> In response to my point on comparison to "ImageBind" [...] I think the authors should compare their method to Shaharabany et al. without p-tuning.

We disagree. While Shaharabany et al. rely on an "audibility score" as part of their training loss to steer their models toward generating audio-centric descriptions, our approach employs p-tuning for the same purpose. This difference in methodology is important, as p-tuning effectively reorients the model's descriptive style from visual to auditory, addressing the inherent visual bias of LLVMs. Without p-tuning, our model would continue to produce visually biased descriptions. Therefore, comparing our method to Shaharabany et al. without p-tuning would not only undermine the intent of our approach but also fail to provide a fair comparison, given that both methods are designed to achieve the same goal through different means.
Thank you for your positive feedback. 

The original image captioner's performance remains **necessarily**  intact. Indeed, **all** the parameters of the original system, especially the LLM’s, are kept frozen during the whole process. Hence, merely  re-using the original image backbone (along with its MLP), we do recover the original model and indeed the original performance. Therefore, there is no need to perform an additional evaluation as the original model remains **intact**. 

We do hope this clarification resolves your last concern.
Thank you for your positive feedback
Thank the authors for providing detailed response, and it basically solves my concerns about the paper. I would like to raise the score to weak acceptance considering the novelty part of the paper and these additional insights brought by the rebuttal response.
I would like to thank the authors for their rebuttal. It has resolved several of my questions and issues around clarity in the paper. Several issues remain, however. 

> Using too many metrics muddies the results.

This summarization is useful and improves over what was in the manuscript original, although it is still not entirely easy to follow.

> We want to clarify that our approach extends image captioners to audio without compromising their original image performance.

Is this demonstrated quantitatively in the paper? If it could be, then that would go a long way to resolving this concern and would likely increase the significance of the work.

As it currently stands, I have read the reviews and rebuttals and would still like to keep my rating as is.
I thank the authors for providing detailed responses. However, I do not think my concerns are adequately addressed for the following reasons:
1. In response to my point 1, the author did not provide examples where zero-shot audio captioning is useful, e.g. categories held out of training so that they could be evaluated in zero-shot category generalization, as pointed out by reviewer rTvm. I still doubt whether this task under the author's specific setup is useful or not.

2. In response to my question about the experimental setup:
  - I disagree with the authors (including the cited paper) that having audio captioning (description particularly used for audio) is a practical setup. After all, how would one generate an audio description when looking at an image? I do not see the point of the experiment when you have images and descriptions for audio for training. Why would we have such a weird dataset? This is fundamentally questionable from a practical point of view. To validate this point, please assume there are also no pairs of image and audio-captioning text and re-run the experiments.
  - In response to my point on comparison to "ImageBind", the authors clearly say "Shaharabany et al. did not perform p-tuning", whereas, given my concerns about their p-tuning setup, I think the authors should compare their method to Shaharabany et al. without p-tuning.

Given all the above, I am not convinced by the task setup and experiments performed. Therefore, I would keep my score as it is.