Title: An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching
Venue: NeurIPS 2024
Accepted: yes
Keywords: Multimodal representation learning

Audio Captioning

Image Captioning

Audio-Visual

Large Language Model
Abstract: Multimodal large language models have fueled progress in image captioning. These models, fine-tuned on vast image datasets, exhibit a deep understanding of semantic concepts.
In this work, we show that this ability can be re-purposed for audio captioning, where the joint image-language decoder can be leveraged to describe auditory content associated with image sequences within videos featuring audiovisual content. This can be achieved via multimodal alignment.
Yet, this multimodal alignment task is non-trivial due to the inherent disparity between audible and visible elements in real-world videos. Moreover, multimodal representation learning often relies on contrastive learning, facing the challenge of the so-called modality gap which hinders smooth integration between modalities. In this work, we introduce a novel methodology for bridging the audiovisual modality gap by matching the distributions of tokens produced by an audio backbone and those of an image captioner. Our approach aligns the audio token distribution with that of the image tokens, enabling the model to perform zero-shot audio captioning in an unsupervised fashion. This alignment allows for the use of either audio or audiovisual input by combining or substituting the image encoder with the aligned audio encoder. Our method achieves significantly improved performances in zero-shot audio captioning, compared to existing approaches.

An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment

Abstract

Multimodal large language models have fueled progress in image captioning. These models, fine-tuned on vast image datasets, exhibit a deep understanding of semantic concepts. In this work, we show that this ability can be re-purposed for audio captioning, where the joint image-language decoder can be leveraged to describe auditory content associated with image sequences within videos featuring audiovisual content. This can be achieved via multimodal alignment. Yet, this multimodal alignment task is non-trivial due to the inherent disparity between audible and visible elements in real-world videos. Moreover, multimodal representation learning often relies on contrastive learning, facing the challenge of the so-called modality gap which hinders smooth integration between modalities. In this work, we introduce a novel methodology for bridging the audiovisual modality gap by matching the distributions of tokens produced by an audio backbone and those of an image captioner. Our approach aligns the audio token distribution with that of the image tokens, enabling the model to perform zero-shot audio captioning in an unsupervised fashion while keeping the initial image captioning component unaltered. This alignment allows for the use of either audio or audiovisual input by combining or substituting the image encoder with the aligned audio encoder. Our method achieves significantly improved performances in zero-shot audio captioning, compared to existing approaches. 1

1 Introduction

Recent progress in image captioning has been driven by methods integrating Large Language Models (LLMs) with vision encoders (1; 2; 3). The impressive capabilities of Vision Language Models (VLMs) stem from supervised training on large image-text collections and extensive parameterization. Even the smallest VLMs exceed more than a billion parameters (4; 5). These models excel in generating human-like text descriptions for images and understanding complex semantic relationships. Such capabilities can be potentially extended to other tasks, beyond image analysis.

One such task is audio captioning, in which large-scale audio-text collections, with precise descriptions of the audio content, are lacking. Recent works have proposed diverse solutions to address this limitation. Solutions include augmenting class labels with phrases such as ' This is a sound of ' (6; 7) or prompting LLMs to generate natural language descriptions directly from class labels (8). Further, datasets from related domains like Speech (9) or Music (10; 11), have been considered to enlarge training data. While such methods show impressive results, scaling them further is challenging due to the rarity and low quality of such labels-captions. In particular, these approaches cannot cope with the annotation noise encountered in existing audio datasets, especially in the largely adopted resource: audio tracks of AudioSet (12), which is only partially and weakly labeled.

Previous

Encoder

Image

Image embeddings

Audio embeddings

talking."

"A person

is

"A person is

Contrastive

Learning

seen eating

donut."

Audio

Encoder

LLM

"The sound of a

bell:

ringing."

"The sound of

a

large stone

Image

structure."

Encoder

"A chainsaw's

Distribution

Alignment

roar through wood."

Audio

Joint audio-visual

"The sound of

小

Encoder

space

dirt being moved.

chainsaws and

11

Ours

We hypothesize that VLM-based image captioning models, inherently possess the knowledge to perform audio captioning. Thus, joint image-language representations of VLMs can be used to describe auditory content associated with image sequences from videos featuring audiovisual content, eliminating the need for handcrafted audio-caption pairs. To our knowledge, this is the first attempt to leverage VLMs for this purpose. Extending VLMs to audio captioning is non-trivial due to the inherent disparity between audible and visible elements in real-world videos. For instance, soundproducing objects in images may be occluded or out of view, while visible objects might not generate sound. This mismatch hinders connecting visual and auditory modalities seamlessly.

Multimodal representation learning (13), crucial for tasks like image captioning, is often accomplished through contrastive learning (14; 6). This paradigm aligns features across modalities by maximizing agreement between similar samples and minimizing agreement between dissimilar ones. Despite its effectiveness, contrastive learning faces the so-called 'modality gap', where embedded data are represented in distinct, non-overlapping regions of the embedding space (15; 16) (the distance between red and green manifolds in Figure 1), limiting flexibility when conditioning modalities.

We propose a novel methodology to bridge the audiovisual modality gap by matching the distributions of tokens produced by an audio backbone and those from the encoder of an image captioner (represented as the green and blue manifolds in Figure 1), enabling the image captioner to perform zero-shot audio captioning in an unsupervised way. We define zero-shot audio captioning as the task of generating audio captions without training on manually assembled audio-caption pairs. We propose and compare two token distribution alignment methods: one based on Maximum Mean Discrepancy (MMD) (17) and a more flexible alternative using Optimal Transport (OT) (18), possibly enhanced with a cross-attention mechanism. This mechanism refines distribution matching by attending to each token within a modality and determining its match with a token from the other modality based on semantic similarity. We find this approach to result in more accurate and coherent joint audiovisual representations.

Acknowledging the challenges of zero-shot audio captioning, we enhance caption quality using 'prefix tuning', a technique commonly employed across domains to adapt LLMs to diverse tasks in a few-shot manner (19; 20; 21). This tuning process conditions the model with few image-audio caption pairs, prompting it to generate audio-centric captions and thereby boosting performance across standard audio captioning metrics (22; 23; 24; 25; 26; 27).

Notably, our methodology allows for the use of either audio or audiovisual input by combining or substituting the image encoder with the aligned audio encoder. As a result, our approach extends the ability of the VLM to audio captioning without compromising its image performance. Moreover, the method achieves significantly improved performances in zero-shot audio captioning, compared to existing approaches.

In summary, our contributions are as follows:

We introduce a novel methodology for unsupervised zero-shot audio captioning through Distribution ALIgnment (DALI), leveraging advanced image captioning models, and suc-

cessfully instantiate it with a particular image captioner (namely Llava 1.5), as we obtain state-of-the-art zero-shot audio captioning results without any supervision from annotated audio data (see Section 5).

To bridge the 'modality gap', we propose and study two multimodal token distribution alignment methods exploiting MMD or Optimal Transport (OT). The latter-never used before in this context-is enhanced with a cross-attention mechanism for greater robustness.

We introduce the use of prefix tuning to guide the image captioner towards audio captioning, adapting the model with a few shots of image and audio-caption pairs (only using the textual descriptions of sound and not the audio signals).

Finally, our method supports both audio and audiovisual inputs, extending the image captioner's capabilities without compromising its performance in image analysis tasks.

2 Related works

Vision Language models Large Vision Language Models (VLMs) typically use a pre-trained LLM that processes both visual and textual information. The image encoder usually consists of a ViT (28) trained via multimodal contrastive learning using millions of image-text pairs such as the ones used in CLIP (14). Integrating visual information into the LLM is usually done by concatenating image tokens extracted by a vision encoder and then further transformed by an MLP, with textual tokens from a prompt. This straightforward architecture can be easily adapted to other modalities by simply replacing the input tokens to accommodate the new input modality.

Audio Language models Recently, a similar paradigm emerged in the audio application domain. Various works rely on both LLMs and audio encoders trained jointly on millions of audio-text pairs, similarly to CLAP (6). Listen Think and Understand (LTU) (29), Pengi (11), QwenAudio (9) and AudioFlamingo(10) rely on a language model fed with tokens from both audio and text modalities. These large models require substantial volumes of training data. For instance, AudioFlamingo has been trained on nearly all publicly available labeled audio data. One limitation is that the quality of the audio-text data is generally not comparable to that of image-text data. This situation is made worse by the fact that audio data labelling is inherently difficult due to the potential overlap of multiple sound sources at different acoustic levels, which sometimes makes it hard to perceive some of the audio classes in presence. Despite this limitation, AudioFlamingo and similar models exhibit remarkable performance. To further enhance the capabilities of audio-language models it is of paramount importance to explore alternative paradigms beyond strong reliance on (often unreliable) annotations and supervised learning. This is the goal pursued in this work.

Audio captioning Training large audio captioner models, as mentioned previously, requires large volumes of audio and textual descriptions and substantial computational resources. To address this limitation, some approaches leverage existing robust backbones to enable zero-shot audio captioning. ZerAuCaps (30) uses CLAP (an 80.8M-parameter model trained on 3.4M audio-text pairs) to infer potential classes from a predefined word bank (45 words) before using them to prompt an LLM (OPT (31) 1.3B), generating multiple plausible captions. The final caption is the closest to the audio in the CLAP space. However, this method is constrained by the finite set of classes in the word bank, limiting its application to predefined scenarios. The most closely related work to ours is that of Shaharabany et al. (32) who also performed unsupervised zero-shot audio captioning. The authors adopted the audio encoder of ImageBind (33) (86M parameters trained on AudioSet), a multimodal model trained to align various modalities in the image space via a contrastive loss. They fine-tuned a GPT-2 (34) model (117M parameters) fed with ImageBind audio tokens using two distinct loss functions for the captioning task: an audibility score ensures the generation of audio-centric captions without visual artifacts, while an ImageBind score measures the similarity between the generated caption and the audio. Despite its innovative approach, its performance falls short compared to ZerAuCaps primarily due to ImageBind's less effective handling of audio.

Audiovisual Alignment Multimodal alignment aims to encode information from different modalities such as text, image, or audio into a shared semantic space, enabling cross-modal relationships and multimodal retrieval ( e.g. , retrieve the image closest to a caption) (13). Despite the abundance of web video datasets, aligning the audio and image/video modalities remains a challenge. The conventional contrastive loss (35) often fails to effectively co-train the encoders. Unlike the relatively clean associations in image-text pairs, audio-image pairs present a greater degree of variability, with

Training

Inference

Audiocaptioningviavisual

Tokendistribution alignment

Audiocaptioningvia

inputs and prefix tuning

tokendistribution alignment

"A dog on the grass."

"A dog is barking."

Stage 1

LLM

LLM

MLP

Image

Audio

MLP

Encoder

Encoder

Image

Audio

Encoder

Encoder

Training

Inference

Audiocaptioningvia

Audio-visual distillation

e

audiooraudio-visual inputs

dog isbarking

+

NLL loss

g

"A dog is barking."

Prefixtokens

Stage 2

LLM

LLM

LLM

Image tokens

Audio tokens

MLP

MLP

MLP

MLP

MLP

..

Prompt tokens

Learnable

Image

Audio

Audio

Image

Audio

Replace

Encoder

Encoder

Encoder

Encoder

Encoder

小小

the content described in the audio often differing from that depicted in the corresponding image or video. Consequently, stable pre-training needs augmenting the contrastive loss with a reconstruction loss (36; 37; 38), reducing the similarity of the representations between the two modalities.

3 Method

Current Large Vision Language Models (VLMs) exhibit a sophisticated understanding of concepts related to real-world objects in the visual domain. This capability arises from optimizing the language model, already containing a lot of intrinsic knowledge, so as to extract additional information from image features and learn how to describe the (visual) world. We hypothesize that the knowledge gained through this process is sufficiently general to be transferable across different modalities. To test this hypothesis, we developed a methodology that employs an VLM's image captioner to generate audio captions from images, audios and audiovisual inputs, while preserving the original performance of the image captioner. Our methodology stands out for its applicability to any modality requiring integration with image data. Here, we focus on audiovisual alignment for audio captioning, benefiting from the abundance of audiovisual content in videos from web data.

3.1 Proposed Framework Overview

An image captioning system outputs descriptions of objects in an image, such as "A dog in the grass," providing visual cues. To adapt this system for audio captioning, we need it to focus on audible information instead of visuals. For example, given an image of a dog, the audio caption should instead be "A dog is barking.". Our methodology enables audio captioning from an image captioner by keeping both the image encoder and LLM frozen, ensuring that the image captioner original

performance is preserved. In our work, we choose Llava (1) (Apache License 2.0) as an instance of an image captioner, for its proven effectiveness. We use the CA V-MAE (36) (BSD 2-Clause License) audio model as our audio encoder, since it is already roughly aligned with an image backbone, which is expected to help reach faster convergence.

As seen in Figure 2, we follow a 2-stage process. First, we use prefix tuning to start re-tasking the system for audio captioning, learning tokens from a few image-caption pairs, ensuring that the textual descriptions target the audio content (stage 1, step (a)). We also align the token distribution of an audio encoder with that of the image encoder (stage 1, step (b)), then replace the image encoder with the aligned audio encoder for direct audio captioning from audio files (step (c)). We can also combine both encoders for visually informed audio captioning, using both audio and visual inputs. This setup generates audio captions to fine-tune the MLP that maps audio tokens to the language model embedding space, a process we call audiovisual distillation (stage 2, step (d)). The fine-tuned MLP then replaces the original MLP, generating robust audio captions from audio or audiovisual inputs (step (e)). In the following, we zoom on specific aspects of the proposed methodology.

Prefix-tuning with image-caption pairs Image captioners, while effective for image analysis tasks, struggle to produce accurate audio-centric descriptions when prompted to describe sounds from images. For example, when asked, "What sounds can the objects in this image generate?", the descriptions provided heavily reflect the visual characteristics of the objects. Instead of focusing on sound-related details, their outputs are dominated by visual descriptions, which are considered artifacts in the context of audio captioning. To address this problem, we adopted a prompt tuning strategy (39), aiming to guide the model to generate captions that exclusively describe audible events as depicted in Figure 2-a. Prompt tuning involves learning a portion of the prompt tokens: the prefix, that guides the model to perform a specified task (39). This method allows the rest of the model to remain frozen, and by simply removing the learned tokens, the model is able to perform its original task with the same performance. This approach requires only a small number of parameters to be trained and can be effectively trained using a few examples, a process akin to few-shot adaptation . We employed this technique to re-task the model to perform audio captioning. We trained the additional tokens using image-text pairs, with the text describing the sounds associated with the images i.e. , audio captions. Specifically, we used images from AudioSet videos and corresponding captions from AudioCaps (40) (MIT). We used negative log-likelihood as the loss function for the prefix tuning:

with n and j being respectively the image and caption token indices, N the number of elements in the batch, l the number of predicted tokens, the index I referring to the vision tokens (from 1 to n i ), C j the j -th caption token and θ the parameters of the model. Using the prefix tokens, the image captioner can now output more plausible sound descriptions.

Audio captioning via token distribution alignment Aligning the distribution of the audio encoder with that of the image one allows for substituting one encoder with another and therefore conditioning the generation either on the audio or image modality. We align the distributions (Figure 2: step (b)) using either Maximum Mean Discrepancy or Optimal Transport. More details are provided in Section 3.2. Using the aligned audio backbone, as represented in Figure 2: step (c), we feed the language model with the audio tokens along with the prefix, which enables the model to perform audio captioning. Additionally, we incorporate both the image and audio tokens, to give the ability to the model to perform audio captioning fusing visual and audible cues. This setup allows us to infer and generate, visually informed, audio captions for AudioSet balanced (12), a standard subset of AudioSet, containing ∼ 20k videos, curated such that they involve the same number of samples for each class.

Audiovisual distillation The captions generated from audiovisual inputs serve as pseudo-labels to supervise the fine-tuning process of an audio captioner with an audio-only setup. This process refines only the MLP that maps the audio tokens to the LLM embedding space. Training employs a negative log-likelihood loss, as per equation (1), albeit without conditioning on image tokens. Instead of relying on a limited number of real captions, pseudo-labels are employed. This procedure is

illustrated in Figure 2: step (d), and optimized through the following loss:

with A referring to the audio tokens (from 1 to n a ).

Audio captioning via audio or audiovisual inputs With all components trained, our model can perform audio captioning using the aligned audio encoder and fine-tuned audio MLP, or visualyinformed audio captioning by feeding to the LLM the image tokens in addition to the audio tokens (shown in Figure 2: step (e)). Notably, the original image captioning task can still be performed at its original performance level by using only the image tokens, without the audio.

3.2 Token distribution alignment

Many studies on multimodal alignment typically use a contrastive loss on the averaged output tokens (14; 41). This method merely aligns token mean-statistics, potentially leading to information loss, when the full token distribution is needed. Llava (1) for instance, makes use of the full token distribution as input, naturally creating the need for a full token distribution alignment, to allow for swapping the pretrained encoder with a new one targeting a new modality. Moreover, contrastive learning faces the so-called 'modality gap' problem where each modality is encoded in distinct sub-regions of the embedding space (see Figure 4). For all these reasons, replacing image tokens with audio tokens obtained by alignment through a contrastive learning approach may yield undesirable responses from the language model. In light of these issues, we propose to learn the full image token distributions, rather than relying solely on contrastive learning, so as to facilitate knowledge transfer. We refer to our distribution alignment framework as DALI which stands for Distribution ALIgnment. We study two variants of distribution matching.

Modality alignment through MMD Maximum Mean Discrepancy (MMD) serves as a robust measure of dissimilarity-or more specifically, discrepancy-between two probability distributions. MMD has been shown to be effective in distribution alignment, notably, but not exclusively, for domain adaptation (42; 43). It computes the distance between the expected values of two distributions, denoted by P and Q , within a feature space characterized by a mapping Φ : X →H where H is a reproducing kernel Hilbert space (RKHS):

In practice the MMD is computed using the kernel trick (44). In our work, we consider the Gaussian Radial Basis Function (RBF) kernel and use MMD as a loss function to directly align the audio and image token distributions. We term this distribution alignment through MMD: DALIMMD.

Modality alignment through Optimal Transport Optimal Transport (OT) stands out as a powerful method for aligning probability distributions (45). Despite its successful application across various domains such as unsupervised domain adaptation (46), image generation (47), and style transfer (48), its application to multimodal representation alignment remains relatively unexplored. In our context, where the alignment of audiovisual modalities is paramount, OT emerges as a natural solution for its intrinsic capacity to integrate the geometry of the underlying space, enabling optimal distribution matching. Discrete optimal transport considers two distributions x ∈ R N and y ∈ R M , where N and M represent the number of samples in each distribution. These distributions are represented by their empirical measures: α = ∑ N i =1 α i δ x i and β = ∑ M j =1 β j δ y j , and OT seeks a coupling γ ∈ Π( α, β ) between them that minimizes a transportation cost. The problem can be formalized as:

where D ∈ R N × M is the cost matrix, ⟨· , ·⟩ F denotes the Frobenius dot product, and γ ∈ R N × M represents the transportation coupling matrix. The minimum of the optimization problem can be interpreted as a distance (45). When considering the square of the l 2 norm, the distance is known as the Earth Mover Distance (EMD). In this work, we align the audio and image token distributions using EMD and refer to this approach as Distribution ALIgnment through Optimal Transport (DALIOT).

Improving cross-modal correspondence with attentive distribution alignment Optimal transport is typically applied assuming uniform distributions of the weights { α i } N i =1 and { β j } j M =1 . However, in audiovisual content, objects producing sound are often occluded, with the visual information serving to complement the auditory content. The classic formulation of OT enforces strict mass preservation, needing that all mass from the source distribution is transported to the target distribution. This constraint becomes problematic in audiovisual alignment, where not all tokens from one modality may find a match in the other. While alternative methods exist to circumvent these constraints, such as Unbalanced Optimal Transport (UOT) (49), which replaces this rigid requirement with a soft penalization term, the tuning of such hyper-parameter poses significant challenges (50). Alternatively, we propose learning the weights α i and β j via a cross-attention mechanism which offers as well a robust approach. This mechanism, attending to both modalities, enables learning which objects and/or sound-related tokens are present in both modalities. Tokens encoding modality-specific information, e.g., an occluded source, or an object apparently not generating a sound, receive lower weights. Thus, this approach promotes the alignment of modality-shared content. We provide the details of this cross-attention mechanism in Appendix F. It is important to note that to avoid the trivial solution that would put all the weights to 0 except the one of the closest tokens, we regularize the entropy of the weights.

The final loss consists of a weighted sum of the optimal transport distance and the entropy of distribution weights:

where H ( · ) denotes the Shannon entropy and OT ( x, y, α Att , β Att ) represents the optimal transport between α Att and β Att , the weights at the output of the attention mechanism, with the cost matrix computed from x and y . An intuitive depiction of the method is illustrated in Figure 3. Note that the cross-attention layers are only used for training and discarded afterward. We refer to this alignment method as DALI Att OT .

Tokendistributionalignment

Tokendistributionalignment

withcrossattention

Att

Image tokens

000

Audio tokens

Image

Audio

Leamable

Image

Audio

Encoder

Encoder

Frozen

Encoder

Encoder

Att

4 Experiments

Our experiments focus on AudioSet, a widely explored large-scale collection of videos from YouTube. We evaluate our proposed approach on two popular and publicly available human-annotated audio captioning datasets: AudioCaps and Clotho (Tampere University licence), using conventional audio captioning metrics. We also provide details of our training and fine-tuning procedures.

Datasets AudioSet (12) is a dataset (license CC BY 4.0) composed of 2 million YouTube videos with annotations indicating the presence of environmental sound events at the clip level (without precise time location), spanning 527 classes. The dataset includes reliability scores for annotations, some below 50%. Despite a diverse range of classes, a significant portion predominantly features music or speech content. Many videos show notable disparities between their visual and auditory content, thus attempting to align audio and image representations without any pre-filtering of the content is challenging. Similarly to Nagrani et al. (51), who filtered the dataset by balancing the classes, we train our models on a subset of 500k videos from AudioSet. We chose them by computing the similarity between the noisy audio labels and an image caption generated by BLIP-2(52) (chosen for the conciseness of the captions), to remove the biggest discrepancies between images and audios. More details on this filtering process are given in Appendix C. For each video, we extracted 10 frames (the first of each second), and we randomly chose, during training, one of these 10 frames for each video.

Evaluation metrics We evaluated our system on all known audio captioning datasets which are publicly available: AudioCaps (40) which contains AudioSet audio samples captioned by humans, and Clotho (53), which is composed of audio samples from FreeSound, also annotated by humans. Performance was measured using 3 standard captioning metrics: METEOR (24), ROUGE-L (23), and SPIDEr (27), which combines both the CIDEr (25) and SPICE (26) scores. Additional evaluation metrics are available in Appendix G. We evaluate our model in the context of zero-shot audio captioning, comparing it against other models in the field. All the training details and hyper parameters are given in Appendix E.

5 Results and discussion

Audio Captioning performance Table 5 presents the audio captioning performance of our methods on the AudioCaps test set, using 16 image captions to train the prefix tokens. We show the relatively small importance of the number of pairs in the audio captioning learning process in Appendix A. Our study compares the alignment variants considered in our zero-shot audio captioning framework against the current state-of-the-art methods, which employ audio encoders trained in either a supervised or an unsupervised manner. Our results indicate that the alignment through contrastive learning without audiovisual distillation does not fully match distributions, compared to DALI. Indeed, even without audiovisual distillation, DALIMMD and DALI Att OT significantly outperform Shaharabany et al. (32), this system being our closest competitor, as it relies on a backbone trained in an unsupervised fashion.

Without audiovisual distillation, DALIMMD performs comparably to DALI Att OT , however, the latter improves with audiovisual distillation. We hypothesize that this occurs because DALIMMD is trained to learn the complete image token distribution, while DALI Att OT only learns a part of it (due to the attentive optimal transport mechanism). Therefore DALI Att OT features can take more advantage of the combination with image tokens. We verify qualitatively this hypothesis in the Discussion paragraph below. Interestingly, both DALIOT and the contrastive backbone benefit from the audiovisual distillation. Given the relatively low-performance scores prior to the distillation process, it is plausible that they primarily learn from the image captioner (as the audiovisual pseudo captions tend to rely solely on visual features due to the inadequate audio representations) which is shown to be already performing well for audio captioning (first line of results). We hypothesize that the second stage is effective even for poorly aligned backbones because the distillation occurs on a subset of AudioSet where the images align well with the audio. If this distillation were applied to the entire dataset, the performance of these backbones might decline substantially. These hypotheses will require further investigation in future work. Lastly, it is important to note that feeding the image tokens in addition to the DALI Att OT ones, increases even more the performances, leaving the door open for further improvements.

Discussion Unlike standard contrastive methods, we do not average the output of the backbone, which allows us to employ token-level distribution alignment methods that do not rely on negative samples. Both Optimal Transport (OT) and Maximum Mean Discrepancy (MMD) focus on directly aligning the embedding's distributions.OT minimizes the cost of transporting one distribution to another, aligning them without the need for negative samples, and avoiding the issue of pushing different modalities into separate subspaces. Similarly, MMD measures the distance between kernel-projected mean embeddings, aligning the distributions without negative sampling. These methods promote a more cohesive embedding space, closely aligning different modalities and thus avoiding the modality gap inherent in contrastive learning approaches.

Indeed, Figure 4 shows a Principal Component Analysis (PCA) of the global representation (the average of the output tokens) of audio and im-

PCAofthemultimodalembeddingspace

15

CLIPImage

DALIOr

DALIMMD

2

10

Contrastive

Dimension

DALIoT

5

0

-10

-15

-10

-5

0

5

10

15

Dimension1

ages from various backbones on a subset of the AudioCaps test set. In our audiovisual experiment, we observe a phenomenon akin to the one demonstrated by Liang et al.(15) for image-text contrastive learning. Specifically, models encode each modality in a narrow cone, resulting in the so-called modality gap where the embeddings of each encoder are confined to a subregion of the embedding space. However, our backbones, which are trained using distribution alignment, do not exhibit this issue. The embeddings generated by both the DALIMMD and DALIOT encoders closely resemble those of the image modality. In contrast, the DALI Att OT ones, while located in the same region of the embedding space, remain distinct from the image tokens and exhibit minimal overlap. This indicates DALI Att OT and image tokens are complementary and provides an explanation for why the former benefits more from audiovisual distillation than the DALIMMD model.

The root cause of this discrepancy lies in the training process: the cross-attention-based DALI Att OT model is trained using only to the matching audiovisual tokens, whereas DALIMMD model is trained on the complete image distribution, which inherently includes a degree of image bias.

To verify this claim, we examined whether the aligned backbone with DALIMMD (after Stage 1) exhibits image bias. We prompted the captioner with "What can you see? Answer concisely" along with the audio tokens without the tuned prefix to avoid constraining it to generate only audio-related captions. Table 2 presents multiple captions generated using the DALI Att OT and DALIMMD backbones. As shown, the encoder trained with MMD often produces captions with notable visual biases, which are absent in the outputs of DALI Att OT . These results confirm our hypothesis and align with the visualizations in Figure 4. Note that the captions may seem incomplete as we ask the model to answer concisely, focusing only on the main element of the scene . Complete generated captions by different models are provided in Appendix D.

Assessing generalization ability The Clotho dataset presents a greater challenge than AudioCaps, primarily due to the complexity of the captions, which is reflected in the relatively lower performance of all the methods. The audio tracks in AudioCaps are sourced from AudioSet, thus the distribution of audio events resembles that of the data on which our models are trained. In contrast, Clotho's audios

are sourced from FreeSound, a dataset distinct from AudioSet. Consequently, our model, having been trained solely on AudioSet's audio excerpts and images, may encounter unfamiliar concepts in Clotho, thereby testing its capacity for generalization. Table 6 presents the models' performance on Clotho's test set. The observed trends are consistent with those on AudioCaps where DALIMMD does not benefit from audiovisual distillation, while DALI Att OT does. However, the absolute results differ, as DALIMMD marginally outperforms DALI Att OT even after the second training stage. We posit that the image bias inherent in DALIMMD becomes beneficial when confronted with out-of-distribution data. Indeed what may be regarded as out-of-distribution for the audio backbone might have been seen by the image one (due to its bigger training set). Consequently, the learning of the image backbone's bias could potentially be beneficial. Notably, our method outperforms Saleswki et al., despite the fact that the latter employs a backbone trained using millions of audio-text pairs (CLAP). It is important to note that our method differs from Saleswki et al. in that we perform unsupervised instead of supervised zero-shot audio captioning.

6 Conclusion

Wepresented a novel methodology for unsupervised zero-shot audio captioning, effectively leveraging advanced image captioning models, such as Llava. We adapted it to the audio captioning task by making use of prefix tuning and using innovative token distribution alignment techniques (using MMDor optimal transport with a cross-attention mechanism) that successfully bridged the modality gap between audio and visual inputs. Our comprehensive evaluation, both quantitative and qualitative, demonstrates that our method achieves state-of-the-art results in unsupervised zero-shot audio captioning. Remarkably, our approach, not using any audio recordings, not only matches but even sometimes, exceeds the performance of existing methods that rely on a backbone trained with extensive audio-text data such as CLAP. Moreover, our method requires only raw videos without any audio annotations, significantly enhancing its potential scalability. This positions our approach as a promising direction as a leading direction for the future of audio captioning.

Limitations Although capable of addressing partial audiovisual mismatches, our method remains insufficiently robust to complete cross-modal mismatches, such as instances where the image is entirely unrelated to the audio. We reserve addressing this limitation for future work. Moreover, to achieve even higher performance in audio captioning, our method would benefit from a degree of supervision. The requirement for an external supervisory source ( e.g. , audio-text pairs) arises due to the method's dependence on audiovisual alignments, which makes it challenging to learn certain sounds because they cannot be seen. For example, it is almost impossible to generate a caption for the sound of the wind by merely using images.

Acknowledgement This work was supported by the Audible project, funded by French BPI. This work was performed using AI resources from GENCI-IDRIS (Grant 2023-AD011014885).

References

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee, 'Visual instruction tuning,' Advances in neural information processing systems , vol. 36, 2024.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al., 'Flamingo: a visual language model for few-shot learning,' Advances in neural information processing systems , vol. 35, pp. 23716-23736, 2022.

Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al., 'Obelics: An open web-scale filtered dataset of interleaved image-text documents,' Advances in Neural Information Processing Systems , vol. 36, 2024.

Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang, 'Tinyllava: A framework of small-scale large multimodal models,' arXiv preprint arXiv:2402.14289 , 2024.

Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang, 'Llava-phi: Efficient multi-modal assistant with small language model,' arXiv preprint arXiv:2401.02330 , 2024.

Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang, 'Clap learning audio concepts from natural language supervision,' in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2023, pp. 1-5.

Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov, 'Large-scale contrastive language-audio pretraining with feature fusion and keywordto-caption augmentation,' in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2023, pp. 1-5.

Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang, 'Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research,' arXiv preprint arXiv:2303.17395 , 2023.

Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou, 'Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,' arXiv preprint arXiv:2311.07919 , 2023.

Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro, 'Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities,' arXiv preprint arXiv:2402.01831 , 2024.

Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang, 'Pengi: An audio language model for audio tasks,' Advances in Neural Information Processing Systems , vol. 36, pp. 18090-18108, 2023.

Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter, 'Audio set: An ontology and human-labeled dataset for audio events,' in 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP) . IEEE, 2017, pp. 776-780.

Arnab Barua, Mobyen Uddin Ahmed, and Shahina Begum, 'A systematic literature review on multimodal machine learning: Applications, challenges, gaps and future directions,' IEEE Access , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al., 'Learning transferable visual models from natural language supervision,' in International conference on machine learning . PMLR, 2021, pp. 8748-8763.

Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou, 'Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning,' Advances in Neural Information Processing Systems , vol. 35, pp. 17612-17625, 2022.

Peiyang Shi, Michael C Welle, Mårten Björkman, and Danica Kragic, 'Towards understanding the modality gap in clip,' in ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls , 2023.

Sijie Mai, Ying Zeng, and Haifeng Hu, 'Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations,' IEEE Transactions on Multimedia , 2022.

Shraman Pramanick, Aniket Roy, and Vishal M Patel, 'Multimodal learning using optimal transport for sarcasm and humor detection,' in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2022, pp. 3930-3940.

Zonghai Yao, Ahmed Jaafar, Beining Wang, Zhichao Yang, and Hong Yu, 'Do physicians know how to prompt? the need for automatic prompt optimization help in clinical note generation,' 2024.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu, 'Learning to prompt for vision-language models,' International Journal of Computer Vision , vol. 130, no. 9, pp. 2337-2348, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu, 'Conditional prompt learning for vision-language models,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 16816-16825.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, 'Bleu: a method for automatic evaluation of machine translation,' in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311-318.

Chin-Yew Lin and FJ Och, 'Looking for a few good metrics: Rouge and its evaluation,' in Ntcir workshop , 2004.

Satanjeev Banerjee and Alon Lavie, 'Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,' in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , 2005, pp. 65-72.

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh, 'Cider: Consensus-based image description evaluation,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 4566-4575.

Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould, 'Spice: Semantic propositional image caption evaluation,' in Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14 . Springer, 2016, pp. 382-398.

Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy, 'Improved image captioning via policy gradient optimization of spider,' in Proceedings of the IEEE international conference on computer vision , 2017, pp. 873-881.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al., 'An image is worth 16x16 words: Transformers for image recognition at scale,' in International Conference on Learning Representations , 2020.

Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James R Glass, 'Listen, think, and understand,' in The Twelfth International Conference on Learning Representations , 2023.

Leonard Salewski, Stefan Fauth, A Koepke, and Zeynep Akata, 'Zero-shot audio captioning with audio-language model guidance and audio context keywords,' arXiv preprint arXiv:2311.08396 , 2023.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al., 'Opt: Open pre-trained transformer language models,' arXiv preprint arXiv:2205.01068 , 2022.

Leonard Salewski, Stefan Fauth, A Koepke, and Zeynep Akata, 'Zero-shot audio captioning with audio-language model guidance and audio context keywords,' arXiv preprint arXiv:2311.08396 , 2023.

Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra, 'Imagebind: One embedding space to bind them all,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 15180-15190.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, Jeffrey Dean, and Sanjay Ghemawat, 'Language models are unsupervised multitask learners,' in OSDI'04: Sixth Symposium on Operating System Design and Implementation , pp. 137-150.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals, 'Representation learning with contrastive predictive coding,' arXiv preprint arXiv:1807.03748 , 2018.

Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R Glass, 'Contrastive audio-visual masked autoencoder,' in The Eleventh International Conference on Learning Representations , 2022.

Mariana-Iuliana Georgescu, Eduardo Fonseca, Radu Tudor Ionescu, Mario Lucic, Cordelia Schmid, and Anurag Arnab, 'Audiovisual masked autoencoders,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 16144-16154.

Po-Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Yanghao Li, Shang-Wen Li, Gargi Ghosh, Jitendra Malik, Christoph Feichtenhofer, et al., 'Mavil: Masked audio-video learners,' Advances in Neural Information Processing Systems , vol. 36, 2024.

Xiang Lisa Li and Percy Liang, 'Prefix-tuning: Optimizing continuous prompts for generation,' in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , 2021, pp. 4582-4597.

Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim, 'Audiocaps: Generating captions for audios in the wild,' in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , 2019, pp. 119-132.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer, 'Sigmoid loss for language image pre-training,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 11975-11986.

SiYa Yao, Qi Kang, MengChu Zhou, Muhyaddin J Rawa, and Aiiad Albeshri, 'Discriminative manifold distribution alignment for domain adaptation,' IEEE Transactions on Systems, Man, and Cybernetics: Systems , vol. 53, no. 2, pp. 1183-1197, 2022.

Jindong Wang, Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, and Philip S Yu, 'Visual domain adaptation with manifold embedded distribution alignment,' in Proceedings of the 26th ACM international conference on Multimedia , 2018, pp. 402-410.

Bernhard Schölkopf, 'The kernel trick for distances,' Advances in neural information processing systems , vol. 13, 2000.

Gabriel Peyré, Marco Cuturi, et al., 'Computational optimal transport: With applications to data science,' Foundations and Trends® in Machine Learning , vol. 11, no. 5-6, pp. 355-607, 2019.

Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty, 'Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation,' in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 447-463.

Martin Arjovsky, Soumith Chintala, and Léon Bottou, 'Wasserstein generative adversarial networks,' in International conference on machine learning . PMLR, 2017, pp. 214-223.

Pan Li, Lei Zhao, Duanqing Xu, and Dongming Lu, 'Optimal transport of deep feature for image style transfer,' in Proceedings of the 2019 4th International Conference on Multimedia Systems and Signal Processing , 2019, pp. 167-171.

Thibault Séjourné, Gabriel Peyré, and François-Xavier Vialard, 'Unbalanced optimal transport, from theory to numerics,' Handbook of Numerical Analysis , vol. 24, pp. 407-471, 2023.

Kilian Fatras, Thibault Séjourné, Rémi Flamary, and Nicolas Courty, 'Unbalanced minibatch optimal transport; applications to domain adaptation,' in International Conference on Machine Learning . PMLR, 2021, pp. 3186-3197.

Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun, 'Attention bottlenecks for multimodal fusion,' Advances in neural information processing systems , vol. 34, pp. 14200-14213, 2021.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi, 'Blip-2: Bootstrapping languageimage pre-training with frozen image encoders and large language models,' in International conference on machine learning . PMLR, 2023, pp. 19730-19742.

Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen, 'Clotho: An audio captioning dataset,' in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 736-740.

Yujia Li, Kevin Swersky, and Rich Zemel, 'Generative moment matching networks,' in International conference on machine learning . PMLR, 2015, pp. 1718-1727.

Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer, 'Pot: Python optimal transport,' Journal of Machine Learning Research , vol. 22, no. 78, pp. 1-8, 2021.

The appendix is organized as follows: The first part presents additional experiments that aid in understanding the behavior of our encoders, followed by a detailed explanation of the pre-filtering process performed on AudioSet and the training parameter details.

A Impact of the number of image captions

To determine whether the tuned prefixes only specify the task and help avoid visual artifacts in the captions, we analyzed the performance of the method by varying the number of image- audio caption pairs. Figure 5 illustrates that only 16 captions are required to achieve good performance, and increasing this number does not lead to further improvement. In fact, while the SPIDER, CIDEr, and ROUGE metrics remain stable, the SPICE, METEOR, and BLEU scores even show a slight decline. This suggests that prefix tokens only serve to specify the task to the language model and do not contribute to learning the captioning task itself. We hypothesize that this prefix tuning could potentially be replaced by a carefully chosen hand-crafted prompt.

Cider_d Scores

Rouge_I Scores

0.16

0.25

0.14 -

0.30

0.20

0.28

Spider

0.12

Cider_d

0.10

0.15

0.24

0.08

DAL

DALIMMD

DAL器

DALIMMD

0.22

DAL

DALIMMD

0.06

CLIP Image

0.10

CLIP Image

0.20

CLIP Image

8

16

32

8

16

32

8

16

32

Shots

Shots

Shots

Spice Scores

Bleu_4Scores

0.0800

DAL

0.135

0.08

0.0775

CLIP Image

DALMD

0.130

0.06

Spice

0.0750

Meteor

0.125

Bleu_4

0.0725

0.04

0.0700

0.120

DAL

DALMD

0.0675

0.115

CLIP Image

0.02

CLIP Image

8

16

32

8

16

32

8

16

32

Shots

Shots

Shots

B Deeper analysis of the learned distributions

Figure 6 presents a 2-D projection through PCA of CLIP image tokens along with the audio tokens learned by DALI Att OT , DALIMMD, and contrastive learning for the audiovisual pair shown in the figure ('A sound of a guitar playing'). The contrastive learning method maps the audio tokens to a very restricted subset of the space, without any overlapping with the image distribution. Interestingly, while DALIMMD only learns to align the expected value of the distribution, it appears to fit the full image distribution quite well. DALI Att OT exhibits similar behavior but is less noisy: the tokens are either very close to the image tokens or much further away. This behavior is inherited from the attentive optimal transport, which assigns low weights to points considered as mismatches.

10

PCAof thecontrastiveembedding space

25

CLIP Image

20

contrastive model

Dimension

2

15

10

5

10

50

40

20

10

0

10

20

Dimension1

PCAof theDALiatembeddingspace

PCAof theDALImMpembedding space

25

CLIP Image

25

CLIP Image

20

DAL

20

DALIMMD

Dimension 2

15

Dimension

2

15

10

0

5

-5

-10

-50

40

30

Dimension 1

20

10

0

10

20

10+

50

40

30

Dimension 1

20

10

0

10

20

Figure 7 illustrates the optimal transport weights computed by the cross-attention mechanism for the given image and its associated audio (an emergency vehicle siren). The size of the circles represents the weights. Although the overall distribution appears uniform, a closer inspection of the bottom right section reveals that many tokens have scores close to zero. These tokens likely represent the road, which does not produce any sound and thus has no corresponding match in the audio tokens. Consequently, these tokens are not considered in the calculation of the transportation coupling

PCA of token embeddings

PCA of token embeddings

'An emergency vehicle

20

image

-3.0

小小

15

audio

3.5

10

-4.0

N

Dimension

Dimension

2

4.5

5.0

0

5.5

-6.0

image

oipne

10

30

-20

-10

0

10

20

30

6.5

15

16

17

18

19

20

21

22

Dimension 1

Dimension 1

C AudioSet pre-filtering to limit audiovisual discrepancies

As AudioSet primarily focuses on audio rather than audiovisual content, it features audio events that may be unrelated to the visual event depicted. For example, many videos feature music as an audio event while displaying a static image of the album cover.

To address this issue, we filtered out audiovisual discrepancies. To achieve this, we generated captions for the first frame of each second of a video using BLIP2, which we then encoded into GPT2-embedding space. Subsequently, we computed the distance between the embedding of each frame's caption and the embedding of the corresponding AudioSet labels. This process is illustrated in Figure 8. The average distance across the video frames was used as the distance between the video and the audio. We retained the 500,000 pairs with the smallest distance.

Q:What sounds can produce

the objects in the image?

Image

Image

BLIP-2

captions

Encoder

Frames

avg(

Distance

AudioSet

Image

labels

Encoder

D Caption examples

Table 4 shows multiple examples of captions generated by the model using the contrastive backbone and DALI Att OT , both after the first stage of training. While the captions from the latter are accurate, using the contrastive backbone generates quite noisy/bad captions. This outlines the fact that contrastive learning is not well suited to swap the encoder of a modality with the one from another.

E Training details

For the MMD alignment, following (54) we use a mixture of K kernels spanning multiple ranges (being the average pairwise squared distance between samples multiplied by k with k = { 0 , ..., K } ), with K = 5 . As far as optimal transport is concerned, the first epoch of the first stage is performed with uniform transport weights. A decreasing value of λ is then used: from 500 to 100 at the end of the 5th epoch. λ = 100 is then used for the remaining epochs. We observed that the training is quite robust to small variations of λ , as long as the first epoch is performed with uniform weights. We used the Python Optimal Transport (POT)(55) library to compute efficiently the EMD. In the prefix tuning stage, we learned 16 prefix tokens on different numbers of image and audio caption pairs. We used cosine decay and a learning rate of 2e-4 for both the prefix tuning and the MLP tuning in the second stage. The prompt used for all the experiment is: 'What sound can be heard in this scene ?'. We used version 1.5 of Llava: the image encoder being the ViT-L@336px version of CLIP (14), and the mapping network between image tokens and the LLM embedding space a 2 layer-perceptron. As for the audio encoder, we start from a pre-trained CAV-MAE (36) model in order to speed-up convergence. This model consists of 12 transformer encoder layers of dimension 768 and one final linear layer that maps the output of the transformers to a 1024-dimensional space (dimension of the CLIP encoder). Note that this encoder is way smaller than the image one (88M vs 428M parameters),

because the training set used is 2 orders of magnitude smaller than the CLIP one.

All the trainings were performed using 4 Nvidia A100 gpus, and all the steps of the training can be performed within 30 hours of compute on those machines.

F Attentive Optimal Transport details

For DALI Att OT , we computed the weights of the optimal transport by averaging on the feature dimension the tokens resulting from the cross-attention and then feeding them through a softmax with temperature (we used τ = 20 ).

More formally, the weights of the transports α Att and β Att are computed as follow:

with A and I being the audio and image tokens, W Q i , W i K , W V i ; W Q a , W K a , W V a the image and audio projections, respectively, and τ a hyper-parameter.

G Additional captioning metrics

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We justify by quantitative and qualitative analysis the claims made in the abstract. This can be found in the Results and discussion section as well as in appendices

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We dedicated a paragraph to this ('Limitations') as part of the Conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: NA

Justification: NA

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the hyper parameters used for the training are given in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Data used are already publicly available, both the model and the training code will be made available on Github along with a tutorial on how to use it. The preparation of this repo is in progress.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the hyperparameters are given in the Appendix, the prefiltering process used on the training dataset as well, and the list of files used for training will be made available on the Github repo.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Experiments were too costly to be run multiple times, however, a lot of ablations are available and the model is tested on multiple datasets.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The machine on which we run the experiments, as well as the computing time needed is mentioned in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research does not involve human subjects or participants. We systematically referred to the licences of the licenced datasets that we used (namely CC BY 4.0). We are not aware of any potential misuse of our algorithm, but there may be some.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: We are not competent for such a discussion.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not believe that our model can present any risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite both the paper and the licence of each model and dataset we used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We did not introduced any assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We did not use crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our research did not involve experiments on humans subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to use a well-trained visual LLM (specifically Llava-v1.5) to perform audio captioning in a zero-shot fashion. The authors propose a training framework which comprises 2 stages and 5 sub-steps. The authors also propose to use Maximum Mean Discrepancy (MMD) and optimal transport (OT) as the loss function to align audio and visual representation space.\n\nSTRENGTHS:\n1. The two loss functions are novel directions to explore for alignments between different modalities\n2. This paper explores audio-visual LLM application scenarios which is timely.\n\nWEAKNESSES:\n1. The motivation is not clear. The author must specify in which scenario zero-shot audio captioning is needed, with concrete examples that the proposed method will help in practice, such as generating a particular captioning style or targeting a specific type of audio. As far as I can see in the paper, the authors only did experiments on two standard tasks (AudioCaps and Clotho) and both tasks have a reasonably-labelled size, and the performance is far from the state-of-the-art.\n\n2. The method lacks theoretical grounding or insights. The author says they randomly choose one frame from the 10 frames in a video as the image and train to pull the image representation space to be close to the audio space. The author did not analyse how related the image is to the audio. How often does the image reflects what can be heard, and how often it does not? \n\n3. The experimental setup is questionable. \n  - The author says clearly in line 167 that they performed p-tuning with audio captioning data \"We trained the additional tokens using image-text pairs, with the text describing the sounds associated with the images\". This clearly sets this method apart from \"zero-shot learning\" which refers to having no data at all. If the author means \"zero-shot\" for a specific audio caption dataset, they need to be clear about it. \n  - Regarding experimental results: I wonder if the comparison between the proposed method and ImageBind is fair. They also did not say how much data they used for ImageBind p-tuning. If the authors just took the numbers from Shaharabany et al., then putting the numbers in the same table is clearly unfair. I might have missed something here but without a convincing explanation, I do not think the experiments are valid. Why is ImageBind not included in Table 2?\n  - I would be interested to see whether adding audio to the audio-visual description would give better performance or not. It seems that the audio information is all included in the visual information. Getting back to Weakness 1 - what is the point of doing this task then? What extra did you gain?\n\nA few writing problems:\n1. The referencing style makes it difficult to locate various papers. \n2. Fig. 2 captions are incomplete.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThere is a limitations section but no discussion on potential negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method to align tokens created by an audio encoder with those from an image encoder to allow zero-shot description of audio and audio-visual recordings. It introduces an attention-based matching to the alignment to be able to account for objects that appear in one modality but not the other (e.g., background music or the visual background). The paper compares two different alignment methods: maximum mean discrepancy and optimal transport, with the attention only applied in the case of optimal transport. And prefix tuning is used to guide the image captioner towards audio captions.\n\nThe model is trained on AudioSet and evaluated on AudioCaps and Clotho using standard captioning metrics. While several metrics are reported, the primary one used by the community is SPIDEr, and by this metric on AudioCaps (in-domain to training) the proposed approach when using audio-visual input (0.1946) outperforms a comparison supervised system (0.1830), an image-only system (0.1499), and other variants of the proposed system such as the audio-only version (0.1592). On the out of domain and audio-only Clotho dataset the supervised baseline performs best (0.097) followed by the MMD variant of the proposed system (0.0655). Note that state of the art performance on Clotho was around 0.3 for systems like QwenAudio and AudioFlamingo and around 0.5 on AudioCaps.\n\nSTRENGTHS:\nSignificance: \n* Cross-modal learning is an important problem, although it is not clear that zero-shot learning is necessary as there is a great deal of audio-visual content available, but if it is possible, then it seems worth exploiting.\n* The ability to leverage visual captioning systems to bootstrap audio captioning systems is valuable, as the former have received much more attention than the latter.\n* The issue of audio-visual correspondence is key in audio-visual processing and the proposed approach is an interesting way to deal with it.\n\nClarity:\n* The paper is generally well written and easy to follow\n* The diagrams are quite helpful in understanding the approach\n* The paper includes a real discussion of its limitations\n* The caption examples included in the appendices are quite informative. It would be nice if there were examples from all of the systems to compare\n\nTechnical correctness:\n* The experiments are well organized and well conducted\n\nWEAKNESSES:\nClarity:\n* The use of many metrics dilutes the clarity of the results and their interpretation. The metrics should be prioritized and the systems they are comparing discussed accordingly. The discussion of results seems to treat all metrics as somewhat interchangeable and makes broad statements about which systems seem better than others. It would be much clearer and easier to follow if differences in performance on specific metrics were tied to qualitative differences in model behaviors and when there is one main metric (i.e., SPIDEr) to focus on that for overall system comparison.\n* Line 96 states that \"Audio data often lacks the richness and context provided through the visual modality, resulting in less precise and more subjective descriptions of the audio content.\" While I agree that audio captioning datasets typically contain more subjective descriptions of their content, I think this is an issue with the data that is selected to be captioned and the question asked of the captioner as opposed to a shortcoming of audio in general. With the right data and right task, audio captioning could be much better defined. I believe this sentence should be modified to focus on existing datasets as opposed to the general modality.\n* No mention is made of the size of the various models or model pieces involved or how much data they were trained on. This is especially acute for comparing the proposed system to existing baselines from Shaharabany et al and Salewski et al.\n* I can't understand what the ablations are showing, these should be described in the body of the paper, not just shown in tables.\n\nPerformance:\n* Overall, the reported SPIDEr results (0.1-0.2) are fairly low compared to SOTA approaches for Clotho (0.3) and AudioCaps (0.5). It's clear that this zero-shot approach does not completely solve the problem. It would be useful to discuss this in the paper.\n* Additionally, the differences in performance between the proposed variants are not that large, bringing into question the value of including multiple-such approaches.\n\nMinor issues:\n* Figure 2: sub-figures (d) and (e) are not mentioned in the caption\n* Equation (4): please use \\langle and \\rangle as opposed to < and > for brackets\n* Tables 1 and 2: Order the rows within each block intentionally. I would recommend ordering by SPIDEr.\n* Figure 4: It's hard to distinguish between this many colors with this level of transparency. Maybe you could use different symbols instead of different colors.\n\nQUESTIONS:\nWhat exactly was done in the ablation studies listed in Tables 1 and 2? What do the rows mean? What was taken away?\n\nWhy wasn't the attention weighting used with MMD in addition to OT?\n\nDoes the audio-visual alignment treat each instance as a \"bag of tokens\"? If so, would there be value in considering the sequence of tokens in this alignment as opposed to just their distribution?\n\nWere any categories held out of training so that they could be evaluated in zero-shot category generalization? Or was the focus only on zero-shot modality generalization?\n\nWas the filtering of audiovisual discrepancies described in appendix C applied to both training and test data or just to training data?\n\nLIMITATIONS:\nThe paper has a section focusing on limitations and does an excellent job discussing them, providing true limitations of the approach.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for how to adapt VLM to perform audio captioning. The pipeline is as follows: 1. perform few-shot prefix tuning on images to caption the audio. 2. Use multi-modal alignment methods, namely MMD or OT, to align token space distribution of audio and visual. 3. Distillation for audio-only model from the visually-informed audio caption model. Experiments show that the proposed method outperforms contrastive-based alignment and also the pretrained CLAP model that is used for audio captioning.\n\nSTRENGTHS:\n1. The main novelty of the paper lies in its usage of MMD / OT to perform multi-modal alignments in audio-visual space, though MMD / OT is not new and has been extensively explored in cross-domain matching literature.\n\n2. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. There is lack of theoretical derivation or enough insights for showing why the OT and MMD can outperform the standard contrastive learning. Figure 4 shows the PCA plots for different methods, however it is unclear why MMD or OT is able to stay closer to CLIP encoder. I think more explanations or theoretical justifications are needed.\n\n2. The experiment results seem mixed, in particular Table 2 for Clotho. Contrastive method is better in several metrics without audio-visual distillation, in particular CIDEr and SPIDEr. Also the SPICE performance is very close to the proposed method. These two metrics, CIDEr and SPIDEr from my point of view is also very important since they are designed specifically for caption task. Why would Contrastive outperform the proposed methods in this dataset? But in AudioCaps, the proposed methods beat the contrastive method by a lot. The inconsistency results across these two datasets make me feel confused.\n\nQUESTIONS:\nSee Weaknesees.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper under review looks at the problem of captioning of short audio-video clips, leveraging existing multimodal large-language-models. While many strong image captioners exist due to the abundance of image/caption pairs for training data, for clips of non-speech audio, the amount of data available are comparably much less. To build a performant captioner that can produce accurate captions for clips contained either of both audio and visual modalities, the paper proposes a framework that leverages LVLMs, adapts them to produce more audio-centric descriptions via prefix tuning, and most critically \"aligns\" the distributions of the audio and visual tokens to match making the tokenized output from the visual or audio encoder interchangeable. The main novelty is the use of Optimal Transport to align the visual and audio modalities and modifying the loss with cross attention terms to improve the alignment. While this DALI_OT^Att is not a large improvement over DALI_OT or DALI_MMD, the experimentation of these modality alignment methods and demonstrated improvement over a constrastive loss is a novel contribution. The analysis and plot of the multimodal embedding space of the various approach to align the A-V modalities further demonstrates the effectiveness. Overall the paper does demonstrate how LVLMs can be utilized to build a strong A-V captioning system.\n\nSTRENGTHS:\nGoal of the paper, contributions, execution are all clear. There are a good set of experiments, visualizations of the expected improvement in alignment of the visual and audio token distributions, and overall the application of MMD or OT learning improves the alignment as demonstrated by the experimental results over the dominant baseline approach of contrastive learning.\n\nWEAKNESSES:\nThe paper has framed the problem of A/V captioning in a somewhat narrow way of leveraging LLVMs with the only comparison to a strong supervised baseline in CLAP while there are others such as GPTo, Gemini 2 or Llama 3.\n\nSection 3.1 is rather confusing. It seems Figure 2 was revamped and the references to the figure in section 3.1 was not completely updated. Figure 2 should also include a description of steps 2d and 2e in the descriptive caption (the description of 1c needs updating too I believe. The reference on line 177 for 2-c should probably be 2-b; line 190 2-c should be 2-d; and on line 196 2-d changed to 2-e. Overall, it would be clearer to first indicate the stage and then explain it, e.g. \"1-a) prefix tuning the prefix tokens\" rather than the other way around. Also each of the bolded titles in section 3.1 should correspond with a stage in Figure 2 and include the stage number for better clarity.\n\nIn section 3.2, it would be helpful to the reader to provide concrete equations on how to compute the quantities and eqns 1-5. Having these in an appendix would be fine. While its understood that the code will eventually be released, having the equations that the code is intended to implement is always useful for clarity.\n\nQUESTIONS:\nHas there been any analysis done in comparison to other large, strong multimodal LLMs such as  GPT-4o, Gemini 2 or Llama 3?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-31"}