Advancing Multi-grained Alignment for Contrastive Language-Audio Pre-training

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 16:36:08

Abstract: Recent advances have been witnessed in audio-language joint learning, such as CLAP, that shows much success in multi-modal understanding tasks. These models usually aggregate uni-modal local representations, namely frame or word features, into global ones, on which the contrastive loss is employed to reach coarse-grained cross-modal alignment. However, frame-level correspondence with texts may be ignored by the above paradigm, making it ill-posed on explainability and fine-grained text-audio challenges (e.g., text-to-audio grounding) which may also undermine performances on coarse-grained tasks. In this work, we aim to improve both coarse- and fine-grained audio-language alignment in large-scale contrastive pre-training. To unify the granularity and latent distribution of two modalities, a shared codebook is adopted to represent multi-modal global features with common bases, and each internal codeword is regularized to encode modality-shared semantics, bridging the gap between frame and word features. Based on the above framework, a locality-aware block is involved to purify local patterns, and a hard-negative guided loss is devised to boost alignment effects. Extensive experiments on eleven zero-shot coarse- and fine-grained evaluation protocols suggest that our model not only surpasses the baseline CLAP significantly but also yields superior or competitive results compared to current SOTA works. The code and model will be released upon paper acceptance.

Main paper: 
 introduction: 
 Sound conveys a lot of information in our daily lives. With the advance of learning theories and data collections [11] , large-scale pre-trained models, such as PANNs [21] and AST [13] , have witnessed extraordinary achievements on sound-related challenges, such as sound classification [38] and sound event detection [26] . Despite such success, these methods still require downstream tuning to adapt to novel scenarios and cannot facilitate tasks related to natural language, e.g., retrieve or generate audio clips [28, 51, 52] to human instructions. Alternatively, Contrastive Language-Audio Pre-training (CLAP) [9] is introduced to learn general and transferable representations by associating audio samples with corresponding captions. Consequently, an aligned feature space is built, making it versatile for several tasks, such as zero-shot audio classification and retrieval, by simply computing the cosine similarity between encoded audio and textual features of sound classes [25] .However, during empirical practices, we notice that current CLAP models lack the capability of capturing the fine-grained alignment like the relationship between acoustic events and textual meanings. An example of this phenomenon is depicted in Figure 1 (a). As seen, although the two kinds of events, namely alarm and speech, are successfully recognized by the original CLAP, the similarity between frame representations and textual sound representations shows much inconsistency with the real temporal locations of sound events. For instance, the sound "alarm" is recorded at 2.8s-4.5s and 5.4s-7.0s, but the corresponding frame-level similarity is high over the whole clip. This may undermine the model's explainability and lead to undesirable results on fine-grained cross-modal understanding tasks, including zero-shot sound event detection and text-to-audio grounding [53] . Moreover, poor performance can also be observed in certain cases when conducting coarse-grained tasks like zero-shot audio tagging and retrieval, since local patterns and temporal information are potentially ignored by the vanilla CLAP paradigm. We attribute the above problem to the lack of interaction between frame and word features during CLAP training, as current CLAP methods reach the cross-modal alignment via solely the similarity of the global features of each modality.To mitigate the research gap, we propose to adopt a modalityshared codebook to encourage the multi-modal features to interact on a finer granularity. The codebook consists of several learnable codewords, and the weighted summation of them will be utilized to represent the global features of each modality, so that they are naturally restricted in the same feature space, making it easier to learn the alignment. To encode cross-modal shared semantic concepts (e.g., sound events) into each codeword, we further revise the traditional working scheme of the codebook to compute the aggregation weights. Practically, we define the affinity scores between a clip (or a caption) and each codeword as the maximum cosine similarity between its frame features (or word features) and the specific codeword. Then, the global feature can be represented with a small number of codewords by applying sparse constraints on the affinity scores to avoid noisy activation before using them as aggregation weights. Through optimizing the contrastive loss, not only the paired global features can be well-aligned, but also the frame features of an acoustic event (e.g., "alarm") and the word features of the corresponding caption can activate the same, small set of codewords, thereby implicitly building a connection between fine level multi-modal features. Moreover, we also notice that local acoustic patterns may be destroyed by the vanilla transformer block and devise a novel locality-aware block to ensure high-quality frame features for codewords aggregation. Finally, a hard-negative guided contrastive loss is reformulated to mine more discriminative representations in order to build a better-aligned global latent space. Equipped with these techniques, our MGA-CLAP reaches a better fine-grained alignment than the original CLAP without losing its natural coarse-grained alignment, as shown in Figure 1 (b) .We conduct extensive experiments on both coarse-and finegrained audio-text tasks. As for the fine-grained ones, MGA-CLAP surpasses the original CLAP to a large extent. Specifically, using WavCaps [30] as the main pre-training dataset, MGA-CLAP achieves 26.4%/10.1% PSDS1 on zero-shot DESED [41] /AudioSet-Strong [15] sound event detection tasks, which is 13.3%/6.7% higher than its baseline CLAP. While for coarse-grained retrieval and tagging tasks, our method also demonstrates noticeable improvements over CLAP and shows better performance on most evaluation protocols compared to previous SOTA works which generally require much more training resources. Besides, several ablation studies are performed to reveal the effects of each component elaborately. Finally, we also visualize the semantic meanings of specific codewords to show their roles in linking different modalities. 
related work 2.1 contrastive language-audio pre-training: 
 By pre-training on 400M image-text pairs, CLIP [33] demonstrates superior performance and transferability on cross-modal vision problems, such as zero-shot image retrieval and classification. Several works, including AudioCLIP [14] and Wav2CLIP [47] , try to leverage visual modality as a bridge to connect text and audio representations, achieving promising results on zero-shot audio tagging tasks. With the collection of large-scale audio caption datasets, namely AudioCaps [19] , Clotho [7] and WavText5K [6] , a lot of works explore contrastive language-audio pre-training without involving the visual modality. MS-CLAP [9] first obtains aligned text and audio encoders on a combination of off-the-shelf audiotext datasets. However, due to the limits of data size, its performance is sub-optimal. A few researchers then turn to expand the scale of audio-text datasets. LAION-Audio-630K [48] and WavCaps [30] , collected and annotated by human professionals and Chat-GPT respectively, are shown to be more effective for pre-training. Besides, BLAT [55] proposes to utilize a well-trained model together with audio tags to automatically generate audio captions for contrastive pre-training while Cacophony [59] combines an audio caption model and Large Language Models (LLMs) to expand the data size to 4M and explore training strategies on such large-scale dataset. Moreover, the intrinsic shortcomings of CLAP are also studied. ACBA [46] and CompA [12] enhance CLAP's compositional reasoning ability while FLAP [57] devises masking strategies to improve both the training efficiency and model performance. By contrast, we notice the unsatisfactory fine-grained alignment of CLAP and aim to discover both fine-grained and coarse-grained correspondence solely from audio-text pairs.Codebook is the key design in vector quantization [42] , which is widely adopted for both understanding [1] and generation [36] tasks. During quantization, encoder features will be substituted by their nearest-neighbor codewords in the codebook, before being utilized to reconstruct original features by the decoder. Finally, by querying the learned codebook, a continuous space can then be transformed into finite discrete tokens. Following this way, modern neural audio codec models [49, 58] learn to convert the raw waveform into several codewords, paving the way for efficient audio compression [5] and auto-regressive audio generation [44] . Besides, BEATs [4] , the state-of-the-art self-supervised learning approach, also employs an acoustic tokenizer to quantize spectrograms into codewords for mask prediction, which demonstrates better performance compared to reconstruction methods, namely AudioMAE [17] . Different from the above, we leverage the codebook to accommodate both text and audio hidden representations instead of single-modality raw signals, which explicitly constructs a shared multi-modal feature space for coarse-grained alignment. Moreover, we dedicatedly redesign the computational rules of the codebook so that it can help discover the fine-grained correspondence.Frame-wise labeling is extremely laborious for audio tasks, hence learning from partially labeled data (e.g., weak labels or audio captions) becomes a promising remedy. Weakly supervised sound event detection [22, 27] aims to recognize the sound event boundary under weak supervision, where only the clip-level annotations are provided but the exact timestamps are inaccessible. However, it solely maps acoustic features to a closed label set, which limits its applications in open-world scenarios. By contrast, learning from audio captions addresses the aforementioned problem by associating frame features with general language descriptions. But it is more challenging due to the intrinsic modality gap. Besides, the noisy information (non-sound words) contained in the captions also increases the difficulty. UACA [50] first learns relationships between sound events and textual phrases from audio captions by aggregating frame-word similarity matrix to clip-caption similarity, while WSTAG [54] improves it by leveraging max-mean instead of mean-mean pooling. However, these works depend on exhaustive score matching while ignoring complex frame-word interaction, leading to suboptimal fine-grained alignment when scaling to a much larger pre-training dataset. In this work, we propose a novel solution to model the frame-word correspondence, demonstrating better performance and scalability than [50, 54] . 
methodology 3.1 overview: 
 An overview of our MGA-CLAP is shown in Figure 2 (a). As illustrated before, we introduce a novel modality-shared codebook, which aggregates frame-and word-level features with shared codewords. Then, in order to refine the frame-wise features, the localityaware block is involved to better capture local patterns. Finally, the CLAP loss is reformulated to emphasize indistinguishable audiotext pairs for contrastive optimization. In the following subsections, we will detail the above three core designs.CLAP employs a biencoder architecture to learn the aligned feature space for both modalities. Specifically, assume that we have a batch of audio-text pairs {(𝑥 𝑖 , 𝑦 𝑖 )} 𝐵 𝑖=1 , where 𝑥 𝑖 , 𝑦 𝑖 represent the 𝑖 th audio clip and its caption, and 𝐵 is the batch size. CLAP audio encoder 𝑓 takes 𝑥 𝑖 as input before generating frame representations 𝑃 𝑖 = 𝑓 (𝑥 𝑖 ) ∈ R 𝑇 ×𝐷 while the text encoder 𝑔 outputs word-level features 𝑄 𝑖 = 𝑔(𝑦 𝑖 ) ∈ R 𝑁 ×𝐷 according to 𝑦 𝑖 , where 𝑁 , 𝐹 and 𝐷 is the number of frames, words and feature dimensions, respectively. Then, to obtain the global clip-and caption-level feature, an aggregator ℎ (𝑎) : R 𝑇 ×𝐷 → R 𝐷 is required to map 𝑓 (𝑥 𝑖 ) to p𝑖 , and ℎ (𝑡 ) works similarly to aggregate 𝑔(𝑦 𝑖 ) to q𝑖 . Finally, the symmetric contrastive loss is optimized to pull together the global features of paired audios and texts while pushing away unpaired ones in the latent space,L CLAP = - 𝐵 ∑︁ 𝑖=1 𝑙𝑜𝑔 𝑒 < p𝑖 , q𝑖 >/𝜏 𝐵 𝑗=1 𝑒 < p𝑖 , q𝑗 >/𝜏 - 𝐵 ∑︁ 𝑖=1 𝑙𝑜𝑔 𝑒 < q𝑖 , p𝑖 >/𝜏 𝐵 𝑗=1 𝑒 < q𝑖 , p 𝑗 >/𝜏 (1)where < •, • > is the inner product function and 𝜏 is a scaling factor.In the above CLAP paradigm, ℎ (𝑎) and ℎ (𝑡 ) are instantiated by mean pooling or attention pooling, suggesting that the global features are essentially weighted sum of two bases: the audio frames and language tokens. However, due to the modality gap, the two bases may exhibit different granularities and semantics thus distributed in distinct hidden spaces, making it challenging to learn the coarse-grained alignment. Moreover, the frame and word representations are separately aggregated to the global features without additional interactions, which may increase the difficulty of discovering more granular correspondence (e.g., the frame-to-word, frame-to-phrase alignment) since only the coarse-level supervision is accessible in audio-text pairs. By the sparse constraints, p𝑖 can be represented by only a few codewords, which helps eliminate the noisy activation and enhance the interpretability. Similarly, the global text feature q𝑖 can also be constructed via the above way. Finally, we provide an intuitive view of how the proposed paradigm reaches fine-grained cross-modal alignment. Under the supervision of contrastive loss, the similarity of paired samples < p𝑖 , q𝑖 > is supposed to be maximized. However, due to the sparse regularization, the model may have to resort to the same, small set of codewords to represent the audio 𝑥 𝑖 and text 𝑦 𝑖 to increase < p𝑖 , q𝑖 >. Let 𝑘 * be one of the activated codewords. It then acts as prior targets, which requires the encoders to refine frame (or word) representations to maximize 𝑠 Obtaining meaningful local representations is crucial, otherwise, some codewords may be activated by mistake during feature aggregation. Recall that in the vanilla CLAP audio encoder, the outputs of the last transformer encoder block will be decoupled to produce the final frame-wise features. Its general architecture can be found in the upper of Figure 2 (c), which first employs self-attention to consider global contexts. Specifically, let 𝑈 = {𝑢 𝑙 |𝑢 𝑙 ∈ R 𝑑 } 𝑇 𝑙=1 be the input sequence of the block, the query, key, value matrices 𝑄, 𝐾, 𝑉 are first calculated by separate linear projections 𝑊 𝑞 ,𝑊 𝑘 ,𝑊 𝑣 ,EQUATIONThen, to compute output feature 𝑢 ′ 𝑙 for each frame 𝑙, the q-k attention is applied as follows,EQUATIONEQUATIONwhere 𝑞 𝑙 and 𝑣 𝑗 is the 𝑙 th and 𝑗 th vector of 𝑄 and 𝑉 . In this way, information from other frames 𝑣 𝑗 can be injected into the current frame 𝑙 by referring to the q-k similarity. However, according to the mechanism of self-attention [43] , we argue that 𝑣 𝑙 computed at each location 𝑙 already captures rich local semantics. By contrast, obtaining a comprehensive view by attention aggregation may impurify the local patterns, which may be a negative for fine-grained alignment. Figure 3 gives two examples to support our hypothesis. As seen, the v-v similarities (computed by Softmax(< 𝑣 𝑙 , 𝑉 > / √ 𝑑), which is similar to Equation ( 5)) of the last block are high within the same sound event while low among different events, meaning that acoustically dissimilar frames may exhibit distinct 𝑣 for finer-level discrimination. In comparison, the q-k similarities show inconsistency with the event boundary.Inspired by the above, we design the locality-aware block as shown in Figure 2 (c), which simply removes the q-k attention and directly leverages the projected value matrix 𝑉 as the output feature sequence 𝑈 ′ with other components unchanged. Besides, we only replace the last block of the audio branch with the locality-aware block, the reasons are: (1) the transformer receptive fields become much more global midway through the network as [35] suggest;(2) since the audio encoder is pre-trained on AudioSet to learn general acoustic patterns (a widely-adopted setting of previous CLAP variants), replacing the last one can retain most prior knowledge.Contrastive learning can benefit a lot from in-batch hard negative samples [37] . For vision-language tasks, several works [23, 45, 56] tend to resample or manually craft hard negative instances to improve alignment effects, which generally involves more training costs. In this work, we devise a simple re-weighting approach to force the modal to pay more attention to hard negative samples during optimization. The loss function is reformulated as follows,L HN_CLAP = - 𝐵 ∑︁ 𝑖=1 𝑙𝑜𝑔 𝑒 < p𝑖 , q𝑖 >/𝜏 𝑒 < p𝑖 , q𝑖 >/𝜏 + 𝑗,𝑗≠𝑖 𝛼 𝑖,𝑗 𝑒 < p𝑖 , q𝑗 >/𝜏 - 𝐵 ∑︁ 𝑖=1 𝑙𝑜𝑔 𝑒 < q𝑖 , p𝑖 >/𝜏𝑒 < q𝑖 , p𝑖 >/𝜏 + 𝑗,𝑗≠𝑖 𝛽 𝑖,𝑗 𝑒 < q𝑖 , p 𝑗 >/𝜏 (7) where 𝛼 𝑖,𝑗 , 𝛽 𝑖,𝑗 is the audio-to-text and text-to-audio difficulty scores for unpaired samples, they are designed so that hard negative pairs (with higher similarity compared to the average score) are emphasized in loss computation, and easier pairs are neglected. As a result, the model will be forced to learn a more discriminative feature space to distinguish confusable pairs for multi-grained alignment. The formula is written as,𝛼 𝑖,𝑗 = 𝐵𝑒 𝛾 < p𝑖 , q𝑗 >/𝜏 𝑘 𝑒 𝛾 < p𝑖 , q𝑘 >/𝜏 , 𝛽 𝑖,𝑗 = 𝐵𝑒 𝛾 < q𝑖 , p 𝑗 >/𝜏 𝑘 𝑒 𝛾 < q𝑖 , p𝑘 >/𝜏 (8)where 𝛾 is a scaling ratio, the larger it is, the more importance we attach to the hard negative samples as the distribution of 𝛼 𝑖,: and 𝛽 𝑖,: can be sharper. 
experimental setup 4.1 pre-training: 
 Dataset. We merge WavCaps, the training set of AudioCaps and Clotho for pre-training, including about 450K audio-text pairs. Architecture. We employ the pre-trained BERT [18] base model as the text encoder which contains 110M parameters. While for the audio encoder, to examine the scalability of the proposed method, we adopt a patch-wise model HTS-AT (27M) [3] and a frame-wise AST (86M) [24] , all of them are trained on the AudioSet by previous works and we directly use the checkpoints. Besides, a two-layer MLP is appended after the encoder, projecting the multi-modal features into the same dimension 𝐷 = 1024. Implementation Details. We train our model for 15 epochs with a batch size of 128 and a learning rate of 5e-5 using the Adam optimizer. The hyper-parameter 𝜏 is learnable with an initial value of 0.07 and 𝛾 and 𝑀 are fixed to 0.15 and 4096 empirically. Besides, all the audio clips and captions are randomly cropped or padded to 10 seconds and 30 words to guarantee the fixed-sized length. We also resample the waveform to 32KHz and 16KHz for HTS-AT and AST following the original works. During training, audio clips with similar durations are grouped within a batch for training efficiency. Finally, model checkpoints are selected based on their performance on validation sets after each epoch and the final model performances are evaluated on corresponding test sets.To comprehensively evaluate the model performance, we conduct experiments on several coarse-grained tasks (including audio retrieval, audio classification, and audio tagging) and fine-grained tasks (including sound event detection and text-to-audio grounding). Note that for each specific task, we would pre-train a new model from scratch with a newly constructed dataset excluding all the overlapped samples in the downstream evaluation, meaning that the zero-shot inference is performed. Specifically, for tasks other than retrieval, we directly use sound class names as the input of the text encoder, which avoids heavy prompt engineering. We will report the averaged metric of 3 different runs and the detailed evaluation protocols are provided in Table 1 . For single-label and multi-label classification tasks, Acc and mAP are widely adopted metrics. For retrieval tasks, R@k is 1 if the positive item appears in the top k retrieved items for a query [20] . And for detection and grounding tasks, PSDS1 is more sensitive to the precise localization Text2Audio Audio2Text Text2Audio Audio2Text R@1 R@5 R@1 R@5 R@1 R@5 R@1 R@5 FLAP (fusion) 41.5 75. 5  
model performance: 
 5.1.1 Performance on Coarse-grained Tasks. We compare our proposed MGA-CLAP not only with the original CLAP but also with the SOTA model in each separate task. Specifically, for zero-shot retrieval, we involve FLAP (fusion) [57] and Cacophony [59] for comparison. The former is trained on LAION-Audio-630K using a more powerful audio encoder MAViL [16] and employs feature fusion proposed in [48] to process audios longer than 10s instead of directly cropping it. And the latter is trained on a 4M audiotext dataset with LLM re-captioning, which is much larger than our 450K pairs. While for zero-shot classification, we additionally involve CompA [12] , which leverages an instruction-tuned Flan-T5-large model (770M) [34] as the text encoder and CompA-661K (an extension of LAION-Audio-630K) as the pre-training set.The results concerning coarse-grained retrieval and classification tasks are reported in Table 2 and 3 , respectively. As for retrieval, the proposed MGA-CLAP largely surpasses the original CLAP no matter which backbone is applied. When compared with current SOTA methods which involve more training resources, our MGA-CLAP is also competitive, achieving the best performance on 6 of 8 metrics. And for classification and tagging tasks, similar improvements over the original CLAP can also be observed. Noticeably, our MGA-CLAP with HTS-AT encoder reaches 31.8% accuracy on VGGSound, the most complex single-label classification dataset with 300+ classes, which is 2.3% higher than the previous SOTA, CompA. These above results underscore MGA-CLAP's ability to capture cross-modal alignment between texts and audio, leading to outstanding performance in versatile classification and retrieval tasks.5.1.2 Performance on Fine-grained Tasks. We reimplement and retrain UACA [50] and WSTAG [54] using the CLAP paradigm since the original works only experiment on tiny datasets. Besides, we also reproduce PACL [31] , a recent vision-language training framework, which employs cross-modal attention pooling to align local features with captions and demonstrates superior performance on fine-grained visual understanding tasks. All the above methods are implemented based on the HTS-AT backbone and we keep the training and evaluation settings consistent with MGA-CLAP. As mentioned before, the original CLAP cannot uncover finegrained alignment between frame features and text descriptions. As depicted in Table 4 , it obtains extremely low scores, especially on time-sensitive metrics, such as PSDS1 and PSDSm. Although UCAC and WSTAG attempt to solve this problem, their results are still unpromising, since the frame-to-word interaction is modeled via simply score pooling. Besides, directly transferring PACL leads to a better but still suboptimal outcome. As a comparison, our MGA-CLAP with HTS-AT backbone obtains PSDS1 scores of 26.4%/8.7%/10.1% on DESED/UrbanSED/AudiosSet-Strong eval sets, which are 2x/5x/3x times those of the original CLAP. When switching the audio encoder to AST, better performances are witnessed on datasets containing more queries, such as AS-S and TAG.In this section, we ablate the module designs and hyper-parameter choices of the proposed MGA-CLAP. All experiments are conducted based on the HTS-AT backbone.Table 5 : Ablation study on sub-modules, where MC, LB, and HN denote modality-shared codebook, locality-aware block, and hard-negative guided loss, respectively. For retrieval and detection tasks, we solely list the R@1 and PSDS1 score. a specific sub-module. As seen, the incorporation of a modalityshared codebook boosts CLAP's understanding capabilities on both coarse-grained and fine-grained tasks, as it not only adopts common bases to represent global audio and text features but also links multi-modal local features with shared codewords. However, solely training with it cannot lead to satisfactory results on finegrained tasks due to the inferiority of frame-wise representations. When further adopting the locality-aware encoder block, the PSDS scores on DESED, AS-S, and TAG datasets are improved by 5.1%, 2.2%, and 5.9%, respectively, suggesting the necessity of acquiring high-quality frame features. Additionally, simply involving the locality-aware block can also enhance the model performance on detection and grounding tasks. Finally, further equipped with the hard-negative loss, the whole system can achieve optimal results on each task as it can enhance the contrastive learning scheme.Study on the Size of Codebook. We compare the modality-shared codebook in different sizes in Figure 5 . As seen from the left figure, the R@1 scores on AudioCaps drop significantly when the number of codewords 𝑀 increases from 4096 to 8192. We argue that the enlarged codebook may bring about noisy activated codewords and irrelevant information while aggregating, making it difficult to retrieve matched pairs. Besides, it is also at risk of underfitting since some codewords may be undertrained. By contrast, although fewer number of codewords generally leads to slightly better outcomes on retrieval tasks, its performance on frame-level tasks decreases a lot as shown in the right figure. The possible reason is that each codeword must convey multiple semantics within a smaller codebook, thereby disturbing the frame-word interaction while seeking fine-grained alignment. Finally, we choose the number of codewords 𝑀 = 4096 to make a trade-off between multigrained tasks.Locality-aware Block. We conduct parameter analysis on the number of vanilla transformer blocks to be replaced by the locality-aware ones in Figure 5 . It can be observed that the incorporation of locality-aware blocks contributes a lot to the enhanced capability due to the refinement of frame-wise features. Additionally, adopting 1 or 2 locality-aware blocks has similar effects on the downstream tasks. However, as the number grows, the performance degradation is witnessed. This is possibly due to more locality-aware blocks destroying the information flow and pre-trained knowledge in the transformer backbone.As stated before, 𝛾 in Equation ( 8) controls the difficulty of negative samples with a higher value paying more attention to harder ones. We then study the effects of its numerical values in Figure 6 . The results indicate that 𝛾 = 0.15 or 𝛾 = 0.10 generally yields better outcomes as a larger one may overemphasize the hard negative samples and potentially neglect the relation with other in-batch data points.We ablate the detailed designs, namely the max pooling to compute the affinity scores and the Sparsemax function to normalize aggregation weights, in the modality shared codebook and provide the related outcomes in Table 6 . As shown, if applying the mean pooling instead of max pooling, some non-salient local cues may be overwhelmed by the primary sound. Then severe performance drops can be found in tagging, detection and grounding tasks, where local patterns play an important role in recognition. And when changing the activation function to Softmax, the system produces poor results on all tasks, which is only slightly better than the original CLAP. We Table 6 : Ablation study on designs of the codebook, where -, (1), (2) denote the current setting, replacing max pooling with mean pooling, replacing Sparsemax with Softmax. argue that with Softmax, the aggregation weights are no longer sparse, introducing many noisy components when representing the global features. As a result, the semantic meanings of codewords may be blurred and the connection between frame and word will be influenced.In this subsection, we try to disclose the meanings of some representative modality-shared codewords. For a specific codeword, we first compute its cosine similarity with textual features of sound classes taken from AudioSet taxonomy to find out its semantics. Then we compute its cosine similarity with frame representations to examine if it also correlates with acoustic features. The results are given in Figure 7 . Taking the 1st row as an example, the 2917th codeword has a large similarity with textual descriptions related to "dog", suggesting its represented semantics. Besides, as shown in the two sub-figures, the cosine similarity between the codeword and frame features also shows synchronization with the temporal location of sound events. Specifically, the scores are high when a dog bark is truly presented while low when the sound is absent. When comparing the 2nd and 3rd rows, it can be seen that the codeword can encode finer acoustic attributes, such as the gender of speakers. Finally, as for the 4th row, the semantics of rare sound classes (e.g., sewing machine) can also be learned from the MGA-CLAP pipeline. And from the last figure, one can see that the semantic mapping is salient even under polyphonic environments.5.3.2 Fine-grained Alignment. We provide several examples of MGA-CLAP achieving fine-grained alignment in Figure 8 . The cases show that our method may capture both frame-to-phrase (seen from the first row) and frame-to-caption (seen from the second row) correspondence, thereby obtaining promising results on zeroshot detection and grounding tasks. Surprisingly, it can tell apart barking and whimpering sounds at frame-level as seen in sub-figure (e), which are both made by dogs but varied in pitches, suggesting that the subtle semantics of captions are also well aligned with acoustic characteristics. Moreover, we also visualize some bad cases in Figure 9 . Currently, our MGA-CLAP may be confused about similar sounds such as blender and vacuum cleaner (seen from Figure 9 (a)) and fail to capture long-duration dependency sometimes (seen from Figure 9 (b)). Moreover, it may omit certain sounds (such as blender in Figure 9 (c)) especially when multiple acoustic events take place simultaneously. 
conclusion: 
 In this work, we devise MGA-CLAP to align audio features with language descriptions from both coarse-grained and fine-grained views. To achieve this goal, MGA-CLAP employs a codebook to construct a shared feature space for cross-modal interaction and optimize its internal codewords carefully to seek frame-word correspondence. Based on the modality-shared codebook, a novel encoder block is designed to enhance the salience of local patterns while a re-weighting loss term is considered to mine hard-negative pairs during optimization for better cross-modal alignment. By pretraining on large audio-text datasets, our MGA-CLAP not only outperforms the baseline CLAP to a large extent but also yields better or competitive outcomes on versatile language-audio understanding tasks when compared with current SOTA variants. Through extensive visualizations and ablation studies, the effectiveness of proposed designs is verified.  
 
 Appendix:  


Decision: The paper introduces MGA-CLAP, which incorporates several small extensions over the original CLAP, where audio features and language descriptions can be aligned from both coarse-grained and fine-grained viewpoints. Authors introduce an additional small codebook which is shared across modalities to encourage the multi-modal features to interact on a finer granularity. Experimental results on various tasks demonstrate the effectiveness of the proposed framework, setting SOTA on some tasks. Reviewers asked for additional comparisons and clarifications which authors were able to provide. While the paper does not have fundamental novelty, it presents a strong contribution to cross- and multi-modal representation learning and is thus a good candidate for ACM MM.

Reasons to accept:
- The paper improves on a fundamental and important approach in audio/ text modeling.
- The modeling framework is well designed and supports temporal localization of sound events, which adds to the quality and novelty of the work over other contrastive frameworks.
- Authors conduct comprehensive experiments from audio-text retrieval to zero-shot audio classification, audio tagging, sound event detection, and text-to-audio grounding tasks, showing the good performance of the proposed approach. In the rebuttal, authors provided additional baselines on these tasks. 
- Authors confirm code and checkpoints will be released.

Reasons to reject:
- Reviewers still have a number of questions that would deserve more detailed answers, e.g. the consequences of using sparsemax, noise robustness, and trade-off between local and global features.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel approach for improving both coarse- and fine-grained audio-language alignment in large-scale contrastive pre-training. The method involves using a shared codebook to unify the granularity and latent distribution of audio and text features. A locality-aware block purifies local patterns, and a hard-negative guided loss boosts alignment effects.  These paper gives the following contributions:
1. Contrastive Language-Audio Pre-training (CLAP): Previous models aggregate uni-modal local representations into global ones for coarse-grained alignment but often ignore fine-grained text-audio challenges.
2. Shared Codebook: This approach uses a shared codebook to represent multi-modal global features with common bases, encoding modality-shared semantics.
3. Locality-aware Block: A new block purifies local patterns to enhance frame-level features.
4. Hard-Negative Guided Loss: This loss function emphasizes hard negative samples to improve alignment.
strength_weakness: strength_weakness: 1. Good Approach: The introduction of a shared codebook for unifying granularity and latent distribution between audio and text features is a novel contribution.
2. Extensive Evaluation: The model is tested on eleven different zero-shot coarse- and fine-grained tasks, showcasing its versatility and robustness.
3. Significant Improvements: MGA-CLAP shows marked improvements over baseline and competitive results against SOTA models.
limitations: limitations: The proposed method is interesting, and the experiments also support the claims.  One of conerns is about the Comparisons method, author should better present the baselines's parameters and training data size in the Table. So reader can better understand the advantiges.
paper_summary: paper_summary: The paper presents an advanced method for improving multi-grained alignment in contrastive language-audio pre-training by introducing a modality-shared codebook and a locality-aware block. This approach enhances both coarse-grained and fine-grained alignment, addressing the limitations of existing models like CLAP.
strength_weakness: strength_weakness: (1) By concentrating on local patterns, the locality-aware block enhances the ability to capture detailed audio events and their corresponding text descriptions. This improvement boosts performance in tasks that demand precise timing information, like sound event detection and text-to-audio grounding.

(2) Authors conducted  extensive experiments on various datasetes for zero-shot coarse- and fine-grained evaluation  on different tasks, demonstrating the model's superior performance compared to both the baseline CLAP and other state-of-the-art methods.

(3) This approach of using a shared codebook for aligning multimodal features is relatively noval and addresses the granularity gap between audio frames and text tokens.

(4) Enhancing contrastive learning with hard negatives is a known technique, but its specific application and adaptation for multimodal audio-text alignment provide a unique contribution.
limitations: limitations: (1) Comparative Analysis with other state of the art papers: Comparative analysis with other state of the arts methods such as such as AudioCLIP [1], Wav2CLIP [2], BEATs [3], and PANNs [4] is  limited. Comparison with these method  would highlight MGA-CLAP’s unique advantages.

(2) My doubt is by removing the q-k attention mechanism, the model loses the ability to incorporate global context. This might lead to fragmented representations, where each frame is considered in isolation, potentially missing out on relationships between frames that span longer durations.

(3) The reliance on Sparsemax for normalizing aggregation can lead to codeword sparsity that could potentially lead to underutilization of available codewords. This might result in suboptimal performance when dealing with highly diverse audio and text data, where more nuanced representations could be beneficial. 

(4) To determine the effectiveness of fine-grained features it will be interesting to see how the model performs on generating captions for given audio clips and capturing detailed semantic information.

(5) How current architecture if effective against noise.?The current evaluation of the MGA-CLAP model lacks an analysis of its robustness to noise. Incorporating noise robustness experiments will provide a comprehensive understanding of the model performance, especially for fine-grainded audio tasks. 



[1] Guzhov, Andrey, Federico Raue, Jörn Hees, and Andreas Dengel. "Audioclip: Extending clip to image, text and audio." In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 976-980. IEEE, 2022.

[2] Wu, Ho-Hsiang, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. "Wav2clip: Learning robust audio representations from clip." In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4563-4567. IEEE, 2022.

[3] Chen, Sanyuan, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. "Beats: Audio pre-training with acoustic tokenizers." arXiv preprint arXiv:2212.09058 (2022).

[4] Kong, Qiuqiang, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley. "Panns: Large-scale pretrained audio neural networks for audio pattern recognition." IEEE/ACM Transactions on Audio, Speech, and Language Processing 28 (2020): 2880-2894.
paper_summary: paper_summary: This paper proposes a framework to improve both coarse- and fine-grained audio-language alignment in large-scale contrastive pre-training. Specifically, the paper employs a codebook to construct a shared feature space for cross-modal interaction and optimize its internal codewords carefully to seek frame-word correspondence. And then a novel encoder block is designed to enhance the salience of local patterns as a hard-negative guided loss is devised to boost alignment effects.
strength_weakness: strength_weakness: The method proposed in the paper is novel and interesting, utilizing a shared codebook to represent multi-modal global features with common bases. This approach could be beneficial for some multi-modal tasks, such as multi-modal alignment and explainability. Additionally, the paper conducts extensive experiments and provides thorough ablation studies on different modules of the model. The experimental results are well visualized. In the end I believe that this work is a good contribution to the community.
limitations: limitations: In Section 4.2 of the paper, it is mentioned that "Note that for each specific task, we would pre-train a new model from scratch with a newly constructed dataset ...", does the newly constructed dataset here refer to the dataset mentioned in Section 4.1 and the task-specific datasets in Table 2 combined? And why not directly merge and use the training sets of all tasks to train a powerful CLAP model?
Additionally, there are some minor errors in the symbols used in the paper, please check and correct them, for example, line 286.
paper_summary: paper_summary: This paper proposes MGA-CLAP, where audio features and language descriptions are aligned from both coarse-grained and fine-grained viewpoints. The authors adopt a modality-shared codebook to encourage the multi-modal features to interact on a finer granularity. Their experimental results in various tasks demonstrate the effectiveness of the proposed framework. In addition, extensive ablation studies are conducted to show the effects of each component.
strength_weakness: strength_weakness: - Dealing with an important topic regarding audio-text cross-modal modeling.
- Well-designed training framework based on a deep insight into the characteristics of audio-text paired data. Each sound event inherently happens in a temporarily local region, whereas most contrastive representation learning frameworks do not consider this property of sound events. The unawareness of this may deteriorate the performance of cross-modal representation. This study addresses this issue.
- Comprehensive experiments from audio-text retrieval to zero-shot audio classification, audio tagging, sound event detection, and text-to-audio grounding tasks. The proposed MGA-CLAP performs very well on all conducted tasks. Ablation studies are also sufficient, which answer questions most readers will have.
limitations: limitations: - I'm wondering if this model is easily trained or not. More specifically, does introducing a codebook/locality-aware block affect convergence speed or training stability?
- The authors can contribute more to the community by making the code and pretrained checkpoint publicly available
- I'm interested in whether trained text features can improve the performance of text-to-audio generation models (such as AudioLDM). This is a minor comment from my curiosity, which I do not take into account in rating the paper because this is out of the scope.