Multi-grained Correspondence Learning of Audio-language Models for Few-shot Audio Recognition

Authors: shengwei  zhao
linhai  xu
yuying  liu
shaoyi 2024 du
  multi-grained
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 19:47:37

Abstract: Large-scale pre-trained audio-language models excel in general multi-modal representation, facilitating their adaptation to downstream audio recognition tasks in a data-efficient manner. However, existing few-shot audio recognition methods based on audio-language models primarily focus on learning coarse-grained correlations, which are not sufficient to capture the intricate matching patterns between the multi-level information of audio and the diverse characteristics of category concepts. To address this gap, we propose multi-grained correspondence learning for bootstrapping audio-language models to improve audio recognition with few training samples. This approach leverages generative models to enrich multi-modal representation learning, mining the multi-level information of audio alongside the diverse characteristics of category concepts. Multi-grained matching patterns are then established through multi-grained key-value cache and multi-grained cross-modal contrast, enhancing the alignment between audio and category concepts. Additionally, we incorporate optimal transport to tackle temporal misalignment and semantic intersection issues in fine-grained correspondence learning, enabling flexible fine-grained matching. Our method achieves state-of-the-art results on multiple benchmark datasets for few-shot audio recognition, with comprehensive ablation experiments validating its effectiveness.

Main paper: 
 introduction: 
 Large-scale pre-trained audio-language models [10, 11, 15, 38] demonstrate powerful general multi-modal representation capabilities. However, directly fine-tuning the audio-language model with downstream task data will disrupt the pre-set embedding space. To adapt the audio-language model to downstream tasks, recent work [24, 43] has achieved good performance on few-shot audio recognition by fine-tuning the audio-language model through efficient adapters for coarse-grained correspondence learning. Among them, Treffadapter [24] mines the correlation between audio and category concepts through coarse-grained key-value cache and zero-shot coarse-grained cross-modal contrast.The above methods achieve good performance, but their coarsegrained correspondence learning is not sufficient to represent the complex matching patterns between audio and category concepts. Taking the audio of "moving train" as an example, the short whistle sound and the continuous wheel rolling sound are the two corresponding discriminant characteristics. The former requires attention to audio details (clip-level), while the latter requires attention to the overall audio (overall-level). Coarse-grained audio representation makes it difficult to pay attention to the whole and details of the audio at the same time, and a short category name ("moving train") cannot fully represent the potential discriminative characteristics of the category concept (e.g. whistle, wheel rolling). Therefore, coarse-grained correspondence learning is difficult to establish multi-grained matching patterns between the multi-level information of audio and the diverse characteristics in category concepts.In order to solve the above problems, we propose to perform multi-grained correspondence learning to better guide the audiolanguage model to adapt to few-shot audio recognition. Our method contains three core designs: gen-assisted multi-modal representation learning, multi-grained key-value cache and multi-grained cross-modal contrast. Specifically, we first perform multi-modal representation learning with the assistance of the generative models, which is used to mine the multi-level information of audio and the diverse characteristics of category concepts. We then propose multigrained key-value cache and multi-grained cross-modal contrast for effectively learning multi-grained matching patterns between audio and category concepts. The former maps the categories of test audio by capturing multi-grained correlations between test audio representations and known category audio representations. While the latter mines caption-audio complementary patterns to learn multi-grained correlations between audio fusion representations and category explanation representations.In addition, we noticed that there are problems of temporal misalignment and semantic intersection when performing fine-grained correspondence learning. Taking the audio of "moving train" as an example, the key information in the audio (whistle sound, rail banging, wheel rolling) can appear at the beginning, end or continuously of the audio. As shown in Fig. 1 -a, the relevant parts of two similar audios may not appear in the same time segment (temporal misalignment). And as shown in Fig. 1-b , some segments in the audio can express multiple semantics at the same time (semantic intersection). In order to deal with the above problems, since optimal transport can establish flexible correspondences between sampling points of different probability distributions, we propose to apply optimal transport to improve fine-grained key-value cache and fine-grained cross-modal contrast. Specifically, we model the empirical distribution of fine-grained audio representation and category explanation representation, and guide flexible fine-grained matching between audio and category by comparing the distance between distributions under the optimal transport framework.To effectively showcase the efficacy and performance of our proposed method, we meticulously carry out a sufficient number of experiments on two prominent few-shot audio recognition benchmark datasets, namely ESC-50 and FSDkaggle18. When contrasted with the previously existing methods, our proposed multi-grained correspondence learning approach successfully attains state-of-theart outcomes and demonstrates superior results. We hope that our findings will inspire more research on few-shot audio recognition. Our main contributions can be summarized as follows: • We propose multi-modal representation learning assisted by generative models, which effectively mine the multi-level information of audio and the diverse characteristics of category concepts. • We propose multi-grained correspondence learning to establish multi-grained matching patterns between audio and category concepts through multi-grained key-value cache and multi-grained cross-modal contrast. • We propose to apply optimal transport to deal with the temporal misalignment and semantic intersection problems existing in fine-grained correspondence learning, and to guide the flexible fine-grained matching. 
related work 2.1 audio-language models: 
 In recent years, audio-language models [10, 11, 15, 38, 44] have attracted increasing attention due to their powerful universal multimodal representation capabilities. Taking the representative CLAP [10] as an example, it follows the idea of Contrastive Image-Language Pre-training (CLIP) [32] . Independent audio encoders and text encoders are employed to extract audio representations and their corresponding text representations. Subsequently, contrastive learning is utilized to project both the audio and text representations into a unified embedding space. This process brings matching audio-text pairs closer together in the embedding space while pushing apart unmatched pairs, thus enhancing the alignment and discrimination between relevant and irrelevant audio-text combinations. With the assistance of training data of large-scale audio-text pairs, audiolanguage model has good generalization ability to a wide range of downstream audio tasks. By evaluating the similarity between audio and category label names, audio-language models can perform downstream audio recognition without training. However, the gap between pre-training data and downstream task data reduces the generalization performance of audio-language models, which has prompted recent research on guiding audio-language models to adapt to downstream tasks, such as efficient adapter [24, 43] for few-shot audio recognition.With the introduction of large-scale audio datasets [9, 12, 13, 18, 28, 30] and the development of audio representation learning theory [2, 3, 14, 27] in recent years, audio classification has made great progress as a benchmark task in the audio field. Since the success of existing audio classification methods is based on labor-intensive acquisition of high-quality audio labels, inspired by the mature few-shot image recognition methods [34, 42, 43, 46] , few-shot audio recognition [24, 33, 35] has received attention in data-limited scenarios, where models are constrained to learn from limited audio with labels. Early metric-based few-shot audio recognition methods [5, 16, 17, 23, 33, 35, 37] aimed to cluster unlabeled examples based on their distance from several examples of each category. Among them, the matching network [35] learns a set of mapping matching relationships between supporting data and queries. The prototype network [33] learns different category prototype representations and maps categories based on the Euclidean distance between the query representation and the prototype representation. However, without the support of multi-modal basic models [10, 11, 15, 38] , it is difficult for early methods to perfectly solve the problem of few-shot audio recognition. Recently, inspired by the success of Tip-adapter [43] in image recognition, Treff-adapter [24] conducts coarse-grained correspondence learning through efficient adapters to guide the audio-language model to adapt to few-shot Audio Recognition. Specifically, Treff-adapter learns the coarse-grained correlation between the test audio representation and the audio representation of the known category, and utilizes the key-value cache to map the category of the test audio. And the zero-shot cross-modal contrast is performed to capture the coarse-grained correlation between the category name representation and the coarse-grained audio representation. Although the above methods achieve good performance, their coarse-grained correspondence learning makes it difficult to establish multi-grained matching patterns between audio and category concepts. Therefore, we propose multi-grained correspondence learning for effectively learning multi-grained matching patterns between multi-level audio information and diverse category characteristics, which benefits from our application of optimal transport for fine-grained correspondence learning.Optimal transport [29] was originally developed to solve the problem of how to move multiple materials at the same time at the lowest cost, and can be used to evaluate the distance between two probability distributions. Since many modern statistical and machine learning problems can be reformulated as finding optimal transport graphs between two probability distributions, optimal transport has recently attracted widespread attention in the fields of machine learning and computer vision, such as document matching [21, 40] , image matching [1, 26, 36, 41, 45] , video retrieval [25] , and visual-language models [4, 8, 19, 31, 39] . However, none of these works specifically focus on few-shot audio recognition and correspondence learning between audio and text, which is the main focus of our research. Experimental results show that fine-grained correspondence learning based on optimal transport effectively establishes flexible fine-grained matching patterns between audio and category concepts. 
methodology: 
 In this section, we elaborate on each component of our proposed method, whose overall structure is shown in Fig. 2 . We first introduce how to learn multi-modal representations with the assistance of generative models in Sec.3.1. Subsequently, we detailed how to promote multi-grained correspondence learning through multigrained key-value cache and multi-grained cross-modal contrast in Sec.3.2 and Sec.3.3. Finally, we describe in Sec.3.4 the evaluation of the overall matching probability for few-shot recognition.3.1.1 Audio representation. For a sampled audio, the mel spectrogram of the audio is input into the CNN14 audio encoder [2] to extract the audio feature map 𝐹 𝑎 ∈ R 𝑛 𝑎 ×𝑓 𝑎 ×𝑑 , where 𝑛 𝑎 /𝑓 𝑎 represent the time/frequency dimension and 𝑑 is the latent feature dimension. The width and height of an audio mel spectrogram represent different information (i.e. time and frequency bins). Since the length of time is usually much longer than the length of frequency bins, we perform average pooling on the audio feature map 𝐹 𝑎 along the frequency direction to obtain a fine-grained audio representation (clip-level)𝑅 𝑓 𝑎 = {𝑎 𝑖 } 𝑛 𝑎 𝑖=1 ∈ R 𝑛 𝑎 ×𝑑, where 𝑛 𝑎 represents the number of time segments. The average of the fine-grained audio representation 𝑅 𝑓 𝑎 can be utilized as the coarse-grained audio representation (overall-level) 𝑅 𝑐 𝑎 ∈ R 𝑑 .We noticed that in real-world scenarios, audio clips on video sharing platforms often come with relevant textual descriptions, such as titles and tags. The semantic information contained in these texts is crucial for improving audio recognition in situations where data availability is limited. This inspires us to utilize pre-trained encoder-decoder models [28] to generate audio captions without training. Specifically, we adopt HTSAT [2] as the audio encoder to extract audio features. The HTSAT audio encoder is an audio transformer similar to the visual transformer (VIT). And we use BART [22] as the language decoder to generate captions based on the audio features extracted from the encoder. From this, we generate a corresponding caption for each audio, and BERT [7] is used as a text encoder to extract the audio caption representation (cap-level) 𝑅 𝑡 ∈ R 𝑑 . Illustration of the proposed multi-grained correspondence learning, which is used to guide the adaptation of audiolanguage models for few-shot audio recognition. Multi-modal representation learning with the assistance of generative models is performed to mine the multi-level information of audio and the diverse characteristics of category concepts. Multi-grained key-value cache and multi-grained cross-modal contrast are then proposed to effectively learn multi-grained matching patterns between audio and category concepts through optimal transport modeling (OT) and late weighted fusion.Traditional audio recognition methods map category labels into discrete numerical codes for classification, which ignores the semantic information contained in category names. We use preset prompts to guide large language models (e.g. chatgpt 1 ) to generate descriptive explanations of category labels, which are used to understand the potential meaning of category concepts. Specifically, we carefully design a variety of prompt templates to guide chatgpt to generate 𝑛 𝑡 descriptive explanations for each category label. Different category explanations illustrate different discriminative characteristics of audio in that category. Since BERT [7] has strong language modeling capabilities, it is used as a text encoder to extract text representations of descriptive explanations 𝑅 𝑒 = {𝑒 𝑖 } 𝑛 𝑒 𝑖=1 ∈ R 𝑛 𝑒 ×𝑑 , where 𝑛 𝑒 represents the number of explanations corresponding to each category.Multi-grained key-value cache is designed to capture multi-grained correlations between test audio and known category audio to map the category of the test audio. We consider key-value cache forms of multi-grained audio representations, including coarse-grained key-value cache and fine-grained key-value cache.The coarse-grained key-value cache is used to map the corresponding categories by mining the similarity between the coarse-grained representations of the test audio and training audios. Under the "K-shot N-class" few-shot recognition setting, audio needs to be divided into 𝑁 categories, and each category provides 𝐾 audio samples of known categories. For each audio 𝑎 𝑖 in the 𝑁 𝐾 training sets, the audio representations are treated as keys and their corresponding categories are treated as values after being converted into N-dimensional one-hot vectors 1 https://openai.com/blog/chatgpt/ 𝑉 𝑖 ∈ R 1×𝑁 . And the representations of test audio of unknown categories are treated as queries. The similarity between the query and the key can be regarded as the affinity between the query and the value, which is used to map the matching probability between the query audio and 𝑁 categories.Specifically, for coarse-grained key-value cache, we consider coarse-grained representations of a test audio 𝑅 𝑐 𝑎,𝑡𝑒𝑠𝑡 ∈ R 𝑑 and training audios 𝑅 𝑐 ′ 𝑎,𝑡𝑟𝑎𝑖𝑛 = {𝑅 𝑐 ′ 𝑎,𝑖 } 𝑁 𝐾 𝑖=1 ∈ R 𝑁 𝐾 ×𝑑 . Inspired by Treffadapter [24] , cosine similarity is used to evaluate the coarse-grained affinity 𝑀 𝑐 𝑎𝑓 𝑓 ,𝑖 between the test audio and the training audio set. Finally, the matching probability 𝑃𝑟𝑜 𝑐𝑘𝑐 ∈ R 1×𝑁 between the test audio and 𝑁 categories is determined by the coarse-grained affinity and the one-hot category vector 𝑉 𝑐 = {𝑉 𝑖 } 𝑁 𝐾 𝑖=1 ∈ R 𝑁 𝐾 ×𝑁 , defined as follows:EQUATIONEQUATIONwhere 𝜑 (𝑥) = 𝑒𝑥𝑝 (-𝛽 (1 -𝑥))) represents the scaling function.Fine-grained key-value cache. The fine-grained key-value cache is used to map the corresponding categories by mining the similarity between the fine-grained representations of the test and training audios. However, as explained in Sec.1, there is a temporal misalignment problem between the test audio and the training audios. The key information in audio is distributed randomly over time, and it can appear at the beginning or end of the audio, so the relevant parts of two similar audios may not appear in the same time segment (Fig. 1-a ). Since fine-grained audio representations encompass audio information across multiple segments in chronological order, We need a metric that is independent of segment order to establish flexible correspondences between fine-grained representations of test and training audios. Therefore, we propose to apply optimal transport (OT) to solve the temporal misalignment problem by modeling the empirical distribution of fine-grained audio representations and learning a transport plan to capture the fine-grained flexible correspondence of test and training audios. This is because the optimal transport (OT) distance, as a metric for distribution comparison, can establish a flexible correspondence (regardless of the order) between sampling points of different probability distributions. This allows optimal transport to align relevant audio information at different time periods between audios without being constrained by time order. Specifically, taking a test audio sample and a training audio sample as an example, we apply two empirical distributions 𝑃 and 𝑄 to model the fine-grained audio representation of a test sampleEQUATIONwhere 𝛿 𝑡 represents the dirac function defined at 𝑡, and the discreteprobability vectors 𝑝 = {𝑝 𝑖 } 𝑛 𝑎 𝑖=1 and 𝑞 = {𝑞 𝑗 } 𝑛 ′ 𝑎𝑗=1 belong to 𝑛 𝑎 and 𝑛 ′ 𝑎 -dimensional simplex. Optimal transport aims to minimize the OT distance between distributions 𝑃 and 𝑄. This is used to capture flexible correlation patterns between test audio and training audios without temporal order constraints, which can be formulated as an optimization problem guided by entropy regularization:EQUATION𝑇 1EQUATION𝑐 𝑖 𝑗 = 1 - 𝑎 𝑖 (𝑎 ′ 𝑗 ) 𝑇 ||𝑎 𝑖 || 2 ||𝑎 ′ 𝑗 || 2 (7)Among them, 𝑇 ∈ R 𝑛 𝑎 ×𝑛 ′ 𝑎 is the transport plan to be learned, 𝑇 𝑖 𝑗 represents the transport probability from the i-th audio segment to the j-th audio segment. The larger its value, the closer the semantic connection between the two segments. ℎ(𝑇 ) is the entropy constraint, and 𝜆 ≥ 0 is the weight of entropy regularization. 𝐶 𝑖 𝑗 is the transport cost between the i-th audio segment and the j-th audio segment. Considering that the greater the transport cost, the lower the transport probability, the cosine similarity between audio segments is used to construct the cost matrix 𝐶 ∈ R 𝑛 𝑎 ×𝑛 ′ 𝑎 . And we consider uniform distribution for 𝑝 and 𝑞. Then, we can obtain a fast optimized matrix solution 𝑇 * for the transport plan 𝑇 through Sinkhorn fixed point iteration [6] :EQUATION𝐼𝑡𝑒𝑟𝑎𝑡𝑖𝑣𝑒𝑙𝑦 : 𝑝/(𝑒𝑥𝑝 (-𝐶/𝜖)𝑘 2 )-> 𝑘 1 , 𝑞/(𝑒𝑥𝑝 (-𝐶/𝜖) 𝑇 𝑘 1 )-> 𝑘 2Among them, 𝑘 1 and 𝑘 2 are predefined left and right scaling vectors.During the above optimization process, we obtain the optimal transport plan 𝑇 * ∈ R 𝑛 𝑎 ×𝑛 ′ 𝑎 between fine-grained representations of test and training audios, which reflects the tightness of the semantic connection between audio segments. Accordingly, the optimal transport plan is used to define the fine-grained similarity 𝑆𝑖𝑚 𝑂𝑇 between test audio representation 𝑅 𝑓 and training audio representation 𝑅 𝑓 ′ as follows:𝑆𝑖𝑚 𝑂𝑇 (𝑅 𝑓 , 𝑅 𝑓 ′ ) = 𝑛 𝑎 ∑︁ 𝑖=1 𝑛 ′ 𝑎 ∑︁ 𝑗=1 𝑇 * 𝑖 𝑗 𝑎 𝑖 (𝑎 ′ 𝑗 ) 𝑇 ||𝑎 𝑖 || 2 ||𝑎 ′ 𝑗 || 2 (9)Similar to coarse-grained, for fine-grained key-value cache, we consider fine-grained representations of a test audio 𝑅 In summary, the overall matching probability of the multi-grained key-value cache module can be defined as follows:EQUATIONMulti-grained cross-modal contrast aims to mine multi-grained correlations between audio and category explanations assisted by audio captions. We explore the fusion method of multi-grained audio representation and caption representation, and consider the correlation form of multi-grained fusion representation and diverse category explanation representation, including fine-grained crossmodal contrast and coarse-grained cross-modal contrast.Coarse-grained crossmodal contrast aims to mine coarse-grained similarity between audio and category explanations. We first perform representation interaction between coarse-grained audio representation and caption representation to obtain a semantically enhanced coarse-grained fusion representation. Our motivation is that although captions and audio are presented in different modal forms, they are essentially expressions of audio information. Captions tend to focus on salient concepts but miss out on details, while audio retains the complete message but is interfered with by redundant segments. A natural idea is to encourage cross-modal interaction of captions with audio to learn complementary patterns of both. Specifically, after the coarse-grained audio representation 𝑅 𝑐 𝑎 ∈ R 𝑑 and caption representation 𝑅 𝑡 ∈ R 𝑑 are spliced, linear transformation is used to project them into a new embedding space as a coarse-grained fusion representation 𝑅 𝑐 𝑓 𝑢 ∈ R 𝑑 , which is defined as follows:EQUATIONwhere 𝑊 𝑐 ∈ R 2𝑑 ×𝑑 represents the learnable linear layer. Then, we propose a late weighted fusion method (LW) to guide the matching of coarse-grained fusion representations 𝑅 𝑐 𝑓 𝑢 ∈ R 𝑑 with diverse category explanation representations 𝑅 𝑒 = {𝑒 𝑖 } 𝑛 𝑒 𝑖=1 ∈ R 𝑛 𝑒 ×𝑑 . Our motivation is that different category explanations illustrate different characteristics of audio in this category, but even if the audio belongs to the same category, the significance of different characteristics will be different. Therefore, we propose a late weighted fusion method to guide the model to focus on the correlation between audio and salient discriminative characteristics. Specifically, we consider the cosine similarity 𝑠𝑖𝑚 𝑐 𝑖 between the coarse-grained fusion representation 𝑅 𝑐 𝑓 𝑢 ∈ R 𝑑 and the category explanation representation 𝑒 𝑖 ∈ R 𝑑 , and adaptively obtain the characteristic saliency weight 𝑤 𝑖 according to the correlation between the audio and different characteristics. The coarse-grained similarity 𝑆𝑖𝑚 𝑐 𝐿𝑊 between coarse-grained fusion representation 𝑅 𝑐 𝑓 𝑢 ∈ R 𝑑 and diverse category explanation representation 𝑅 𝑒 = {𝑒 𝑖 } 𝑛 𝑒 𝑖=1 ∈ R 𝑛 𝑒 ×𝑑 can be defined as follows:EQUATION)EQUATION)From this, the matching probability 𝑃𝑟𝑜 𝑐𝑐𝑐 = {𝑃𝑟𝑜 𝑐𝑐𝑐,𝑗 } 𝑁 𝑗=1 ∈ R 1×𝑁 between the test audio and 𝑁 categories can be defined as follows:EQUATIONAmong them, 𝑅 𝑒,𝑗 represents the diverse category explanation representation of the j-th category.3.3.2 Fine-grained cross-modal contrast. Fine-grained cross-modal contrast aims to mine fine-grained similarity between audio and category explanations. We first perform representation interaction between fine-grained audio representation 𝑅 𝑓 𝑎 = {𝑎 𝑖 } 𝑛 𝑎 𝑖=1 ∈ R 𝑛 𝑎 ×𝑑 and caption representation 𝑅 𝑡 ∈ R 𝑑 to obtain semantically enhanced fine-grained fusion representation 𝑅 𝑓 𝑓 𝑢 = {𝑓 𝑖 } 𝑛 𝑎 𝑖=1 ∈ R 𝑛 𝑎 ×𝑑 . Specifically, after each segment representation is spliced with the caption representation, the learnable learnable linear layers 𝑊 𝑓 ∈ R 2𝑑 ×𝑑 ,𝑊 𝑙 ∈ R 𝑑 ×𝑑 are used to project it into a new embedding space as a fine-grained fusion representation 𝑅 𝑓 𝑓 𝑢 , which is defined as follows:EQUATIONHowever, as explained in Sec.1, we noticed that there is a semantic intersection problem between audio and category explanations, that is, some segments in the audio can express multiple semantics at the same time (Fig. 1-b ). Since optimal transportation can establish flexible associations between sampling points of different probability distributions (e.g. one-to-many, many-to-many), we propose to apply optimal transportation (OT) to solve the semantic intersection problem. Specifically, similar to Sec.3.2.2, we first conduct empirical distribution modeling of fine-grained fusion representations𝑅 𝑓 𝑓 𝑢 = {𝑓 𝑖 } 𝑛 𝑎 𝑖=1 ∈ R 𝑛 𝑎 ×𝑑 and diverse category interpretation representations 𝑅 𝑒 = {𝑒 𝑖 } 𝑛 𝑒 𝑖=1 ∈ R 𝑛 𝑒 ×𝑑 .We then facilitate flexible fine-grained correspondences between audio and corresponding category explanations by minimizing the optimal transport (OT) distance between distributions. After an optimization process similar to Eq.8, the fine-grained similarity 𝑆𝑖𝑚 𝑓 𝑂𝑇 between the fine-grained fusion representation 𝑅 𝑓 𝑓 𝑢 and the diverse category explanation representation 𝑅 𝑒 is defined as follows:EQUATIONThe fast optimization matrix solution 𝑇 ′ * 𝑖 𝑗 of the optimal transport plan represents the closeness of the semantic connection from the i-th audio segment to the j-th category explanation. From this, the matching probability 𝑃𝑟𝑜 𝑓 𝑐𝑐 = {𝑃𝑟𝑜 𝑓 𝑐𝑐,𝑗 } 𝑁 𝑗=1 ∈ R 1×𝑁 between the test audio and 𝑁 categories can be defined as follows:EQUATIONIn summary, the overall matching probability of the multi-grained cross-modal contrast module can be defined as follows:EQUATIONThe overall matching similarity is jointly evaluated by the two modules of multi-grained key-value cache and multi-grained crossmodal contrast, and is defined as follows:EQUATIONAmong them, 𝑃𝑟𝑜 𝑡𝑜𝑡𝑎𝑙 ∈ R 1×𝑁 represents the overall matching probability between a piece of test audio and the 𝑁 categories to be classified, and 𝛼 represents the hyperparameter that adjusts the effect of multi-grained key-value cache on the overall matching probability. And we predict the audio category by choosing the category with the highest probability. Cross-entropy loss is used for optimization during training:EQUATIONwhere 𝐵 is the total number of training examples.If the j-th category of the i-th sample corresponds to the true category label, then 𝑦 𝑖,𝑗 = 1, otherwise 𝑦 𝑖,𝑗 = 0. ŷ𝑖,𝑗 represents the matching probability between the i-th audio sample and the j-th category, which can be defined as the j-th element of 𝑃𝑟𝑜 𝑡𝑜𝑡𝑎𝑙 corresponding to the i-th sample. 𝜃 = {𝑊 𝑐 ,𝑊 𝑙 ,𝑊 𝑓 } represents learnable parameters. 
experiments 4.1 datasets: 
 ESC-50 The ESC-50 dataset [30] is a widely used audio recognition dataset. It consists of 2,000 environmental recording clips, each being 5 seconds long, and has 50 semantic category labels. There are 40 sample recordings for each category. For evaluation purposes, the label set is repeatedly sampled into 5 subsets, with 15 label classes in each subset, enabling 5-fold cross-validation, following the setting in Treff-adapter [24] . FSDkaggle18 The FSDkaggle18 dataset [12] contains 11,073 audio clips with durations ranging from 300ms to 30s. Each clip is annotated with one of the 41 labels from the AudioSet Ontology [13] . All audio samples have a corresponding label. Similar to ESC-50, the class label set is repeatedly sampled into 5 subsets, each with 10 label classes, for 5-fold cross-validation, following the data segmentation method used in Treff-adapter [24] .Our framework selects the audio-language model CLAP [10] as the backbone, in which CNN14 [2] is selected as the audio encoder backbone and BERT [7] is selected as the text encoder. Pre-trained weights from CLAP are loaded by the model and frozen during training. For audio data, we adopt the same data preprocessing method as Treff-adapter [24] , and use the pre-trained encoder-decoder model [28] to generate a caption for each audio. For category names, based on our preset multiple prompt templates (e.g. what are the characteristics of the sound of [class]?), we use the api provided by chatgpt to generate 8 category explanations for each category name. During the training phase, the training epochs are set to 20, 𝛽 is set to 5, and 𝛼 is set to 1.1. The optimizer is Adam [20] and the learning rate is set to 10-4. And like previous studies on few-shot audio recognition [24, 43] , we adopt the "N-way K-shot" setting. In this case, 𝑁 categories are randomly selected for classification, and each category contains 𝐾 training samples. During the testing phase, we performed five-fold cross-validation and evaluated the model performance as an accuracy metric. All experiments are performed on a single 48GB A40 GPU based on the pytorch framework.Main Results Tab.1 presents a comparative analysis of few-shot audio recognition results achieved by our proposed framework against other leading methods on the ESC-50 and FSDkaggle18 datasets. Notably, methods leveraging audio-language models (CLAP) generally surpass metric learning techniques [33, 35] , emphasizing the significance of guiding audio-language models to adapt to downstream audio recognition tasks. Among the various CLAP-adapt methods [24, 43] , our approach excels in comparison, delivering superior performance under diverse shot settings. Furthermore, as the number of training samples increases, our method exhibits a commensurate enhancement in performance, highlighting its potential to effectively facilitate the adaptation of audio-language models to downstream audio recognition tasks.In contrast to previous approaches, such as the state-of-the-art Treff-adapter [24] , which relies solely on audio data from limited training sets for coarse-grained learning, our framework offers a more comprehensive and nuanced approach to audio recognition. As a testament to its effectiveness, our framework has demonstrated reliable recognition accuracy across multiple shots settings in the widely used ESC-50 dataset. Notably, we have achieved a significant improvement of 2.26% in recognition performance compared to Treff-adapter. Moreover, in the more challenging FSDkaggle18 dataset, our framework has achieved a remarkable recognition accuracy of 91.74%, surpassing the previous best method by an absolute improvement of 4.19%.Furthermore, it is noteworthy that our proposed method significantly enhances the performance of zero-shot CLAP [10] when guided by a limited amount of training data. Remarkably, we observed absolute improvements of 3.78% and 7.24% on the ESC-50 dataset and FSDkaggle18 dataset respectively. These results clearly demonstrate that our method effectively boosts the adaptability of the audio-language model to downstream audio recognition datasets, making it a highly versatile and effective approach.In this section, we conduct comprehensive ablation experiments to elucidate the impact of each component in our proposed method. All ablation experiments were performed on the FSDkaggle18 dataset. Effect of generated audio captions To delve deeply into the influence of captions on multi-grained correspondence learning, we meticulously crafted three models: a caption-solely dependent model, an audio-solely dependent model, and a model that integrates multi-modal information. As exhibited in Tab.2, it is challenging to attain reliable audio recognition solely relying on information from a single modality. Conversely, the integration of multi-modal information enhances the caption-only model by 10.18% and the audio-only model by 2.81%. This is attributed to the fact that captions and audio present audio information at different levels and in distinct modal forms. Captions tend to concentrate on prominent concepts but overlook finer details, whereas audio preserves the entire message yet is often clouded by redundant segments. The multi-modal representation interaction discussed in Sec.3.3 effectively learns the complementary nature between captions and audio, resulting in a semantically enriched fusion representation. Effect of generated category explanations As shown in Tab.3, we discuss the impact of the number of generated category explanations on model performance and compare the performance of models using category names instead of category explanations. We observe that models that rely solely on category name guidance do not perform well. However, as the number of category explanations gradually increases, the model performance shows an improvement trend. This should be attributed to the fact that rich category explanations illustrate the diverse characteristics of category concepts and effectively capture the potential meaning of category concepts. Considering the quality of text generated by large language models, generating 8 corresponding category explanations for each category is an efficient choice. Effect of multi-grained key-value cache As shown in Tab.4, we examine the impact of the multi-grained key-value cache module and its components on model performance. We observe that the multi-grained key-value cache module achieves an absolute improvement of 1.17%, while the coarse-grained and fine-grained key-value caches achieve an absolute improvement of 0.35% and 0.45% respectively. This is because key-value caches of different granularities capture diverse correlations between audios and promote each other to achieve stronger recognition capabilities. Effect of multi-grained cross-modal contrast As shown in Tab.5, we explored the impact of the multi-grained cross-modal contrast module and its components on model performance. Among them, the multi-grained cross-modal contrast module achieves an absolute improvement of 12.07%, while the coarse-grained/fine-grained cross-modal contrast achieves an absolute improvement of 5.22% and 0.74% respectively. This may be due to the fact that each granularity of cross-modal contrast plays a different role in the audio recognition task, and the different granularities of cross-modal contrast can promote each other.  
conclusion: 
 In this paper, we propose multi-grained correspondence learning for bootstrapping audio-language models to improve audio recognition with few training samples. Multi-modal representation learning assisted by generative modeling is proposed for extracting potential meanings of multi-level audio information and category concepts. Multi-grained key-value cache and multi-grained cross-modal contrast are proposed to establish flexible correspondences between multi-level audio information and diverse discriminative characteristics through optimal transport modeling and late weighted fusion.Extensive experiments validate the effectiveness of our proposed multi-grained correspondence learning. 
 
 Appendix:  


Decision: The authors aim to improve the adaptability of audio-language models for few-shot audio recognition. Although this work is interesting, potential issues should be explained and addressed. More details on the comparison with existing state-of-the-art methods, including potential trade-offs, would be useful. Additionally, more methods that incorporate LLMs for few-shot and zero-shot recognition, such as [1], should be discussed or compared.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel approach to enhance audio recognition capabilities with limited training samples. The core of the methodology lies in multi-grained correspondence learning, which leverages multi-modal representation learning aided by generative models. This approach is designed to extract meaningful insights from various levels of audio information and category concepts. The paper proposes the use of multi-grained key-value cache and multi-grained cross-modal contrast mechanisms to establish flexible correspondences between different levels of audio information and the discriminative characteristics of these sounds. These mechanisms rely on optimal transport modeling and late-weighted fusion for efficiency. To validate the effectiveness of their proposed method, the authors conducted experiments on two benchmark datasets for few-shot audio recognition, ESC-50, and FSDkaggle18, achieving state-of-the-art results.
strength_weakness: strength_weakness: 1. The paper introduces a novel multi-grained correspondence learning framework that effectively leverages multi-modal representation learning and generative models. This innovative approach addresses the challenge of few-shot audio recognition by mining multi-level audio information and category concepts.
2. The technical aspects of the proposed methods, including the multi-grained key-value cache and multi-grained cross-modal contrast, are well-founded. These methods are based on principles including optimal transport modeling and late weighted fusion, which contribute to their effectiveness.
3. The authors provide an empirical evaluation of their approach on two benchmark datasets, ESC-50 and FSDkaggle18. The state-of-the-art results achieved by their method serve as strong evidence of its effectiveness and applicability to few-shot audio recognition tasks.
limitations: limitations: 1. While the paper presents impressive results on two benchmark datasets, the scope of these datasets is somewhat limited(only two benchmark datasets).  Has the proposed approach been evaluated on a broader range of datasets, including those with more diverse audio samples and real-world scenarios?
2. The paper could further address the generalizability of the proposed method across different audio recognition tasks beyond the few-shot scenario. Is the approach able to perform in more generalized settings or when applied to other related tasks?
3. When the paper analyzes the effect of multi-grained key-value cache, it is observed that the absolute improvements brought by the coarse-grained and fine-grained key-value cache modules are relatively small, at 0.35% and 0.45% respectively. Does it suggest that the effect of these modules is limited? Designing complex modules for a limited performance improvement is not suggested. 
4. The few-shot methods are usually sensitive to the hyper-parameters and introduce larger performance variations. Are the results of the proposed methods statistically significant compared to the baselines?
paper_summary: paper_summary: The paper introduces a novel approach to few-shot audio recognition by leveraging multi-grained
correspondence learning. This method aims to improve the adaptability of audio-language models
to audio recognition tasks with limited training samples through generative model-assisted multimodal
representation learning and optimal transport for fine-grained matching.
strength_weakness: strength_weakness: This paper has these strengths:
1) The paper proposes a significant advancement in few-shot audio recognition by focusing on the
intricate patterns between audio and category concepts, the idea is novel.
2) The concept of multi-grained correspondence learning is theoretically sound and provides a
more nuanced approach to capturing the complexities of audio data compared to coarse-grained
methods.
3) The use of optimal transport to address temporal misalignment and semantic intersection is a
clever application of OT, demonstrating both effectiveness and innovation.
4) The paper presents some experimental results to show the effectiveness of the proposed method,
achieving state-of-the-art results on ESC-50 and FSDkaggle18 datasets.
limitations: limitations: 1) The experiment should be conducted on more and larger datasets to show its generality.
2 )While the paper mentions state-of-the-art results, a more detailed comparison with the
existing state-of-the-art methods, including potential trade-offs, would be valuable.
Also, more methods like [1] that incorporate LLM for few-shot and zero-shot recognition
should be discussed or compared in the experiment part.

[1]Liang J, Liu X, Wang W, et al. Acoustic Prompt Tuning: Empowering Large
Language Models with Audition Capabilities[J]. 2023.
paper_summary: paper_summary: To improve few-shot audio recognition, the authors proposed multi-grained correspondence learning for bootstrapping audio-language models. Existing methods often struggle with capturing intricate matching patterns between audio and category concepts. The proposed method addresses this by enriching multimodal representation learning through generative models, establishing multi-grained matching patterns via key-value cache and cross-modal contrast. Additionally, the authors integrate optimal transport to handle temporal misalignment and semantic intersection issues, enabling flexible fine-grained matching. Experimental results on ESC-50 and FSDkaggle18 datasets demonstrate the effectiveness of the proposed approach, achieving state-of-the-art performance.
strength_weakness: strength_weakness: - The authors have a good point of view, recognizing the significant challenges in acoustic signals shown in Figure 1, namely temporal misalignment and semantic intersection. To address these challenges, the proposed method based on multi-grained correspondence learning with optimal transport is very novel.
- From Table 1, the effectiveness of the proposed method is demonstrated by comparing it with state-of-the-art methods using two different datasets.
limitations: limitations: - Although the classification accuracy shows the effectiveness of the proposed method, it does not indicate whether the problems (a) and (b) in Figure 1 have been solved. It also does not indicate the limitations of the proposed method.
- It is not clear what the authors want to show in Table 2. What are "a caption-solely dependent model", "an audio-solely dependent model" and "a model that integrates multi-modal information"?
- The effectiveness of optimal transport should be verified. Couldn't a simpler distance be used?
- In Table 3, what happens if n_e, the number of explanations corresponding to each category, is set to 9 or over?
- Preset prompts should be indicated. It was difficult to imagine what descriptive explanations of category labels generated using the large language model would look like.