Revisiting Deep Audio-Text Retrieval Through the Lens of Transportation
Authors: Manh Luong
Khai Nguyen
Nhat Ho
Gholamreza Haffari
Dinh Phung
Lizhen Qu
Venue: ICLR 2024 poster (2024)
Keywords: Cross-modal
Audio-text Retrieval
Representation Learning
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: The Learning-to-match (LTM) framework proves to be an effective inverse optimal transport approach for learning the underlying ground metric between two sources of data, facilitating subsequent matching. However, the conventional LTM framework faces scalability challenges, necessitating the use of the entire dataset each time the parameters of the ground metric are updated. In adapting LTM to the deep learning context, we introduce the mini-batch Learning-to-match (m-LTM) framework for audio-text retrieval problems. This framework leverages mini-batch subsampling and Mahalanobis-enhanced family of ground metrics. Moreover, to cope with misaligned training data in practice, we propose a variant using partial optimal transport to mitigate the harm of misaligned data pairs in training data. We conduct extensive experiments on audio-text matching problems using three datasets: AudioCaps, Clotho, and ESC-50. Results demonstrate that our proposed method is capable of learning rich and expressive joint embedding space, which achieves SOTA performance. Beyond this, the proposed m-LTM framework is able to close the modality gap across audio and text embedding, which surpasses both triplet and contrastive loss in the zero-shot sound event detection task on the ESC-50 dataset. Notably, our strategy of employing partial optimal transport with m-LTM demonstrates greater noise tolerance than contrastive loss, especially under varying noise ratios in training data on the AudioCaps dataset. Our code is available at https://github.com/v-manhlt3/m-LTM-Audio-Text-Retrieval

Published as a conference paper at ICLR 2024
REVISITING DEEP AUDIO-TEXT RETRIEVAL THROUGH THE LENS OF TRANSPORTATION
Manh Luong', Khai Nguyen?, Nhat Ho², Dinh Phung', Gholamreza Haffari', Lizhen Qu' 'Monash University. Australia, 2 University of Texas at Austin, USA {tien. luong, dinh phung, gho hollazzzza haffari, Pefi,,izzie lizhen qujemonash.edu [khainb, minhnhat @utexas.... edu
ABSTRACT
The Learning-to-nen mo-tnchh (LTM) framework proves to be an effective inverse opti- mal transport approach for learning the underlying ground metric between two sources of data, facilitating subsequent matching. However, the conventional LTM framework faces scalability challenges, necessitating the of the entire dataset each time the parameters of the ground metric are updated. In adapting LTM to the deep learning context, introduce the mini-batch I searnnng-to-maan framework for audio-text retrieval problems. This framework leverages mini-batch family of ground metrics. Moreover, cope with misaligned training data in practice, we propose variant using partial optimal transport to mitigate the harm of misaligned data pairs training data. We conduct extensive experiments on audio-text matching problems using three datasets: AudioCaps, Clotho, and ESC-50. Results demonstrate that our proposed method is capable of learning rich and expressive joint embedding space, which achieves SOTA performance. Beyond this, the proposed m-LTM framework able close the modality gap across audio and text embedding, which surpasses both triplct and contrastive loss the zero-shot sound event detection task on the ESC-50 dataset. Notably, our strategy of employing partial optimal transport with m-L.TM demonstrates greater noise tolerance than contrastive loss, especially under varying noise ratios in training data the AudioCaps dataset. Our code is available at http:://github.cco/ waapmm/-mmaaiitmmmmm------ Texttt-t---eettttiiiiiallll
(m-LTMM
subsampling and Kahalanobis-enhaneee
INTRODUCTION
Audio-text matching is crucial task that has tremendous applications. such as audio-text search- ing (Elizalde et al. 2019 Koepke et al. 2022), audio captioning (Kim et al.) 2019), and text-to-audio generation (Liu al. 2023). ). The mainstream approaches for audio text matching involve the con- struction of a shared embedding space that encompasses both audio and text modalities. The shared embedding space should ideally be endowed with richness and expressiveness. The primary objective of learning a joint audio-text embedding space is to establish a seamless alignment between all text captions and their corresponding audio counterparts. Moreover, it is crucial that this embedding space should minimize the modality gap between audio and text embeddings, thereby enhancing its transferability downstream tasks (Liang et al. 2022)
The widely adopted approaches for learning the joint embedding space are contrastive learning (Jia al. 202T, Radford et al. 2021, Mei et al., 2022); Wu et al., (022b) and triplet loss (Wei al. 2021). Both methods have shown their success in learning the joint embedding space cross modalities. Even though they are effective in training cross-modal matching models, these methods only aim to minimize the ground metric, either Euclidean distance or cosine similarity, for an anchor point at a time and push away all negative samples. They treat all negative samples equally, therefore, they might ignore the geometric properties of the joint embedding space. As consequence, suboptimal metric may make the relative distances between a set of positively aligned embeddings similar to those between misaligned embeddings. Furthermore, (Liang al., 2022) showed that contrastive learning preserves the modality gap in the joint embedding space. Therefore, contrastive learning hinders the transferability of the joint embedding space downstream tasks.

---

Published as a conference paper at ICLR 2024
An additional hurdle encountered in the process of acquiring a joint embedding space across modalities is the presence of noisy data, which is prevalent in audio. The noisy data for multimodal matching is misaligned data pairs due to the data collection process. It is desirable that the examples in a training dataset for multimodal matching are precisely aligned across modalities. If the multimodal training data is collected from the Internet (Tia et al., 2021 [Sharma et al. 2018), it is unavoidable to include substantial noisy correspondence pairs as ground truth. To address this issue, (Huang et al. 2002I) introduces the Noise Correspondence Rectifier approach for image-text matching. However, this technique heavily depends on a particular choice of neural network architectures, therefore, it is less amenable to straightforward adaptation with alternative tasks.
To deal with the aforementioned issues, we propose the minibatch Learning-tee to-Match (m-LTM) framework to learn the joint embedding space across audio and text modalities through the lens of optimal transport. Specifically, we formulate the cross-modal matching problem as solving an optimal transport problem with entropic regularization in the embedding space to match the point sets between audios and texts. We derive a scalable solution to the problem based mini-batches to ensure that it scales up to large audio-text datasets. The resulting optimal plan is an implicit ranking matrix in which each row points out the best match between two modalities and ranks all negative points from one modality to the other. To further reduce the modality gap between audio and text embeddings, we replace the widely used Euclidean distance with Mahalanobis distance (Dhouib) et al.] 2020) as the ground metric, which is introduced for the first time for audio-text matching. To mitigate the prevalent alignment noises in the training data for the first time for audio-text matching, we address the problem by using Partial Optimal Transport (POT) (Chapel & Alaya 2020). To sum up, our key contributions are three-fold:
1. We adapt the learning-to-nn -o-ring framework (Li et al. 2018) to the deep learning setting based on subsampling training procedure for audio-text matching, which leads to the mini-batch learning-to-nen to-match framework. We leverage the soft-matching of the entropic optimal plan and Mahalanobis distance as ground metric to learn shared embedding space. Our approach facilitates the acquisition of a shared embedding space that is rich and expressive, thus we acquire SOTA performance in audio-text retrieval tasks on AudioCaps and Clotho datasets.
2. The mini-batch LTM framework is capable of bridging the modality gap. therefore, is more transferable downstream tasks than triplet and contrastive learning. We evaluate the modality gap across audio and text embedding on three datasets: AudioCaps, Clotho, and ESC50. We also carry out zeroshot sound event detection ESC50 dataset illustrate the transferability of the M- framework.
3. We propose a variant of the m-LTM framework which utilizes Partial Optimal Transport (POT) to deal with noisy training data circumstances. Our experiments demonstrate that the minibatch LTM with POT is more noise-tolerance than triplet and contrastive loss on AudioCaps dataset.
2 PRELIMINARY
2.1 DEEP AUDIO-TEXT TEXT RETRIEVAL.
Audio-text retrieval is to learn the cross- cross-modality alignment between audio and text captions. The aligmment can be achieved by lcarning a joint embedding space across audio and text modalities. Recently, contrastive learning has been the most effective learning method for learning expressive
cross-modal embedding space (Jia al. 2021 Wu et al. 2022b Radford et al. (021).
Training. Denote D {x;, y:) all pairs of audio and caption in the training dataset. The goal is to learn a joint embedding space between audio and text embedding. Given an audio encoder fo(.) and a text encoder g+)-), the objective function for learning the embedding space is:
cxp(s(fo(ts)
min Exxt, 0,0
log
km cxp(s((f(2z)
b
exp(s(fs(t3)
log
Eh= (xp((((f((a)
2

---

Published as a conference paper at ICLR 2024
where (Xx,, y* is a minibatch of b audio-caption pairs drawn from training data D, T is the temperature hyperparameter, and s(..-) is the cosine similarity between audio and text embedding
Retrieval. Given n' audio test = {m;) Yr'n and m' caption Yest = {y, 1 the ranking matrix R( R(Xtest, Yesst) between two sets of audio and caption can be retrieved by computing pairwise cosine similarity between audio and text embedding. For example, the correspondence caption of given audio 24 is
9 arg M1X exp(s(fo(z)).
My M Lext k= exp(s(fo(zz))
2.2 LEARNING TO MATCH
Entropic optimal transport. Given two empirical probability measures Px = 1 Ei= 82, and Py # ش m 6y, where X for i = I,..., and y/j E y for j 1,..., m, the entropic optimal transport between Px and Py (Cuturi 2013) with the ground metric c: X x - R+ is defined as:
X.Y = arg min
Tidllz Uy) -
Tay log Ttj.
=EII(x.PP)) 2=1
1
where Pv) ( [ = 1n/n,, x² 1, = m denotes the set of transportation plans or couplings between Px and Py. When e 0, the optimization problem Equation 3 can be solved by using the differentiable Sinkorn algorithm (Marshall & Olkin 1968).
Learning match. In practice, we are given 9:)]== which is all pairs of audio and caption. Therefore, we would like to infer the underlying ground cost metric ca U5) (parameterized by 6). Learning-to soonnggtt-mne to-match (LTM) (Li et al. 2019) is one of the most effective solutions for doing such inference. The LTM framework considers the following optimization:
inf cEC
where C is set of ground metrics, X denotes the set of {I1...., .n } and Y denotes the set of ( ). and म matrix of size x n such that Ru 1/n and म.j 0 (i + j) for i == and I....,
Retrieval. Given n' audio Xiest = ( m' caption Yrest {u, Jyh and the optimized ground the ranking matrix R(Xeest, Ytest) between two sets of audio and text caption is the optimal solution of the Equation, between Px... Σn 6z,, where 2; E Xtest mi Σø= 8y,, where y, Ytest. For example, the correspondence caption of given
metric e from the Equation. 4
and PY 1 audio n44 is
i arg max TT.c
where 21.30 is the coupling value of i-th audio and j-th text from the optimal plan. The same retrieval procedure can be done for text-to-audio retrieval.
3 DEEP AUDIO-TEXT MATCHING VIA MINI-BATCH LEARNING TO MATCH
This section introduces our proposed method, the L-LM framework, for the audio-text matching task and variant to deal with noisy correspondence training data. Our model consists of the audio encoder (-).,, J the text encoder (-). and the interaction matrix M. Our proposed framework utilizes the soft-matching from entropic optimal transport and incorporates the Mahanalobis distance ground metric to learn flexible ground metric to solve audio-text matching problems.
3.1 MINI-BATCH LEARNING TO MATCH
By abuse of notation, we denote (X*, y²) ~ D as sampling uniformly a mini- which contains b pairs of audio and caption. The mini-batch learning-to-mate framework is defined follows:
3

---

Published as a conference paper at ICLR 2024
Given a paired training data D {(2,, y:) 3:))-n the mini-batch sarming-to-matte (m-LTM) framework solves the following optimization problem:
Definition 1.
inf E ((x),r")=
I| x" y
(6)
where b 2 is the mini-batch size, Tee xx is the optimal transportation plan from Equation 3 Swtt the corresponding empirical measures over mini-batches X and Y and it. matrix of size x such that tH 1/b and Rg = for .,...,b and j = L.... b.
Neural network parameterization. As in the preliminary section, we parameterize the ground metric çg y) di ( (V;))) where fe X 2 (0 E +) and Is - 2 (Ø +) two neural networks parameterized by 0 and ó respectively, and d Z x 2 - R is chosen metric e.g., Lg norm. Therefore, the set of ground metric becomes the of all possible weights of two neural networks i.e., θ x Φ. As result, Equation is equivalent to:
E E(xerr) (0.0)
min
E,c0,s
Stochastic gradient estimation. To solve the optimization in Equation 7 we perform stochastic gradient descent algorithm to update and Φ. In particular, we sample B mini-batches (X*, y"), (Xb, )O)B hid D, then form the following stochastic estimation of gradients:
1 B
E
V (0
[| (X")""" ())).
(X"
Since this estimation is unbiased, the stochastic optimization procedure can converge to local minima. In practice, we often set B for having a computationally fast estimation.
Retrieval. During the inference phase, it is noteworthy that the quantity of audio and caption data considerably smaller compared to the extensive training dataset. Moreover, the conventional evaluation protocol for retrieval necessitates that each query must be compared to the entirety of available items in the test set to compute the evaluation metric. Consequently, employing the minibatch configuration during the retrieval phase does not yield discernible advantages. For more detailed information on retrieval, please refer to Section 2.2
3.2
MAHALANOBIS-ENHANE
GROUND METRIC
Although the parameterization of the ground metric in Section 3.1 i.e., ca y)
d(fa(z;), (3;))) has been widely used in practice, it still has a limitation. In particular, the embedding from the two encoders (.) and (.).) might not be well aligned in terms of scaling in each dimension. To partially address the issues, we propose a more generalized family of ground metric which is based on the Mahalanobis distance.
Definition 2. Given two encoder functions fo X + Z and g6 y + Z, metric d Z Z Rt. the Mahalanobis enhanced ground metric is defined as:
ca.0,M(Z, (2, y) = (fo(=)) (o(1))) ( (fe(zz) go(3)));
for 0 Q and & which are spaces of parameters and M is a positive definite matrix.
Mini-batch learning to match with Mahalanobis- Ground Metric. By using the family of Sewallanoobi-EEhannnee ground metries in Definition 2, the m-LTM objective in Definition becomes:
(8.6. Exxt,">> KL(
min
(10)
where M is the set of all possible positive definite matrices e.g., x: Mx > for all x E Z.
Hybrid stochastic gradient descent. the optimization problem in Equation I0 consists of three parameters 8, o. and M. In contrast to 0 and @ which are unconstrained, M a constrained parameter. Therefore, we propose to use hybrid stochastic gradient descent algorithm. In particular, still
4

---

Published as a conference paper at ICLR 2024
update 0, & using the estimated gradients in Equation However, update M using the projected gradient descent update (Rockafellar 1997). We first estimate the stochastic gradient with respect to M:
B E B
V V KL [ ) =
1
V MM n (10))) ccc,,..
(11)
After that, we update M = (rM VM)) where F(M, VM) denotes the one-step update from a chosen optimization scheme e.g.. Adam, gradient descent, and so on, and Proj(-) de- notes the projecting function that maps a unconstrained matrix to the space of positive def- inite matrix M. In greater detail, Proj(A) Udiaa(S) with U,S.V (VD(A) and = diag(max ai ) max (0, Jn We refer the reader to the training algorithm in the Algo- rithm. iin the Appendix A.2
3.3 PARTIAL OT FOR NOISY CORRESPONDENCE
Noisy correspondence setting. We first introduce the noisy correspondence setting for the audio- text matching task. That is, given the training data D { 31));= where N is the number of training samples, proportion of training data Neor, NNoo < N. is corrupted, for instance, due to the data collection process. The amount of data corruption is unknown, and we do not know which samples corrupt, thus presenting challenging learning scenario. Due the noisy correspondence phenomenon, the empirical matching Å is now corrupted to an incomplete matching म, in which there is no matching in some rows. Assuming an absence of noise in the original training data, introduce a noisy correspondence training dataset by randomly shuffling a portion of audio-text pairs within the dataset. We denote a random variable 2 E {0,1} which is sampled from binomial distribution Binomial(NV N= z 1 indicates the audio-text pair is shuffled. The training data is now D = {(z,, Fi, y:) where 2; Binomial Nx
Given the absence of knowledge regarding the nature of data corruption, the training process relies solely on the empirical matching Åt The training objective is to infer the incomplete matching म from the noisy training dataset D. However, it is important to note that the m-LTM framework may encounter challenges when attempting recover the incomplete matching म. This difficulty arises due to the constraint imposed by transportation preservation, which dictates that each row must be associated with at least one column.
To solve the aforementioned issue, we propose to use Partial OT, which relaxes the transportation preservation constraint, to mitigate the harmfulness of noisy empirical matching for approximating the incomplete matching म. The objective function 10 is rewritten as
E (X* (L 8h
min (0,, M
(12)
where (X")Y*) minibatch sampled from noisy training data D, and n is the optimal solution of the equation
xh."
8".80 Ns,eico.r.11
arg min (P
ΣΣ i=1
Tiy myylIi (acc v,)
Ti log Tijs
(13)
1=1
where IT,(Px xx. Ppp) fTr E | S optimization can be solved by using Bregman projection (Benamou et al.) 2015). Intuitively, the partial transportation plan should be such that true correspondence audio-text pairs within
TI < Pro. TT = 8). The partial OT
minibatch are mapped to each other, while noisy correspondence audio-text pairs are discarded from the transportation plan. We conducted experiment to examine the relationship between the transportation mass and noise ratio in training data in the Section. 5.2
4 RELATED WORKS
Cross-modal Matching, Cross-modal matching is to learn a shared representation space (Wang et al. 2013) between two modalities like image-text (Jia et al. 2021 Radford et al.) 2021 Wei et al.
5

---

Published as a conference paper at ICLR 2024
2020) or audio-text (Wu et al. 2022b Deshmukh et al. 2022 Oncescu al. 2021 Mei et al.
2022).
The most popular approach for gross-modie matching is metric learning, such as triplet loss (Wei
et al. 2020) and contrastive loss (Radford et al.] 2021: Yang et al. 2022). The cross-modie training procedure exposes great benefit of the transferability to downstream tasks (Zeng et al. 2022). (Xu be used train a single model for both text and video understanding tasks without fine-tuning, However, cross-modal training beneficial for downstream tasks, this training procedure requires huge amount of data with high alignment quality. (Huang et al., (021) introduces noisy correspondence training circumstances for image-text matching. This work demonstrates that their proposed method capable of dealing with high-level image-caption mismatching, but it heavily depends on neural network architecture. To handle the noisy correspondence training data, we introduce a new loss function that not only capable of mitigating the harmfulness of label mismatch for cross-modal matching but also easy to incorporate into existing backbones for cross-modal matching. Furthermore, we apply a new ground metric, Mahalanobis distance, which is able to learn a rich and expressive joint embedding space across audio-text modalities and close the gap between audio and text embedding space.
et al. (02T)) shows that contrastive cross-modal pretraining procedure
Optimal transport. Optimal transport (OT) is powerful mathematical tool to measure the discrep- ancy between two probability distributions. Therefore, has a wide range of applications. such as domain adaption (Courty et al.] (2017), generative models (Genevay al. 2018 Nguyen & Ho 2024
Nguyen et al. 2024). and representation learning (Chen et al. (020). Our work is close to the Inverse
Optimal Transport (IOT) problem, which has been well-studied (Dupuy et al. 2016 Li al. 2018
Stuart & Wolfram 2019: Chiu et al., 2022) by the research community aiming to learn matching between two sources of interest. (Wu et al.) 2022a) proposed the OTTER framework which leverages
entropic optimal plan as soft matching labels to learn many- maan-to--mnyy many relationships in the image-text matching task based on contrastive learning procedure. (Shi et al.) (023) proposed unifying under- standing for contrastive learning from an inverse optimal transport view. All previous works either adapt cosine similarity as the ground metric (Shi et al. (023) or study the whole empirical plan of tabular data for learning-to- sermiing-to-matchin mothhing (Dupuy et al.) [2016]] Li et al. 2018 [Stuart& & Wolfram 2019).
Mini-batoh optimal transport has been investigated in Fatras et al. (2020); Nguyen et al. (2022b a). In
paper, we adapt the learningtto-mate framework (Li et al. 2018) for the minibatch setting with more flexible ground cost metric, Mahalanobis distance, to model the expressive shared embedding space between audio and text modalities. Mahalanobis distance represents an infinite set of cost functions in the theoretical analysis of robust optimal transport optimization (Paty& & Cuturi] 2019 Dhouib et al. 2020). However, there is a gap between theory and application of Mahalanobis distance as ground metric for optimal transport optimization due to the difficult constraint of Mahalanobis distance. We close the gap by successfully applying Mahalanobis distance for the audio-text matching application. The strict constraint for learning Mahalanobis distance is relaxed by Projection Gradient Descent.
EXPERIMENTS
In this section, we design experiments examine the effectiveness of our framework in order to learn a joint embedding space across audio and text modalities. We conduct the audio-text retrieval task on two datasets: AudioCaps (Kim et al.) (2019) and Clotho (Drossos et al. 2019) to answer the question of whether our framework is able to learn a rich and expressive joint embedding space. In addition, we evaluate the modality gap metric (Liang et al. 2022) in the joint embedding space on three datasets: AudioCaps, Clotho, and ESC-50 (Piczak 2015) to illustrate that the m-LTM framework is capable of bridging the modality gap between audio and text embedding space. We perform zero-short sound event detection on ESC-50 test set to demonstrate the transferabilityy of our method. We verify the noise-tolerance capability of the M- with POT framework in a variant of noise ratio training data on the AudioCaps dataset. Finally, we conduct ablation studies for hyperparameters tuning. The implementation details of our framework are provided in Appendix. A.
Baselines. We compare against all state-of-the-ar models Mei et al. 2022 Deshmukh et al. 2022 Oncescu et al. 2021 Wu et al., 2022b) for audio-text retrieval tasks to illustrate the remarkable performance of our framework. All the results for these baselines are copied from the reference papers. Furthermore, we examine our framework with diverse learning objectives, such as contrastive and triplet loss.

---

Published as a conference paper at ICLR 2024
Table 1: The comparison of m-LTM framework with baselines audio-text retrieval task benchmark datasets, AudioCaps and Clotho dataset.
Text- >-Audio
Audio-> >> Text
Dataset
Method
R@1
R@5
R 10
R@1
R@5
R 10
Oncescu
2021
28.1
33.7
83.7
eei 2022
33.9
39.4
Audiocups
Deshmukk et
2022)
33.07
67.30
80.3
39,76
73.72
84.64
Wi ml. 2022h
36.7
70.9
83.2
45.3
78
87.7
m: (ourr
39.10
74.06
85.78
49.94
80.77
90.49
Oncescu et al
2021
9.6
40.1
10.7
40.8
Mci ct al. 2022)
14.4
36.6
49.9
16.2
37.5
50.2
Clotho
Deshmukh et
2022)
15.79
36,78
49.93
17.42
40.57
54.26
Wi 2022h
12.0
31.6
43.9
15.7
36,9
51.3
M-LlT((uur)
16.65
39.78
52.84
22.1
44.4
56.74
Table 2: Experiments of loss functions on the AudioCaps dataset. There are two audio encoders, ResNet38 and HTSAT, used to extract audio embedding, and the text encoder is the BERT text encoder.
Text-> Audio
Audio-> Text
Loss
R@1
R@55
Rd 10
R@1
R@5
R 10
Triplet
32.2
68.2
81.6
36.1
69.2
81.4
ResNei38 Audio
Contrastive
33.9
82.6
39.4
72.0
83,9
Encuder
m-LTM(our)
39.10
74.06
85.78
49.94
80.77
Triplet
33.87
69.91
83.23
33.95
70.03
HTSAT Audio
Contrastive
34.38
71.16
84.64
35.84
71.06
83.39
Encoder
m-I.TM(our)
40.02
75.86
86.42
40.13
74.09
83.89
Evaluation metrics. To evaluate our proposed framework, we use the recall at rank k(tkk) metric, which is the standard evaluation metric for cross-modal retrieval tasks. R@K reports the percentage of the ground- -trth retrieval within the top-k ranks, thus, higher is better. R@ 1, RR05, and R@10 10 are shown to compare performance among baselines and our method. We also report the modality gap metric (Liang et al. 2022) to examine the shared embedding space. The lower modality gap is better for downstream tasks.
5.1 EXPRESSIVENESS AND TRANSPERABILITY OF THE JOINT EMBEDDING SPACE
We compare proposed framework with state-of-the-arr models in the audio-caption retrieval task to verify the expressiveness of our proposed framework. To have a fair comparison, we train all the models and objective functions with the same amount of training data. As shown in Table. our minibatch beariingto-maale framework outperforms four baseline methods in terms of R&1, R@5, and R@ 10 evaluation metrics on two datasets in two retrieval tasks, audio-to-text and text-to-audio retrievals. Furthermore, we carry out the experiment to compare the L- objective function with contrastive and triplet loss on the AudioCaps dataset shown in Table 2 Our method illustrates the highest performance for cross moosll retrieval tasks, there is significant gap regarding the R@1 score compared with the best loss function, contrastive loss. For the ResNet audio encoder, the utilization of L-LMM results in a marked and notable augmentation of the R@l score, showcasing an improvement from 33.9% 37.97% for the text-to-audio matching task, and a distinct escalation from 39, 39.4% to 47.43% for the audio-to-text matching task. This performance increase is also discernible in the case of the HTSAT audio encoder. We also provide some qualitative results for audio-text retrieval tasks in Appendix A.3
Table 3: The zero-shot sound event detec- tion on the ESC50 test set, the R11 score is equivalent accuracy.
Table 4: The modality gap between audio and text embedding in the shared embedding space. Lower is better for downstrcam tasks.
Audio
Sound
Modality gap(]] A ||)
Loss
Loss
R@5
R@
mAP
AudioCapp aps
Clotbo
ESC50
Triplet
71.25
91.75
95.75
80.09
Triplet
0.149
0.283
0.937
Contrastive
0.181
0.266
0.922
97.0
87.57
0.117
0.142

---

Published as a conference paper at ICLR 2024
shared bedding
shared aribedding space
1.5
1.0-
0.5
0.6
0.4
0.4
0.2
0.
0.0 0.4
0.6
14
0.0
0.6
1.0
(a) Contrastive Learning
(b) Minibatch learning-to-matti
The visualization of the shared embedding space between audio and text embedding on the ESC50 test set based on tSNE algorithm. Each text embedding represents label in the test set, and each audio embedding represents an audio the test set.
Figure 1:
Table 5: The performance of learning to- to-match and metric learning methods audio-ter text retrieval task under the variant ratio of noisy training data.
Text>>
Audio->> Text
Noise
Method
R
R 10
R
Triplet
23.01
54.98
69.98
28.52
58.09
70.
Contrastive loss
31.34
67.73
81.27
40.12
70.84
82.54
20%
m LTM
35.51
71.32
84.01
46.64
78.68
87.87
M-LTM with POT
35.92
72.28
84.11
47.12
79.2
88.19
Triplct loss
0.1
1.19
2.75
1,25
5,43
9.4
Contrastive loss
26,68
62.98
78.18
34.69
66.66
78.99
40%
m-I.TM
32.58
67.75
80.89
40.31
71,16
84.57
m-L.M with POT
33.64
69.23
82.27
42.63
73.35
86.1
Triplet
0.52
1.06
0.52
1.46
Contrastive
20.58
53.96
70.72
27.37
58.72
75.21
60%
m LTM
25.26
59.72
75.03
34.08
66.77
79.62
m-LTM with POT
27.73
62.61
76.17
35.42
68.65
80.56
We conduct zero-shot sound event detection experiment the ESC50 test set to examine the transfcrability of the learned embedding space. We replace the task of sound event detection with audio-text retrieval, all the classes in the test set converted to the template caption "This is sound of {class}". We pretrain all models on the AudioCaps dataset on the audio-caption matching task. We report R@1, R@5, R@10, and mAP scores for triplet loss, contrastive loss, and the LTM loss in Table. The [eemm-LMM]oos acquires the highest R&1 score, equivalent to achieving the highest accuracy for the sound event detection task. Also, there is large performance gap in terms of R@5, R@ 10, and mAP metrics between In-LTMloss and contrastive loss. As depicted in Figure. []] audio embeddings learned from the Ln-LLMMloss are clearly clustered by the text sound event classes, which means the learned embedding space is more expressive for the zero-shot sound event detection setting. In contrast, audio embeddings learned from contrastive loss are spread uniformly in the joint embedding space. This observation consolidates to the result in Table. 3 which is the LT is more transferable than contrastive loss for zero-shot setting. Furthermore, we study the modality gap of our proposed loss in Table. the Agup modality gap metric between two modalities is computed as the mean of embedding between two modalities A Agnp = 1 fxa) EE ((v)). Our proposed loss achieves the smallest gap between audio and text embedding, which means it is more transferable to downstream tasks than triplet and contrastive loss.
5.2 NOISY CORRESPONDENCE TOLERANCE
To study the robustness of our proposed framework with noisy correspondence, we carry out the experiment under varying noisy correspondence training data. As defined in the section [33 we use three different noise ratios {0.2, 0.4, 0.6) of training data. We perform the experiment AudioCaps dataset for four losses: triplet loss, contrastive loss, m- mn-LTM]ooss, and m- with POT loss described in section 3.3 First, the binary value z; is sampled from Binomial (N, distribution
the
8

---

Published as a conference paper at ICLR 2024
for each audio-text pair in the training data, we have N tuples of training data (2,, ys f If 2 = 1 we replace the text y with a random text ys in training data, the noisy training tuple is {2, = 1,, 2,, J. The random variable
Binomial([N, E) is only used for data sampling, and it is not used during training. Table. shows the audio-text retrieval performance under the variant noise ratio of four loss functions: triplet loss, contrastive loss, and our proposed methods. The triplet loss is less robust to high levels of noise, its performance plunges from 28.5 28.52% to 1.25% and from 23. 23.01% 0.1% in terms of ft@l score when the noise ratio increases from 20% 40% for audio-to-ted and text-to-audio retrieval, respectively. Contrastive loss and two variants of our proposed method are robust to a wide range of noise ratios, however, our proposed losses surpass contrastive loss for all levels of noise in terms of recall at rank k metrics. There is significant gap between contrastive loss and the [m with POT loss on the R&1 metric for two retrieval tasks. At 60% noise of training data, the m-LTM with POT loss boosts the performance from 20.58% and 27.3 37% of contrastive loss to 27. and 35.42% in terms of R&1 metric for text-to-audio and audio-to-text, respectively. Furthermore, it is noteworthy that in both of the two retrieval tasks, the [T-LTM with POT consistently outperforms the conventional MT across all levels of noise. As result, it can be inferred that the MTTM with POT exhibits state-of of-e-of-theearr (SOTA) performance in effectively addressing challenging scenarios characterized by noisy correspondences.
5.3 ABLATION STUDY
Table 6: Ablation studies on AudioCaps dataset for choosing the ground metric and hyper- parameter for minibatch learning-to- to- match loss function.
no6s000.2 nose-0.4
Avgrage.
Text > Audio
Audio >> Text
36
R@1
R@5
R 10
R@1
R@5
R 10
0.05
39.10
74.06
85.78
49.94
80.77
90.49
24
Moha
0,1
29.86
65.51
79.64
39.81
70.53
83.07
32
0.5
9.59
31.99
48.19
10.55
34.16
46.91
0.6
0.7
0.0
0.9
73.47
85.43
48.15
79.10
89.23
Eacli
0.1
29.55
60
79.43
38.45
70.84
82.23
Figure 2: Ablation study on the Au- dioCaps dataset for transportation mass of m-LTM with POT.
0.5
10.17
33.06
48.50
13.06
34.69
48.27
In this section, we conduct two ablation studies. As illustrated in Table, 6 Mahalanobis distance as the ground metric for learning- mothh framework surpasses the Euclidean distance for audio-text retrieval task. Also, the c hyper-paramette is crucial in our framework. If e is too high, the optimal transportation plan will be uniform distribution and lead decrease in performance. The value of e = 0.05 acquires the highest performance, thus, all the reported results in this paper using 0.05. The second ablation study is the effect of transportation mass for noisy correspondence training data demonstrated in Figure. 2 we report the average H@ 1 score for both audio-to-tex. textext and text-to-audio retrieval tasks. For different noise ratios, there is best transportation mass value, which corresponds to noise level, to acquire the highest performance. This observation supports our hypothesis in section. 3.3 which is choosing the appropriate transportation mass for noisy correspondence circumstances essential
CONCLUSION
In this paper, we introduce the mini-batch learning-to-men -aath (m-LTM) framework for audio-text text matching, the m-LTM utilizes the Mahalanobis distance and soft-matching from the entropic optimal plan to facilitate learning of a shared embedding space for audio and text modalities. By leveraging soft matching from the optimal plan, our framework is capable of learning rich and expressive joint embedding space across audio and text modality, therefore, the m-LTM acquires -tate-of-the-art performance for audio-text retrieval on AudioCaps and Clotho datasets. Furthermore, we adopt the Mahalobis distance as the ground metric to reduce the modality gap, thus, our framework is more transferable to downstream tasks compared to conventional methods relying triplet and contrastive loss functions. Finally, we propose a variant of the m m-LTM with POT to alleviate the adverse effects of noisy correspondence in training data under various noise conditions.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
NH acknowledges support from the NSF IFML 2019844 and the NSF AI Institute for Foundations of Machine Learning.
REFERENCES
Jean-David Benamou, Guillaume Carlier, Marco Cuturi, Luca Nenna, and Gabriel Peyré. Itera- tive bregman projections for regularized transportation problems. SIAM Journal on Scientific Computing, 37(2)AA111-AII 138, 2015.
Lactitia Chapel and Mokhtar Z. Alaya. Partial optimal tranport with applications on positive-unlabeled learning. arXiv: Machine Learning, 2020.
Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection. ICASSP 2022 - 2022 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 646-650, 2022.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference computer vision, pp. 104-120. Springer, 2020.
Wei-Ting Chiu, Pei Wang, and Patrick Shafto. Discrete probabilistic inverse optimal transport. In International Conference on Machine Learning, pp. 3925-3946. PMLR, 2022.
Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation, Advances in neural information processing systems, 30, 2017.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NIPS, 2013.
Soham Deshmukh, Benjamin Elizalde, and Huaming Wang. Audio retrieval with wavtext5k and clap training. ArXiv, abs//209. 14275, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Sofien Dhouib, levgen Redko, Tanguy Kerdoneuff, Rémi Emonet, and Mare Sebban. A swiss army knife for minimax optimal transport. In International Conference on Machine Learning, pp. 2504-2513, PMLR, 2020.
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: an audio captioning dataset. ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 736-740, 2019.
Amaud Dupuy, Alfred Galichon, and Yifei Sun. Estimating matching affinity matrix under low-rank constraints. Econometric Modeling: Theoretical Issues in Microeconometrics eJournal, 2016.
Benjamin Elizalde, Shuayb Zarar, and Bhiksha Raj. Cross modal audio search and retrieval with joint embeddings based on text and audio. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 4095-4099. IEEE, 2019.
Kilian Fatras, Younes Zine, Rémi Flamary, Remi Gribonval, and Nicolas Courty. Learning with minibatch wasserstein: asymptotic and gradient properties. International Conference on Artificial Intelligence and Statistics. pp.
PMLR, 2020.
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and cuman--lablled dataset for audio events. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776-780, 2017.
10

---

Published as a conference paper at ICLR 2024
Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with sinkhorn di- vergences. In International Conference on Artificial Intelligence and Statistics, pp. 1608-1617. PMLR. 2018.
Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, Hua Wu, and Xi Peng. Learning with noisy correspondence for cross-modal matching. Advances in Neural Information Processing Systems, 34:29406-2944 29419,, 2021.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision- lsionnlanguuae representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In North American Chapter of the Association for Computational Linguistics, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
A Sophia Koepke, Andreea-Marin Oncescu, Joao Henriques, Zeynep Akata, and Samuel Albanie. Au- dio retrieval with natural language queries: benchmark study. IEEE Transactions on Multimedia, 2022.
Qiugiang Kong, Yin Cao, Turab lqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. /EEE/ACM Transactions Audio, Speech, and Language Processing, 28:2880-222 2894, 2019.
Ruilin Li, Xiaojing Ye, Haomin Zhou, and Hongyuan Zha. Learning match via inverse optimal transport. J. Mach. Learn. Res., 20:30:1- 2018.
Ruilin Li, Xiaojing Ye, Haomin Zhou, and Hongyuan Zha. Learning to match via inverse optimal transport. Journal of machine learning research. 20, 2019.
Weixin Liang. Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. ArXiv, abs/2203.02053, 2022.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12500 2023.
Albert W. Marshall and Ingram Olkin. Scaling of matrices to achieve specified row and column sums. Numerische Mathematik, 12:83-90, 1968.
Xinhao Mei, Xubo Liu, Jianyuan Sun, MarkD , Plumbley, and Wenwu Wang. On metric learning for audio-text cross-modal retrieval. In Interspeech, 2022.
Khai Nguyen and Nhat Ho. Sliced Wasserstein estimation with control variates. In The Twelfth International Conference on Learning Representations, 2024.
Khai Nguyen, Dang Nguyen, Quoc Dinh Nguyen, Tung Pham, Hung Bui, Dinh Phung, Trung Le, and Nhat Ho. On transportation of mini-batches: A hierarchical approach. In International Conference on Machine Learning, pp. 16622-16655. PMLR, 2022a.
Khai Nguyen, Dang Nguyen, Tung Pham, Nhat Ho, et al. Improving mini-batch optimal transport via partial transportation. In International Conference on Machine Learning, pp. 16656-16690. PMLR, 2022b.
Khai Nguyen, Shujian Zhang, Tam Le, and Nhat Ho. Sliced wasserstein with candom-pati projecting directions. arXiv preprint arXiv:2401.75880 2024.
Andreca-Maria Oncescu, A. Sophia Koepke, João F. Henriques, Zeynep Akata, and Samuel Albanie. Audio retrieval with natural language queries. In Interspeech, 2021.
11

---

Published as a conference paper at ICLR 2024
Frangois-Pierre Paty and Marco Cuturi. Subspace robust wasserstein distances. In International conference on machine learning, pp. 5072-5081. PMLR, 2019.
Karol J. Piczak. Esc: Dataset for environmental sound classification. Proceedings of the 23rd ACM international conference on Multimedia, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8766 PMLR, 2021.
Tyrrell Rockafellar. Conver analysis, volume 11. Princeton university press, 1997.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556-2565, 2018.
Liangliang Shi, Gu Zhang, Haoyu Zhen, Jintao Fan, and Junchi Yan. Understanding and generalizing contrastive learning from the inverse optimal transport perspective. 2023.
Andrew M. Stuart and Marie-Therese Wolfram. Inverse optimal transport. SIAM J. Appl. Math., 80: 599-619, 2019.
Kaiye Wang, Ran He, Wei Wang. Liang Wang, and Tieniu Tan. Learning coupled feature spaces for cross-modal matching. In Proceedings of the IEEE international conference on computer vision, pp. 2088-2095, 2013.
Jiwei Wei, Xing Xu, Yang Yang, Yanli Ji, Zheng Wang, and Heng Tao Shen. Universal weighting metric learning for cross-modal matching. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 13005-13014, 2020.
Jiwei Wei, Yang Yang, Xing Xu, Xiaofeng Zhu, and Heng Tao Shen. Universal weighting metric learning for mooss- retrieval. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(10)x6534-6555 2021.
Bichen Wu, Ruizhe Cheng, Peizhao Zhang, Tianren Gao, Joseph E. Gonzalez, and Peter Vajda. Data efficient aunuuage-uuperviee zero-shot recognition with optimal transport distillation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022a.
Yusong Wu, K. Chen, Tianyu Zhang. Yuchen Hui, Taylor Large-scale contrastive language-audin pretraining with feature fusion and keyword-to-caption augmentation. ArXiv, abs/2211 1.06687,6 2022b.
and Shlomo Dubnov.
Hu Xu, Gargi Ghosh, Yao Huang. Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109, 14084, 2021.
Jinyu Yang. Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre-training with triple contrastive learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15671-15680, 2022.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari. Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 IMPLEMENTATION DETAILS
AudioCaps is the biggest audio captioning dataset that consists of around 50k audio-caption pairs. All audio clips are extracted from AudioSet (Gemmeke et al., 2017). a large-scale dataset for audio tagging. There are a total of 40,582 audio clips in training data, and all audio clips are 10 seconds long. Each audio clip has a single human-annotated caption. The validation and test sets have 494 and 957 audio clips, respectively, and each audio clip has five ground-truth captions.
Clotho is an audio captioning dataset collected from the Freesound platform. The audio clips' length varies from 15 to 30 seconds. We use the second version of the Clotho dataset to carry out the experiment. There are 3839 audio clips in the training set and over 1,000 audio clips in the validation and test set. All captions in the dataset consist of 5 suman-annotatee captions.
ESC50 is an audio dataset for sound event detection. The dataset incorporates 2,000 labeled environmental recordings with 50 classes. We only use the test set of the ESC50 dataset verify the transferable ability of our proposed method. There are 400 audio clips in the test set used in our experiment.
Implementation details. We follow the implementation details in (Mei et al. 2022) Log mel- spectrograms are used as the audio features for the audio encoder, and the settings used to extract Log mel-spectrograms are window size of 1024, Hanning window, hop-size of 320, and 64 first mel bins. The audio encoder is the ResNet-38 model (Kong et al., 2019) that is pretrained on the AudioSet dataset with the audio tagging task. After pertaining. then apply an everage-pooligg layer to accumulate features along the frequency dimension on the extracted feature map. We also conducted experiment on the HTSAT audio encoder (Chen et al. encoder. The text encoder the BERT model (Devlin et al.
discard the last two layers and
2022), which is a transformer-ban 2018), which is pretrained on various NLP tasks to be able to extract contextual-awart embeddings. Due to the dimension mismatch between output of the audio encoder and the text encoder, we utilize two linear project blocks to project two encoder outputs to the shared embedding space. The shared embedding space is a 1024-dimessional space. The interaction matrix M is initialized as follows M = (A + A") + 1, where A random initialized matrix of size 1024 by 1024. All the models and the matrix M are trained for 30 epochs with Adam optimizer (Kingma & Ba 2014). The hyperparameters for training are learning rate Ir x 10-4 batch size b 256, and dropout ratio p 0.2. All experiments performed on single A100 GPU.
A.2 ALGORITHM
Algorithm Learning ground cost metric usingIm-LTM
Algorithm 2: Sinkhorn Algorithm
framework and Mahanalobis distance
Input : Raxa and Output: Te
Input Initialize audio encoder for text encnder go- interaction matrix M, and training data p(x; y)
K = exp (-C/e), )) In:
Output: I.carned audio encoder fo. text encoder Oo and interaction matrix M
while until converged do
u In 0 Ke; In 2 KTu:
while until converged do
T. = diag(u) return m.
(x", V= ~ p(æ, v):
Z, )
Z, = (oo(y'))
# = Eye(H);
C. (Z Zo V(Zi -1) Z2);
TeCOO..M Sinkhorn(C2,.2.
y
[(0, M)
KL(
:c..o.o.M)
0 = 8 VoLi
Φ 0- noLC:
M Proj (M- -1 M)
return fo, Gd. and M
13

---

Published as a conference paper at ICLR 2024
A.3
QUALITATIVE EXPERIMENTS
Audio
Audio
Audio
Caption
Caption
Caption
(a) True matching
(b) Contrastive loss
(c) m LTM loss
Figure 3: The visualization of the true matching, and inference matching from pretrained models using contrastive loss and m-LTM loss on ten audio and ten corresponding captions from the test set of AudioCaps dataset.
Captoo
Cortestive Top-3 retrivedd audio
Top-3
sudning apouuhing purping and acall male apcatol
sucacs
train rabrosd haols so bell bevtggaan
Figure 4: Qualitative results for texi-to-audio retrieval task. top-1, top-2, and top-3 retrieved audio results are from left to right in the figure. The ground ground-truth audio for the caption is marked in red border.
Captis
Contrasivve
IHLTM retreved
ocean werne crathin and opkaling gosla acunak opir dhe didaree
aion
ncce car cerrewing is engine
Figure Qualitative results text-to-audio retrieval task. top-1, top-2, top-3 retrieved audio results are from left to right the figure. The ground- t-tth audio for the caption marked in red border.
A.4 ADDITIONAL EXPERIMENTS AND ABLATION STUDY
We compare the m-LTM loss with contrastive loss and variant contrastive loss with entropic regularization in Table. 7 The training objective of contrastive loss with entropic regularization is as follows
L(0,0) LCL Lone R)
(14)
14

---

Published as a conference paper at ICLR 2024
Table 7: The additional baseline experiment of loss functions, the m-LTM, contrastive loss, and contrastive loss with entropy regularization, on the AudioCaps dataset.
Audio
Text Audio
Audio-> Text
Method
Epsilon
Encoder
R@D
R@5
@@ 10
R@1 1
R@5
R@ 10
CL.
33.9
69.7
82.6
39.4
72.0
83.9
0.05
34.04
69.11
82.51
44.11
74.71
85.37
ResNet38
CL with entropic
0.1
34.27
70.21
82.92
44.41
73.87
86.41
Audio Encoder
regularization
0.5
33.79
69.51
82.90
42.52
72. 10
85.78
m-LTM
0.05
39.10
74.06
85.78
49.94
80.77
90.49
CL
34.38
71.16
84.64
35.84
71.06
83.39
0.05
35.71
72.3
85.
37.45
73.91
84.44
HTSAT Audio
CL. with entropic
35.12
72.05
85.01
36.96
73.55
84.34
Encoder
regularization
0.5
34.45
71.33
85.11
35.98
72.53
83.78
m-LTM
0.05
40.02
75.86
86.42
40.13
74.09
85.89
where LeL is contrastive loss described in the Equation. 1 R is the ranking matrix between two sets audio and caption, and is the coefficient of the entropic regularization term. The experiment shows that entropic regularization can help boost the performance of contrastive loss from 33.9% to 34.04% and from 39.4% to 44.11% for text-to-audio and audio-to-text retrievals, respectively. However, there is remarkable gap in performance between the m-LTM and the best performance acquired by contrastive loss with entropic regularization at = 0.05.
We also provide additional ablation study for choosing the best hyperparameter for the m-LTM framework in Table. g and conducted across datasets retrieval experiment to illustrate the robustness of the m-LTM framework compared with triplet and contrastive loss in Table.
Table 8: Further ablation study AudioCaps dataset for choosing the hyperparameter for the minibatch learning-to toonnngttt-aant to-match framework utilizing Mahalanobis distance.
Text- Audio
Audio > Text
Epsilon
R@1
R@5
R@ 10
R@1
R@5
R 10
0.01
0.10
0.52
1.04
0.10
0.31
0.41
0.02
0.17
0.77
2.06
0.45
1.31
3.78
0.03
38.28
74.85
85.13
49.88
79.77
89.86
0.04
37.22
73.58
85.37
48.38
80.04
90.17
0.05
39.10
74.06
85.78
49.94
80.77
90.49
Table 9: The robustness minibatch learning-ton learning-to-matee to-match, triplet, and contrastive loss for audio text retricval across datasets.
Texto>Audio
Audio- Text
Train-Test
Method
R@ 1
R@5
R 10
R@
R@5
R@ 10
Triplet
10.41
27.38
38.04
11.39
28.42
39.80
Coooo
CL
12.31
30.93
43.44
13.87
33.30
43.54
m-LTM
15.01
35.42
47.71
19.42
37.03
48.61
Triplet
14.33
40
54.42
17.34
43.05
55.69
Cloto->>AuiiCaap
CL
14.64
38.64
53.58
17.76
42.11
55.59
m-LTM
17.38
43.78
58.87
22.04
47.85
61.65
15


Reviews:
[{"review_id": "3HgmiRExXY", "reviewer": "Reviewer_efev", "summary": "The authors revisit the \"Learning to Match\" (LTM) framework (Li et al., 2018), investigating its utilization to learn cross-modal embeddings for text-to-audio and audio-to-text retrieval.\n\nLTM is based on entropic inverse optimal transport (eq. 3), where the goal is to learn the underlying ground metric c which minimizes the KL divergence of the optimal transfer plan $\\pi^{XY}$ determined from (eq. 3) and the empirical joint distribution (eq. 4). Here c is taken as the Mahalanobis distance (eq. 9) between the text and audio embeddings, and a minibatch version (m-LTM) is proposed, where deep networks are used to learn the embeddings, making the parameters of the cost function c the Mahalanobis matrix, and the parameters of the embedding networks.\n\nResults on the AudioCaps and Clotho datasets show large gains over previous approaches (table 1), and large gains over triplet and bi-directional contrastive (eq. 1) losses (table 2). Large gains in terms of zero shot sound event detection (table 3) and modality gap (table 4) are also shown. Large gains in noise tolerance are also shown (table 5). Ablations around using POT for additional robustness show small but consistent gains, and ablations on utlizing Mahalanobis vs. L2 distance also show small but consistent gains.", "questions": "See previous section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Well motivated, inverse optimal transport seems worth exploring in the context of deep networks and minibatch training.\n- Strong results. The approach consistently outperforms existing SOTA text-audio retrieval results on the most popular datasets, and the most widely used contrastive objectives.\n- Generally well presented.", "weaknesses": "- As their results are much better than previous approaches and standard contrastive training methods, I feel that this warrants further investigation. The training sets for AudioCaps and Clotho are rather small at 46K and 5K audio examples, respectively, and so regularization may be a very important factor. Their m-LTM approach is entropy regularized, while their Triplet and Constrastive baselines are not. An entropy-regularized constrastive loss baseline is the most natural analog here, and would more firmly establish the importance of the optimal transport formulation. This feels essential to establishing the significance of the method and results.\n- The m-LTM method presented is somewhat lower in novelty as a variation on LTM that investigates the use of minibatch training and deep networks, but 'revisiting' is explicitly called out in the title, and this seems a worthy exploration.\n- There are a few grammatical errors throughout the paper, but the paper is in general well structured, and adequately well written. A glaring exception is the abstract, which is in really poor shape. Authors, please resolve this.\n- In table 6, the best results for the ground metric hyperparameter $\\epsilon$ are at 0.05, but no values less than it are tested, while the results at 0.5 are very poor, suggesting that results for $\\epsilon<0.05$ should be instead included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JfqIx6y4K2", "reviewer": "Reviewer_Mpor", "summary": "This paper describes an approach for audio-text retrieval using a minibatch version of Learning-to-Match with an optimal transport optimization .  The result of this approach is strong retrieval performance across three datasets.", "questions": "Partial Optimal Transport appears to be performed by noising some percentage of the data prior to learning.  When this noise is applied with a batch size of 2, there is a substantial likelihood that both batch elements will contain incorrect (corrupted) examples.  Is there any risk in this from an algorithmic perspective?  \n\nHow does this approach deal with instances where no match is available?  \n\nThe authors describe one of the positive attributes of this work as identifying a joint embedding space for both speech and text through which to perform retrieval.  Has this embedding space been used for other downstream tasks? This would strengthen the argument that this is a good joint space rather than solely useful for retrieval.\n\nHow effectively does the learned representation work across corpus?  For example, when training on Audiocaps, how effective is retrieval of Clotho test sets and vice versa? This would be a more effective measure of the robustness of the algorithm than the intra-corpus analyses.\n\nThe delta_gap described in table 4 is the difference between the means of the embeddings.  First, is this gap an absolute value? it’s remarkable that mean(f(x_i)) is always greater than mean(g(y_i)).  Second, is the corpus mean of text and audio embeddings a reasonable measure of performance? it seems like this measure could be easily gamed by normalizing the embedding spaces to have a 0-mean, but this wouldn’t add anything to the transferability of the embedding.  Also distance in the embedding space isn’t well defined.  It’s not clear that a unit of distance in the triplet loss space can reasonably be compared to a unit in the m-LTM space.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The algorithmic design of this approach is well motivated and (for the most part) well-described.  (The application of Projection Gradient Descent is effective.)\n\nThe performance is particularly good compared to triplet and contrastive loss.", "weaknesses": "Retrieval applications have an expectation of scaling.  Ideally a single query would be used to retrieve one or more corresponding examples from an extremely large source.  However, in this paper the datasets (particularly the test sets) have a fairly small source to retrieve from (a few thousand examples typically).  It would strengthen the work substantially to demonstrate the capabilities of the algorithm to scale to instances where there are orders of magnitude more examples to retrieve from than queries.\n\nThere is a claim that the correspondence in Figure 1 is obviously better in figure 1b than figure 1a.  I think this is not as obvious as claimed.  I don’t think this image adds particularly value  to the understanding of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EC6LDQyCEM", "reviewer": "Reviewer_Rq97", "summary": "This paper provides a novel method via learning-to-match mechanism, approaching current audio-text retrieval framework and learn the embedding space. The method itself is based on optimal transport, with solid mathematical foundations. The method itself leads to notable  improvements in cross-modal retrieval on common audio datasets, with event detection as the downstream task and additional analysis. Although there are some gaps between the framework design and the scenarios it targets to, it is a well-formed study, which the reviewers learns a lot from.", "questions": "Most of the questions have been asked as weaknesses in the above section. Please answer them. \n\nI also have additional trivial questions.\n1. Do you think if it is possible to open-source the models?\n2. Do you think your models will be benefitted by further fine-tuning the network model or pre-trained encoders? Or you think basically that's it (which is totally fine)?\n3. Do you think your model can be adopted to other sound-related tasks, with rather little amount of data for domain adaptation? Or maybe collection of experts is needed?\n4. Would you classify your approach as essentially an \"unsupervised\" or \"supervised\" contrastive loss?\n5. Do you see any possibility on developing a \"parameter-free\" variant of your framework?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The study itself is well-motivated, novel and practical.\n2. The mathematical foundation is good.\n3. The experiments have been done on commonly-known datasets and the improvements are clearly-observed.", "weaknesses": "1. Apart from the performance, it would be good to also show the acquired network architecture, and run-time efficiency for cross-modal inference.\n2. The reference of \"noise\" is not clear and potentially confusing, even with clear references. When talking about the noise, it can be many things. Especially for speech people who are very likely refer to this paper, seems like the definition of \"noise\" is different from real-world interruption - it is totally fine, but please spend some text on clarifying it in the introduction and methodology (aka Section 3.3).\n\nAlso some minor issues:\n1. Although I did not find significant grammatical flaws, please do a thorough check on the language usage. For example, at the beginning of Section 4, \"Cross-modal matching is an essential task.....\"sounds a bit weird.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)