SupCLAP: Controlling Optimization Trajectory Drift in Audio-Text Contrastive Learning with Support Vector Regularization

## Abstract

Abstract Contrastive language-audio pretraining, which aims to unify multimodal representations in a shared embedding space, serves as a cornerstone for building a wide range of applications, from cross-modal retrieval to cutting-edge multimodal large language models. However, we find that the perpendicular component of the pushing force from negative samples in contrastive learning is a double-edged sword: it contains rich supplementary information from negative samples, yet its unconstrained nature causes optimization trajectory drift and training instability. To address this, we propose Support Vector Regularization (SVR), a method that introduces an auxiliary support vector to control this perpendicular component, aiming to harness its rich information while mitigating the associated trajectory drift. The efficacy of SVR is critically governed by its semantic radius, for which we explore two unsupervised modeling strategies: direct parameterization and an adaptive radius predictor module enhanced with constraints to improve its predicting accuracy. Extensive experimental results demonstrate that our method surpasses widely used baselines like InfoNCE and SigLIP loss across classification, monolingual retrieval, and multilingual retrieval on standard audio-text datasets. Both the theoretical analysis and the experimental results on optimizing trajectory drift validate the correctness and effectiveness of our SVR method.

## 1 Introduction

Contrastive Language-Audio Pretraining (CLAP) Wu et al. (2023); Ghosh et al. (2025) aims to learn a unified audio-text embedding space by pulling corresponding pairs closer and pushing others apart. This paradigm, which powers applications like cross-modal retrieval Xie et al. (2024) and multimodal LLMs Xue et al. (2024); Lam et al. (2025), has achieved great empirical success. However, standard InfoNCE-based CLAP methods still struggle to learn ideal representations, facing limitations such as poor temporal alignment of audio events Yuan et al. (2024) and inconsistent multilingual alignment Yin et al. (2025). Therefore, achieving optimal alignment between the language and audio representation spaces remains an open challenge.

Figure: Figure 1: Optimization Trajectory Drift Analysis. Drift is measured by the cosine similarity between the update vector and the ”pulling force” vector; a higher similarity indicates lower drift. Compared to InfoNCE loss, our SVR method effectively mitigates this drift. The increase in drift observed in later training stages is an expected phenomenon because the model’s focus shifts to distinguishing hard negatives. This process requires retaining more information from the perpendicular component, which is the source of the drift. This result illustrates the existence of optimization trajectory drift and supports the hypothesis that the perpendicular component is not entirely detrimental. Our designed SVR enables the model to effectively manage the trade-off between reducing drift and retaining sufficient negative sample information.
Refer to caption: img/cosine_similarity.jpg

In this paper, we uncover a complex yet overlooked dynamic in the optimization process of standard InfoNCE-based contrastive learning Wu et al. (2021): optimization trajectory drift. We conceptualize the contrastive learning process as an interplay between a ”pulling force” from positive pairs and a ”pushing force” from negative pairs within the embedding space. According to the analysis in Section [2](https://arxiv.org/html/2509.21033v1#S2) and the experimental results in Figure [1](https://arxiv.org/html/2509.21033v1#S1.F1), we find that this pushing force is generally not collinear with the pulling force. This phenomenon stems from both the inherent structure of the data distribution and the stochasticity of mini-batch sampling. The resulting perpendicular component of the pushing force is a double-edged sword. On the one hand, it contains supplementary information from negative samples with rich learning signals. On the other hand, its uncontrolled and fluctuating nature exerts a sideways force, causing the optimization trajectory to drift. This instability not only slows convergence but also hinders the final alignment quality.

To address the optimization trajectory drift problem, we perform a detailed theoretical and qualitative analysis of the optimization direction of standard InfoNCE-based CLAP methods. Based on this analysis, we design a CLAP framework called SupCLAP, which leverages our proposed Support Vector Regularization (SVR) method. The SVR method introduces a new regularization term to the training objective, formulated as an additional contrastive loss computed between the audio embeddings and newly constructed text support vectors. These support vectors are created by displacing original text embeddings toward their positive audio embedding. The magnitude of this displacement is determined by a semantic radius $R$, which we model in an unsupervised manner. Our analysis demonstrates that SVR effectively reshapes the gradient space, adaptively suppressing the perpendicular component while retaining sufficient supplementary information from negative samples. This yields a more stable optimization trajectory, leading to improved alignment quality.

Building on our analysis for SVR, we further posit that a more precise semantic radius $R$ enables finer control over the perpendicular component, directly enhancing SVR’s efficacy. To this end, we explore its unsupervised modeling through two primary strategies. The first strategy is StaticSVR, where the radius is treated as a learnable parameter. The other is the DynamicSVR, which uses an adaptive radius predictor module that utilizes embedding similarity information in the mini-batch to predict the semantic radius $R$. We further propose constraints for the radius predictor module to enhance the quality and stability of the modeled semantic radius. Our main contributions are listed as follows:

- •
We find and analyze the optimization trajectory drift problem in contrastive learning from a force decomposition perspective, linking it to the perpendicular component of the pushing force from negative samples.
- •
We propose the SupCLAP scheme with Support Vector Regularization (SVR) and rigorously justify its ability to control optimization trajectory drift. By reshaping the gradient landscape, SVR leads to more stable and effective audio-text contrastive learning.
- •
We conduct a systematic exploration into unsupervised semantic radius modeling, proposing two strategies: StaticSVR and DynamicSVR. Furthermore, we propose constraints for the DynamicSVR to improve the quality of the predicted radius, validating our hypothesis on the importance of radius precision.
- •
Extensive experiments on diverse audio-text benchmarks not only demonstrate that SupCLAP achieves significant improvements over baseline methods such as InfoNCE and SigLIP loss with negligible additional overhead, but also provide strong validation for our theoretical analysis of optimization trajectory drift and the effectiveness of the SupCLAP.

## 2 Analysis of Optimization Trajectory Drift

In this section, we analyze the gradient space of contrastive learning by decomposing its gradient into a pulling force from the positive audio embedding and pushing force from the negative audio embeddings. We then further analyze how the component of the pushing force perpendicular to the pulling force affects model optimization, to clearly illustrate the optimization trajectory drift problem.

### 2.1 Contrastive Learning Loss Function and Gradient

First, we define the basic framework for contrastive learning. Assume we have a text embedding $t^{+}\in\mathbb{R}^{d}$, a matching positive audio embedding $a^{+}\in\mathbb{R}^{d}$, where $d$ is the size of embedding, and a batch of $N-1$ mismatched negative audio embeddings $\{a_{j}^{-}\}_{j=1}^{N-1}$. To simplify the derivation, we assume all embedding vectors are L2-normalized, i.e., $||t||=||a||=1$. The similarity function used is the scaled dot product (Scaled Cosine Similarity): $s(a,t)=\cos(a,t)/\tau=\frac{a^{T}t}{\tau}$, where $\tau$ is the temperature hyperparameter.

The standard InfoNCE loss function Koromilas et al. (2024) (for text-to-audio, as an example) is:

$$ $\displaystyle L_{orig}(t^{+},a^{+},\{a^{-}_{j}\})=-\log\frac{\exp(s(t^{+},a^{+}))}{\exp(s(t^{+},a^{+}))+\sum^{N-1}_{j=1}\exp(s(t^{+},a^{-}_{j}))}.$ (1) $$

To analyze the optimization direction, we calculate the gradient of the loss function to the text embedding $t^{+}$, denoted as $\nabla_{t}\mathcal{L}_{orig}$. We define:

$$ $\displaystyle P^{+}=\frac{\exp(s(t^{+},a^{+}))}{\exp(s(t^{+},a^{+}))+\sum^{N-1}_{j=1}\exp(s(t^{+},a^{-}_{j}))},$ (2) $$

$$ $\displaystyle P^{-}_{j}=\frac{\exp(s(t^{+},a^{-}_{j}))}{\exp(s(t^{+},a^{+}))+\sum^{N-1}_{j=1}\exp(s(t^{+},a^{-}_{j}))},$ (3) $$

where $P^{+}+\sum P^{-}_{j}=1$.

According to the chain rule, $\nabla_{t}s(t,a)=a/\tau$. The gradient of loss function can be derived as:

$$ $\displaystyle\nabla_{t}L_{orig}$ $\displaystyle=\frac{1}{\tau}\Bigg[\Bigg(\sum^{N+1}_{j=1}P^{-}_{j}a_{j}^{-}+P^{+}a^{+}\Bigg)-a^{+}\Bigg]$ (4) $\displaystyle=\frac{1}{\tau}\Bigg[(P^{+}-1)a^{+}+\sum^{N-1}_{j=1}P^{-}_{j}a^{-}_{j}\Bigg].$ $$

We decompose the gradient into two parts:

- •
Pulling Force: $F_{pull}=\frac{1}{\tau}(P^{+}-1)a^{+}$. Since $0\textless P^{+}\textless 1$, the term $(P^{+}-1)$ is negative. In gradient descent, we update text embedding $t^{+}$ in the direction of $-\nabla_{t}L_{orig}$ (i.e., $t^{+}\leftarrow t^{+}-\eta\nabla_{t}L_{orig}$). Therefore, this term is equivalent to a force that pulls text embedding $t^{+}$ towards positive audio embedding $a^{+}$.
- •
Pushing Force: $F_{push}=\frac{1}{\tau}\sum^{N-1}_{j=1}P^{-}_{j}a^{-}_{j}$. This term is a weighted average of all negative example embeddings, and its effect is to push the text embedding $t^{+}$ away from all negative audio embeddings $a^{-}_{j}$. It should also be noted that a negative audio embedding $a^{-}_{k}$ with high similarity to the text embedding $t^{+}$ will yield a larger probability $P^{-}_{k}$, thereby exerting a greater pushing force on the text embedding.

#### 2.1.1 Suppress Trajectory Drift Problem

We find that the pushing force $F_{push}$ is generally not parallel to the direction of the pulling force $F_{push}$. We can decompose the pushing force from a single negative example $a^{-}_{j}$, which is given by pushing force $f_{push,j}=\frac{P^{-}_{j}}{\tau}a^{-}_{j}$. Let us define the unit vector in the pulling direction as $\hat{u}=\frac{a^{+}-t^{+}}{||a^{+}-t^{+}||}$. The negative pushing force $f_{push,j}$ can be decomposed into a component parallel $f_{||,j}$ to $\hat{u}$ and a component perpendicular $f_{\perp,j}$ to $\hat{u}$, which is denoted as $f_{push,j}=f_{||,j}+f_{\perp,j}$. The parallel component is computed as $f_{||,j}=(f_{push,j}\cdot\hat{u})\hat{u}$ while the perpendicular component $f_{\perp,j}=f_{push,j}-f_{||,j}=f_{push,j}(I-\hat{u}\hat{u})$.

During the optimization process, the parallel component $f_{||,j}$ shares the same direction as the pulling force from the positive sample, differing only in magnitude. It thus primarily affects convergence speed and contains little additional information from the negative samples beyond what is already present in the pulling force. In contrast, the gradient direction of the perpendicular component $f_{\perp,j}$ is rich with supplementary information from the negative samples, which is distinct from that of the positive sample. But this perpendicular component acts as a double-edged sword: while its direction provides additional information, its uncontrolled magnitude can cause the optimization path to drift continuously. The nature of this drift can be understood on both global and local levels, which is illustrated in Figure [2](https://arxiv.org/html/2509.21033v1#S2.F2).

Figure: Figure 2: Illustration of Global and Local Perpendicular Components. The left subfigure depicts the global perpendicular component. The subfigure on the right illustrates the local perpendicular component. For clarity in demonstrating the local perpendicular component, the negative audio embeddings are shown in the right subfigure with distinct distributions across batches. In practice, the negative distributions across batches are more likely to overlap, as shown in the left subfigure.
Refer to caption: img/perpendicular_component.png

- •
Global Perpendicular Component: From a global perspective, even if we could access all negative examples in the dataset, the direction of their weighted resultant force would seldom be collinear with the direction of the pulling force from a specific positive example. This creates a systematic, global perpendicular component. Throughout the entire model training process, this global component continuously pushes the optimization path of the text embedding away from the ideal straight-line trajectory, resulting in a systematic drift in the entire alignment path.
- •
Local Perpendicular Component: In practice, training is performed using a mini-batch strategy, which means that at each update step, the negative examples seen by the model are only a random subset of the total population of negative examples. The distribution of negative examples varies between batches, causing the direction and magnitude of the perpendicular component to change drastically with each update. This uncertain and random sideways push is the direct cause of the local, high-frequency oscillations observed in the optimization path.

Acting in concert, these two perpendicular components give rise to the problem of optimization trajectory drift in audio-text contrastive learning. This reduces the model’s convergence efficiency and ultimately limits the alignment accuracy of the learned representations if the perpendicular components are not well controlled.

## 3 Method

To address the optimization trajectory drift identified in our analysis, we propose the SupCLAP framework built around the Support Vector Regularization (SVR) method. The main idea of SVR is to control the perpendicular component of pushing force by using an auxiliary regularization term to align text support vectors and audio embeddings, thereby guiding the optimization towards a more stable and direct trajectory. The construction of this support vector is critically dependent on a semantic radius $R$, which dictates the magnitude of the regularization. As the semantic radius $R$ lacks direct supervision, we explore two unsupervised strategies for modeling the semantic radius: a straightforward direct parameterization (StaticSVR) and a more sophisticated adaptive radius predictor module (DynamicSVR), which is further enhanced with a constraint term to improve the quality of the predicted semantic radius. We analyze SVR’s effect mathematically in Appendix [C](https://arxiv.org/html/2509.21033v1#A3).

### 3.1 Framework of SupCLAP

The SupCLAP framework is built upon the standard symmetric contrastive learning objective common in CLAP architectures. We begin by defining this baseline. For a given batch, we have a positive audio-text pair $(a^{+},t^{+})$ and $N$ negative pairs $\{(a^{-}_{j},t^{-}_{j})\}^{N}_{j=1}$, all represented in a $d$-dimensional embedding space. The affinity between audio and text embeddings is measured by the scaled cosine similarity $s(a,t)=\cos(a,t)/\tau$, where $\tau$ is a temperature hyperparameter. The standard training objective $L_{orig}$ consists of two InfoNCE loss terms: one for text-to-audio alignment loss $L_{orig,t2a}$ and one for audio-to-text alignment $L_{orig,a2t}$. The total original loss is formulated as follows:

$$ $\displaystyle L_{orig}=$ $\displaystyle L_{orig,t2a}+L_{orig,a2t}$ (5) $\displaystyle=$ $\displaystyle-\log\frac{\exp(s(t^{+},a^{+})/\tau)}{\exp(s(t^{+},a^{+})/\tau)+\sum^{N}_{j=1}\exp(s(t^{+},a^{-}_{j})/\tau)}$ $\displaystyle-\log\frac{\exp(s(a^{+},t^{+})/\tau)}{\exp(s(a^{+},t^{+})/\tau)+\sum^{N}_{j=1}\exp(s(a^{+},t^{-}_{j})/\tau)}.$ $$

To address the trajectory drift problem, we design the Support Vector Regularization (SVR) term, denoted as $L_{svr}$. The final training objective of SupCLAP is a weighted sum of the original loss and our SVR regularization term:

$$ $\displaystyle L_{SupCLAP}=$ $\displaystyle L_{orig}+\alpha L_{svr}$ (6) $\displaystyle=$ $\displaystyle L_{orig}-\alpha\log\frac{\exp(s(t_{sup},a^{+}))}{\sum^{N}_{j=1}\exp(s(t_{sup},a^{-}_{j}))},$ $$

The text support vector $t_{sup}=t^{+}+R\hat{u}$ is constructed by displacing the original text embedding $t^{+}$ by the semantic radius $R\in\mathbb{R}$ along the unit vector $\hat{u}=\frac{a^{+}-t^{+}}{||a^{+}-t^{+}||}$. The hyperparameter $\alpha$ controls the influence of this SVR term in the final loss function. We will detail the unsupervised strategies for modeling the semantic radius $R$ in the following subsection. The intuition behind this design is that by creating an auxiliary SVR term, it reshapes the gradient space. It is engineered to selectively control the perpendicular component of the pushing force from negative samples, which is the primary source of trajectory drift. A detailed mathematical analysis of SVR’s mechanism is provided in Appendix [C](https://arxiv.org/html/2509.21033v1#A3). We conclude that after adding the SVR term, the parallel component of pushing force from the $j$-th negative audio embedding $a^{-}_{j}$ is

$$ $\displaystyle F_{||,push,j}=\bigg(\frac{P_{j}^{-}}{\tau}+\alpha\frac{P^{-}_{sup,j}}{\tau}\bigg)a^{-}_{||,j},$ (7) $$

while the perpendicular component of the pushing force is

$$ $\displaystyle F_{\perp,push,j}=\bigg[\frac{P_{j}^{-}}{\tau}+\alpha\frac{P^{-}_{sup,j}}{\tau}\bigg(1-\frac{R}{||a^{+}-t^{+}||}\bigg)\bigg]a^{-}_{\perp,j}.$ (8) $$

The probability $P^{-}_{j}$ and $P^{-}_{sup,j}$ can be calculated following equation [3](https://arxiv.org/html/2509.21033v1#S2.E3) using text embedding $t^{+}$ and text support vector $t_{sup}$ separately. $a^{-}_{||,j}$ denotes the parallel component of audio embedding $a^{-}_{j}$ to the unit vector $\hat{u}=\frac{a^{+}-t^{+}}{||a^{+}-t^{+}||}$, while $a^{-}_{\perp,j}$ denotes the perpendicular component of audio embedding $a^{-}_{j}$ to the unit vector $\hat{u}$. The SVR term works by reshaping the total pushing force: while the parallel component is preserved, the perpendicular component is uniquely scaled by the factor $(1-\frac{R}{||a^{+}-t^{+}||})$. This factor allows SVR to selectively suppress the magnitude of the perpendicular component of the pushing force. A larger semantic radius R leads to a higher degree of suppression on this component. By exerting this precise control, SVR effectively controls the trajectory drift while harnessing the rich information from negative samples.

Note: Although our illustration and analysis of SVR are focused on the unidirectional (text-to-audio alignment) SVR for clarity, the principle is not exclusive to this direction. As demonstrated by the experimental results in Subsection [4.2](https://arxiv.org/html/2509.21033v1#S4.SS2), incorporating bidirectional SVR in both text-to-audio and audio-to-text alignments yields superior performance.

Inference pipeline: SupCLAP’s inference pipeline is identical to that of standard CLAP methods Elizalde et al. (2023). Retrieval and classification are performed solely by ranking the embedding similarity between audio and text, without the need to compute auxiliary support vectors like $t_{sup}$.

### 3.2 Unsupervised Semantic Radius Modeling

The effectiveness of our proposed SVR critically depends on the semantic radius $R\in\mathbb{R}$. This parameter is central to our method as it directly governs the factor $(1-\frac{R}{||a^{+}-t^{+}||})$, which scales the perpendicular component of the pushing force. Since datasets provide no ground-truth values for the semantic radius $R$, we frame its estimation as an unsupervised modeling problem. To this end, we propose and investigate two distinct strategies:

- •
Static modeling (StaticSVR):
The StaticSVR is a modeling strategy focused on suppressing the global perpendicular component. We model the semantic radius R as a single, globally shared, learnable scalar. This scalar is jointly optimized with other model parameters to minimize the total loss $L_{SupCLAP}$. For any text embedding $t^{+}$, this constant radius is used to construct the support vector $t_{sup}=t+R\cdot\hat{u}$, where $\hat{u}$ is the unit vector of the pulling force.
This method’s primary advantages are its simplicity and the stability of the global radius $R$. However, its limitation is the idealized assumption that a single, constant radius is optimal for all instances. This static approach lacks the flexibility to handle the varying complexity and alignment difficulty among different audio-text pairs.
- •
Dynamic modeling (DynamicSVR): The DynamicSVR is a modeling strategy focused on suppressing the local perpendicular component. For adaptive, instance-specific control, we propose a radius predictor. We hypothesize the optimal semantic radius $R$ is not fixed; it should adapt dynamically to the local embedding geometry, defined by the relationship between a text embedding $t^{+}$ and its positive and negative audio counterparts. To learn this relationship, we design a radius predictor—a lightweight Multi-Layer Perceptron (MLP) $f_{\theta}$ which learns to output a suitable radius in a fully unsupervised manner. Formally, the predictor $f_{\theta}:\mathbb{R}^{N}\rightarrow\mathbb{R}$ maps a local similarity vector $S$ to an instance-specific semantic radius $R=f_{\theta}(S)$. The similarity vector $S$ is composed of $S=[s(t^{+},a^{+}),s(t^{+},a^{-}_{1}),...,s(t^{+},a^{-}_{N-1})]$.
The similarity vector $S$ informs the radius predictor by capturing local embedding geometry. Each values in $S$ measures the proximity between text embedding $t^{+}$ and audio embeddings, collectively signaling potential trajectory drift; for instance, high similarity to a negative sample implies a higher drift risk. Learning these patterns allows the predictor to estimate a custom semantic radius $R$ for precise control. This adaptive approach’s main advantage is its flexibility to adjust the optimization trajectory to each mini-batch’s alignment difficulty. However, DynamicSVR’s performance is heavily dependent on the prediction accuracy of the batch-level radius $R$. Inaccurate predictions of $R$—caused by noisy data, a noisy embedding space, or a weak pretrained model—may make DynamicSVR’s performance inferior to that of the simpler StaticSVR.
We propose a constraint term $L_{cons}$, for the radius predictor module to mitigate the problem of the predicted semantic radius R being unstable or inaccurate. Due to space limitations, the introduction of this constraint term is provided in Appendix [B](https://arxiv.org/html/2509.21033v1#A2).

## 4 Experiment

In this section, we present a comprehensive evaluation of our proposed SupCLAP framework. We begin by detailing the datasets, models, evaluation metrics, and experimental setup. We then benchmark our main results on monolingual and multilingual audio-text retrieval, as well as on the zero-shot classification tasks.
Subsequently, we conduct detailed ablation studies to dissect the contribution of each component in our method, the effectiveness of SVR under different batch sizes, and the impact of the SVR term’s weight $\alpha$. Finally, we evaluate the additional time and GPU memory overhead and show that they are negligible. In addition, we experimentally analyze how the modeled semantic radius changes with training epochs. These experiments further demonstrate the theory of trajectory drift and the effectiveness of our proposed approach. Due to space limitations, the evaluation results and corresponding analysis on the multilingual dataset, partial ablation study results, and the analysis of SVR’s overhead are presented in Appendix [D](https://arxiv.org/html/2509.21033v1#A4).

### 4.1 Experimental Setup

#### 4.1.1 Datasets and Metrics

Our experiments are conducted on the AudioCaps Kim et al. (2019) and Clotho Drossos et al. (2020) datasets. AudioCaps consists of approximately 49k 10-second audio clips, where each training sample is paired with a single caption, and each validation/test sample has five captions. Clotho contains 6974 clips of 15-30 seconds in duration, all of which are annotated with five captions. For our multilingual scenario, we augment these datasets by translating all English captions into seven other languages (fra, deu, spa, nld, cat, jpn, zho). All audio clips are resampled to 16 kHz.

We evaluate retrieval performance in both monolingual and multilingual settings using Recall at rank k (R@k), which is 1 if the target is found in the top-k results, and mean Average Precision at 10 (mAP10), which evaluates precision scores across all queries for the top 10 retrieved items.

#### 4.1.2 Models and Implementation Details

Model Architecture: Our framework for audio-text retrieval leverages two powerful pretrained encoders to handle the respective modalities. We adopt the CED-Base model Dinkel et al. (2024) as our audio encoder. We employ the multilingual SONAR-TE model Duquenne et al. (2023) as text encoder, which has demonstrated good performance on cross-lingual similarity benchmarks like xsim and xsim++, making it well-suited as the text encoder for this paper. The sentence embeddings are computed by pooling the encoder’s token-level hidden states. The semantic radius predictor is implemented as an MLP with 3 layers.

Implementation Details: To ensure a fair and controlled comparison, all models are initialized with weights from CED-Base and SONAR-TE and trained for 10 epochs on a single NVIDIA H800 GPU.
The training employs the Adam optimizer with a learning rate of $5\times 10^{-5}$. The batch size is set to 24, and a temperature of $\tau$=0.07 for the contrastive loss. The SVR weight $\alpha$ is set to $1$. We investigate two distinct audio-text retrieval scenarios: a monolingual scenario, utilizing only English captions, and a multilingual scenario. The model checkpoint yielding the highest recall on the test set is selected for final evaluation.
We name our SVR methods like ”bi-DynamicSVR” and ”uni-StaticSVR”. The prefix ”bi” or ”uni” indicates bidirectional (both audio-to-text and text-to-audio) or unidirectional (text-to-audio) SVR, respectively. The suffix ”StaticSVR” or ”DynamicSVR” denotes the different ways of modeling the semantic radius $R$ mentioned in [3.2](https://arxiv.org/html/2509.21033v1#S3.SS2).

### 4.2 Main Results

#### 4.2.1 Audio-Text Retrieval

To evaluate the effectiveness of our proposed framework, we compare four methods: (1) the standard InfoNCE loss; (2) the standard SigLIP loss Zhai et al. (2023); (3) our bi-StaticSVR, where the semantic radius $R$ is a single, globally shared, learnable parameter; and (4) our bi-DynamicSVR, which utilizes an adaptive predictor to estimate an instance-specific radius $R$.

**Table 1: Recall and precision results under monolingual AudioCaps and Clotho dataset**
| Model | AudioCaps | Clotho |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| T2A | A2T | T2A | A2T |  |  |  |  |  |
| R@1 | R@10 | R@1 | R@10 | R@1 | R@10 | R@1 | R@10 |  |
| Oncescu et al. (2021) | 36.10 | 84.50 | 39.60 | 86.70 | 6.70 | 33.20 | 7.00 | 34.60 |
| Mei et al. (2022) | 33.90 | 82.60 | 39.40 | 83.90 | 14.40 | 49.90 | 16.20 | 50.20 |
| Elizalde et al. (2023) | 34.60 | 82.00 | 41.90 | 84.60 | 16.70 | 54.10 | 20.00 | 58.70 |
| Ghosh et al. (2023) | 36.10 | 81.60 | 45.20 | 86.70 | 16.80 | 56.10 | 19.70 | 55.60 |
| Wu et al. (2023) | 34.50 | 80.20 | 42.50 | 87.40 | 15.80 | 52.90 | 19.10 | 54.90 |
| Xie et al. (2024) | 36.20 | 82.90 | 44.20 | 86.70 | 15.70 | 50.90 | 18.60 | 55.30 |
| Yuan et al. (2024) | 39.70 | 86.90 | 49.80 | 91.90 | 17.30 | 53.60 | 21.80 | 57.40 |
| Ghosh et al. (2025) | 37.10 | 85.00 | 48.00 | 90.80 | 18.90 | 59.00 | 20.50 | 58.90 |
| SigLIP | 36.74 | 85.71 | 48.00 | 88.03 | 13.58 | 51.21 | 17.10 | 52.56 |
| -bi-StaticSVR | 42.54 | 87.61 | 55.25 | 90.55 | 16.21 | 53.60 | 21.26 | 59.13 |
| -bi-DynamicSVR | 43.09 | 89.26 | 56.30 | 92.67 | 17.51 | 56.85 | 22.71 | 60.87 |
| InfoNCE | 41.87 | 87.69 | 56.72 | 92.33 | 18.67 | 58.42 | 22.61 | 63.09 |
| -bi-StaticSVR | 43.89 | 88.78 | 57.77 | 92.75 | 19.50 | 58.86 | 24.93 | 63.19 |
| -bi-DynamicSVR | 44.16 | 89.24 | 59.66 | 93.49 | 19.75 | 59.13 | 25.31 | 63.29 |

Monolingual Retrieval: Results from the monolingual audio-text retrieval task, as shown in Table [1](https://arxiv.org/html/2509.21033v1#S4.T1), demonstrate that both bi-StaticSVR and bi-DynamicSVR effectively enhance the performance of InfoNCE and SigLIP. This validates our analysis of the perpendicular component in contrastive learning and the effectiveness of our proposed SVR term in resolving the trajectory drift problem. The superior performance of InfoNCE compared to SigLIP stems from its Softmax-based competitive mechanism, which provides a stronger gradient signal for effective discrimination on diverse audio datasets like AudioCaps and Clotho that contain numerous hard negatives.

#### 4.2.2 Zero-Shot Audio Classification

We also assess zero-shot audio classification on the ESC-50 and US8K benchmarks using our monolingual model loaded with pretrained weights provided in ML-CLAP Yan et al. (2024). Text labels are constructed using the template ”This is a sound of {classlabel}”, and we report top-1 accuracy based on the highest cosine similarity between audio and text embeddings.

**Table 2: Zero-shot Audio Classification Performance of the CLAP Model**
| Model | Audio Classification Dataset & Setting |  |  |
| --- | --- | --- | --- |
| ESC-50 | US8K | VGGSound |  |
| Wu et al. (2022) | 41.4 | 40.4 | 10.0 |
| Guzhov et al. (2022) | 69.4 | 65.3 | - |
| Deshmukh et al. (2022) | 82.6 | 73.2 | - |
| Elizalde et al. (2023) | 83.4 | 74.5 | 16.4 |
| Wu et al. (2023) | 89.5 | 76.3 | 23.1 |
| Silva et al. (2023) | 84.0 | 77.0 | - |
| InfoNCE | 89.6 | 81.63 | 24.57 |
| -bi-StaticSVR | 90.7 | 83.63 | 24.65 |
| -bi-DynamicSVR | 92.1 | 83.74 | 25.11 |

As shown in Table [2](https://arxiv.org/html/2509.21033v1#S4.T2), the bidirectional dynamic SVR achieves the highest classification accuracy. This further demonstrates the generalization capability of SVR, which can learn more robust and semantically meaningful feature representations by effectively suppressing trajectory drift, thereby enhancing the model’s performance in both retrieval and classification tasks.

#### 4.2.3 Ablation Studies

We conducted extensive ablation studies on the AudioCaps dataset. These studies include evaluating the impact of SVR components with different directions and modeling strategies on model performance, the generalization of SVR under different batch sizes, and the effect of the SVR weight $\alpha$ on model performance. Due to space limitations, the ablation experiment results and analysis for different batch sizes and the SVR weight are presented in Appendix [D.3](https://arxiv.org/html/2509.21033v1#A4.SS3). The results show that both StaticSVR and DynamicSVR can effectively improve the performance of contrastive learning across various batch sizes, and the best results are achieved when the SVR weight $\alpha$ is set to 1.

**Table 3: Ablation Study of SVR Variants on Monolingual Text-Audio Retrieval.**
| ID | Model | T2A | A2T |  |  |
| --- | --- | --- | --- | --- | --- |
| R@1 | mAP10 | R@1 | mAP10 |  |  |
| 0 | InfoNCE | 41.87 | 56.74 | 56.72 | 35.36 |
| 1 | -bi-DynamicSVR | 44.16 | 58.79 | 59.66 | 36.69 |
| 2 | -bi-DynamicSVR wo/ constraints | 44.01 | 58.47 | 59.24 | 36.64 |
| 3 | -uni-DynamicSVR | 43.63 | 58.16 | 58.51 | 36.00 |
| 4 | -uni-DynamicSVR wo/ constraints | 43.53 | 58.11 | 57.67 | 35.96 |
| 5 | -bi-StaticSVR | 43.89 | 58.36 | 57.77 | 35.72 |
| 6 | -uni-StaticSVR | 43.28 | 57.95 | 57.56 | 34.62 |

Effectiveness of SVR Components: The results in Table [3](https://arxiv.org/html/2509.21033v1#S4.T3) systematically demonstrate the effectiveness of our proposed components for SVR. The fully-equipped bidirectional DynamicSVR model with constraints (bi-DynamicSVR) achieves the best results. We find that the unidirectional SVR outperforms the baseline model. This performance gain is further amplified when bidirectional SVR is simultaneously applied to both audio-to-text and text-to-audio directions. Furthermore, the experimental results indicate that introducing the constraint term in equation [9](https://arxiv.org/html/2509.21033v1#A2.E9) improves the accuracy of DynamicSVR in modeling the semantic radius $R$, further enhancing the effectiveness of SVR.

### 4.3 Semantic Radius Analysis

Figure: Figure 3: Results of Semantic Radius Changes
Refer to caption: img/semantic_radius.jpg

We extract the parameterized radius for bi-StaticSVR at each epoch and compute the average predicted radius for bi-DynamicSVR. As depicted in Figure [3](https://arxiv.org/html/2509.21033v1#S4.F3), the semantic radius $R$ decreases as training progresses. This aligns with our drift analysis in Figure [1](https://arxiv.org/html/2509.21033v1#S1.F1): as positive pairs converge, the model reduces the semantic radius $R$ to intentionally preserve a larger perpendicular component. This mechanism is crucial for distinguishing hard negatives and achieving fine-grained cross-modal alignment. Furthermore, bi-StaticSVR’s radius curve is smoother, reflecting its stable, global modeling of the perpendicular component, whereas bi-DynamicSVR’s local approach results in greater fluctuations across batches.

## 5 Conclusion

In this paper, we find and analyze a critical yet overlooked issue in audio-text contrastive learning: optimization trajectory drift. We conceptualized this problem as a double-edged sword caused by the perpendicular component of the pushing force from negative samples, which contains valuable supplementary information but also leads to training instability. To address this, we proposed the SupCLAP framework, which incorporates our Support Vector Regularization (SVR) method. Our theoretical analysis and extensive experiments empirically validate that SVR effectively controls this perpendicular force by introducing an auxiliary support vector, thus reshaping the gradient space to enable a more stable and direct optimization path. We systematically explore two unsupervised strategies for modeling the semantic radius: StaticSVR and DynamicSVR. Our results on standard audio-text datasets show that SupCLAP consistently outperforms the InfoNCE loss baseline across classification, monolingual, and multilingual retrieval tasks. The minimal additional computational overhead of our method further underscores its practical viability. In the future, we plan to validate our SVR method on a wider range of cross-modal tasks and explore more precise semantic radius modeling strategies further to improve optimization stability in contrastive learning.

## References

- Deshmukh et al. (2022)
Soham Deshmukh, Benjamin Elizalde, and Huaming Wang.
Audio retrieval with wavtext5k and clap training.
*arXiv preprint arXiv:2209.14275*, 2022.
- Dinkel et al. (2024)
Heinrich Dinkel, Yongqing Wang, Zhiyong Yan, Junbo Zhang, and Yujun Wang.
Ced: Consistent ensemble distillation for audio tagging.
In *ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pp. 291–295. IEEE, 2024.
- Drossos et al. (2020)
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen.
Clotho: An audio captioning dataset.
In *ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pp. 736–740. IEEE, 2020.
- Duquenne et al. (2023)
Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot.
Sonar: sentence-level multimodal and language-agnostic representations.
*arXiv e-prints*, pp. arXiv–2308, 2023.
- Elizalde et al. (2023)
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang.
Clap learning audio concepts from natural language supervision.
In *ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pp. 1–5. IEEE, 2023.
- Ghosh et al. (2023)
Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Evuru, S Ramaneswaran, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha.
Compa: Addressing the gap in compositional reasoning in audio-language models.
*arXiv preprint arXiv:2310.08753*, 2023.
- Ghosh et al. (2025)
Sreyan Ghosh, Sonal Kumar, Chandra Kiran Reddy Evuru, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha.
Reclap: Improving zero shot audio classification by describing sounds.
In *ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pp. 1–5. IEEE, 2025.
- Guzhov et al. (2022)
Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel.
Audioclip: Extending clip to image, text and audio.
In *ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pp. 976–980. IEEE, 2022.
- Hearst et al. (1998)
Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf.
Support vector machines.
*IEEE Intelligent Systems and their applications*, 13(4):18–28, 1998.
- Kim et al. (2019)
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim.
Audiocaps: Generating captions for audios in the wild.
In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pp. 119–132, 2019.
- Koromilas et al. (2024)
Panagiotis Koromilas, Giorgos Bouritsas, Theodoros Giannakopoulos, Mihalis Nicolaou, and Yannis Panagakis.
Bridging mini-batch and asymptotic analysis in contrastive learning: From infonce to kernel-based losses.
*arXiv preprint arXiv:2405.18045*, 2024.
- Lam et al. (2025)
Max WY Lam, Yijin Xing, Weiya You, Jingcheng Wu, Zongyu Yin, Fuqiang Jiang, Hangyu Liu, Feng Liu, Xingda Li, Wei-Tsung Lu, et al.
Analyzable chain-of-musical-thought prompting for high-fidelity music generation.
*arXiv preprint arXiv:2503.19611*, 2025.
- Liang et al. (2024)
Xin Liang, Yanli Ji, Wei-Shi Zheng, Wangmeng Zuo, and Xiaofeng Zhu.
Sv-learner: Support-vector contrastive learning for robust learning with noisy labels.
*IEEE Transactions on Knowledge and Data Engineering*, 36(10):5409–5422, 2024.
- Liu et al. (2024)
Shengqiang Liu, Da Liu, Anna Wang, Zhiyu Zhang, Jie Gao, and Yali Li.
Dsclap: Domain-specific contrastive language-audio pre-training.
*arXiv preprint arXiv:2409.09289*, 2024.
- Mei et al. (2022)
Xinhao Mei, Xubo Liu, Jianyuan Sun, Mark D Plumbley, and Wenwu Wang.
On metric learning for audio-text cross-modal retrieval.
*arXiv preprint arXiv:2203.15537*, 2022.
- Oncescu et al. (2021)
Andreea-Maria Oncescu, A Koepke, Joao F Henriques, Zeynep Akata, and Samuel Albanie.
Audio retrieval with natural language queries.
*arXiv preprint arXiv:2105.02192*, 2021.
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In *International conference on machine learning*, pp. 8748–8763. PmLR, 2021.
- Shah et al. (2022)
Anshul Shah, Suvrit Sra, Rama Chellappa, and Anoop Cherian.
Max-margin contrastive learning.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 36, pp. 8220–8230, 2022.
- Silva et al. (2023)
Dadallage AR Silva, Spencer Whitehead, Christopher Lengerich, and Hugh Leather.
Collat: on adding fine-grained audio understanding to language models using token-level locked-language tuning.
*Advances in Neural Information Processing Systems*, 36:63197–63209, 2023.
- Wang et al. (2024)
Jiamian Wang, Guohao Sun, Pichao Wang, Dongfang Liu, Sohail Dianat, Majid Rabbani, Raghuveer Rao, and Zhiqiang Tao.
Text is mass: Modeling as stochastic embedding for text-video retrieval.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 16551–16560, 2024.
- Wu et al. (2021)
Chuhan Wu, Fangzhao Wu, and Yongfeng Huang.
Rethinking infonce: How many negative samples do you need?
*arXiv preprint arXiv:2105.13003*, 2021.
- Wu et al. (2022)
Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello.
Wav2clip: Learning robust audio representations from clip.
In *ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pp. 4563–4567. IEEE, 2022.
- Wu et al. (2023)
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov.
Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation.
In *ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pp. 1–5. IEEE, 2023.
- Xie et al. (2024)
Yuxin Xie, Zhihong Zhu, Xianwei Zhuang, Liming Liang, Zhichang Wang, and Yuexian Zou.
Gpa: Global and prototype alignment for audio-text retrieval.
In *Proc. Interspeech*, volume 2024, pp. 5078–5082, 2024.
- Xue et al. (2024)
Jinlong Xue, Yayue Deng, Yingming Gao, and Ya Li.
Retrieval augmented generation in prompt-based text-to-speech synthesis with context-aware contrastive language-audio pretraining.
*arXiv preprint arXiv:2406.03714*, 2024.
- Yan et al. (2024)
Zhiyong Yan, Heinrich Dinkel, Yongqing Wang, Jizhong Liu, Junbo Zhang, Yujun Wang, and Bin Wang.
Bridging language gaps in audio-text retrieval.
*arXiv preprint arXiv:2406.07012*, 2024.
- Yin et al. (2025)
Yuguo Yin, Yuxin Xie, Wenyuan Yang, Dongchao Yang, Jinghan Ru, Xianwei Zhuang, Liming Liang, and Yuexian Zou.
Atri: Mitigating multilingual audio text retrieval inconsistencies by reducing data distribution errors.
*arXiv preprint arXiv:2502.14627*, 2025.
- Yu et al. (2022)
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.
Coca: Contrastive captioners are image-text foundation models.
*arXiv preprint arXiv:2205.01917*, 2022.
- Yuan et al. (2025)
Ruibin Yuan, Hanfeng Lin, Shuyue Guo, Ge Zhang, Jiahao Pan, Yongyi Zang, Haohe Liu, Yiming Liang, Wenye Ma, Xingjian Du, et al.
Yue: Scaling open foundation models for long-form music generation.
*arXiv preprint arXiv:2503.08638*, 2025.
- Yuan et al. (2024)
Yi Yuan, Zhuo Chen, Xubo Liu, Haohe Liu, Xuenan Xu, Dongya Jia, Yuanzhe Chen, Mark D Plumbley, and Wenwu Wang.
T-clap: Temporal-enhanced contrastive language-audio pretraining.
In *2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP)*, pp. 1–6. IEEE, 2024.
- Zhai et al. (2023)
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.
Sigmoid loss for language image pre-training.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 11975–11986, 2023.
- Zhu et al. (2024)
Ge Zhu, Jordan Darefsky, and Zhiyao Duan.
Cacophony: An improved contrastive audio-text model.
*IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 2024.
- Zhuang et al. (2025a)
Xianwei Zhuang, Yuxin Xie, Yufan Deng, Liming Liang, Jinghan Ru, Yuguo Yin, and Yuexian Zou.
Vargpt: Unified understanding and generation in a visual autoregressive multimodal large language model.
*arXiv preprint arXiv:2501.12327*, 2025a.
- Zhuang et al. (2025b)
Xianwei Zhuang, Yuxin Xie, Yufan Deng, Dongchao Yang, Liming Liang, Jinghan Ru, Yuguo Yin, and Yuexian Zou.
Vargpt-v1. 1: Improve visual autoregressive large unified model via iterative instruction tuning and reinforcement learning.
*arXiv preprint arXiv:2504.02949*, 2025b.

## Appendix A Related Work

Learning joint representations for audio and text has become a pivotal area of research, enabling applications from cross-modal retrieval Elizalde et al. (2023) to text-to-audio generation Yuan et al. (2025). A dominant paradigm in this domain is contrastive learning, which aims to create a unified embedding space where audio and its corresponding text description are brought closer together, while dissimilar pairs are pushed apart. Drawing inspiration from the success of CLIP Radford et al. (2021); Zhuang et al. (2025b) in the vision-language domain Yu et al. (2022); Zhuang et al. (2025a), the Contrastive Language-Audio Pretraining (CLAP) framework based on InfoNCE loss has emerged as a foundational approach Wu et al. (2022); Guzhov et al. (2022). Researchers have since focused on refining and extending the capabilities of CLAP. For instance, to better manage audio inputs of varying durations and boost overall performance, Wu et al. (2023) integrated a feature fusion mechanism with a keyword-based description enhancement strategy. Others have focused on improving the model’s fine-grained understanding. Silva et al. (2023) introduced COLLAT, a framework that achieves nuanced audio comprehension by freezing the language model’s parameters and training the audio encoder with a specialized audio-text alignment objective. Similarly, to address the challenge of distinguishing between closely related but distinct sounds, such as hard negative samples. Ghosh et al. (2023) developed a modular contrastive loss designed to improve the model’s discriminative capabilities. In subsequent work, Ghosh et al. (2025) further explored the impact of textual data quality, demonstrating that rewriting audio captions to be more descriptive significantly enhances the model’s comprehension of real-world acoustic scenes. Beyond these, domain specialization has gained attention. Liu et al. (2024) proposed DSCLAP for domain-specific audio-text pre-training, emphasizing tailored representations for specialized contexts. Meanwhile, Zhu et al. (2024) introduced Cacophony, which strengthens retrieval with auxiliary objectives.

Existing works have explored integrating principles from Support Vector Machine (SVM) Hearst et al. (1998) into contrastive learning from different perspectives. Existing approaches include Max-Margin Contrastive Learning Shah et al. (2022), which adapts the SVM max-margin principle to identify and push away hard negative samples treated as support vectors. SV-Learner Liang et al. (2024) is another approach that employs SVM to select reliable data pairs for contrastive learning, thereby enhancing robustness in noisy-label scenarios. Concurrently, the T-MASS Wang et al. (2024) approach also utilizes a support vector to model text as a stochastic mass, aiming to enrich text’s semantic representation for text-video retrieval. Compared to the above approaches, our work SupCLAP introduces Support Vector Regularization (SVR) to address a different core problem: controlling the optimization trajectory drift caused by the perpendicular pushing force from negative samples in contrastive learning.

## Appendix B Constraint Term in DynamicSVR

To improve the stability and precision of the predicted semantic radius $R$, we propose a constraint term $L_{cons}$ on the radius predictor module. While the adaptive predictor offers flexibility, an unconstrained radius vector $R$ could lead to two potential failure modes:

- •
Excessive Magnitude: A semantic radius $R$ with an excessively large magnitude (i.e., $R\gg||a^{+}-t^{+}||$) causes the factor $(1-\frac{R}{||a^{+}-t^{+}||})$ to become negative. This may lead to a directional inversion of the perpendicular component representing the additional information of the negative samples. As a result, the effective utilization of this information is impeded, which in turn leads to optimization instability.
- •
Counterproductive Direction: Without constraints on the semantic radius $R$ during unsupervised modeling, the radius predictor may output $R\textless 0$, which results in the factor $(1-\frac{R}{||a^{+}-t^{+}||})\textgreater 1$. This adversely magnifies the perpendicular component, thereby aggravating the problem of optimization trajectory drift.

To simultaneously address both issues, we formulate the constraint term $L_{cons}=Relu(R-||a^{+}-t^{+}||)+Relu(-R)$. This term is composed of two components, each targeting one of the aforementioned failure modes. The first $Relu(R-||a^{+}-t^{+}||)$ directly penalizes large magnitudes to prevent overshooting. The second, $Relu(-R)$, explicitly encourages directional alignment with the pulling force. By incorporating this constraint, the total loss function becomes:

$$ $\displaystyle L_{SupCLAP-Cons}=L_{orig}+\alpha L_{svr}+\beta L_{cons}.$ (9) $$

These constraints ensure that the support vector provides a stable and meaningful regularization signal, transforming SVR into a more robust and effective fine-tuning mechanism. We use a default value of $\beta=0.01$ for the constraint weight, which is intended to impose a slight penalty, ensuring the semantic radius $R$ remains within a reasonable range without dominating the entire loss function.

## Appendix C Analysis of SVR’s Mechanism

In this section, we analyze the gradient space of the SVR term to show that SVR can effectively suppress the perpendicular component and improve the accuracy of the learned representations in theory.

The support vector is located in the direction pointing from text embedding $t^{+}$ to positive audio embedding $a^{+}$ and lies on the surface of the text embedding distribution centered at text embedding $t^{+}$. Its mathematical expression is $t_{sup}=t^{+}+R\frac{a^{+}-t^{+}}{||a^{+}-t^{+}||}=t^{+}+R\hat{u}$, where $\hat{u}=\frac{a^{+}-t^{+}}{||a^{+}-t^{+}||}$ is the unit vector in the pulling direction and $R$ is an adaptive and learnable radius.

The gradient of SVR term in equation [6](https://arxiv.org/html/2509.21033v1#S3.E6) is computed by the chain rule:

$$ $\displaystyle\nabla_{t}L_{svr}=\bigg(\frac{\partial t_{sup}}{\partial t^{+}}\bigg)^{T}\nabla_{t_{sup}}L_{svr}.$ (10) $$

The Jacobian matrix $\frac{\partial t_{sup}}{\partial t^{+}}$ is computed as

$$ $\displaystyle\frac{\partial t_{sup}}{\partial t^{+}}$ $\displaystyle=\frac{\partial}{\partial t^{+}}\bigg(t^{+}+R\frac{a^{+}-t^{+}}{||a^{+}-t^{+}||}\bigg)$ (11) $\displaystyle=I+R\frac{\partial}{\partial t^{+}}\bigg(\frac{a^{+}-t^{+}}{||a^{+}-t^{+}||}\bigg).$ $$

After derivation, we get $\frac{\partial\hat{u}}{\partial t^{+}}=\frac{1}{||a^{+}-t^{+}||}(\hat{u}\hat{u}-I)$. The matrix $(\hat{u}\hat{u}-I)$ is a projection operation that projects any vector onto the hyperplane orthogonal to $\hat{u}$. We denote $P_{\perp}=I-\hat{u}\hat{u}$. Therefore, the Jacobian matrix is denoted as

$$ $\displaystyle\frac{\partial t_{sup}}{\partial t^{+}}=I-\frac{R}{||a^{+}-t^{+}||}P_{\perp}.$ (12) $$

We further analyze the pushing force component from a single negative example $a^{-}_{j}$, denoted as $f_{sup,push,j}$, within the gradient generated by $L_{svr}$ term:

$$ $\displaystyle f_{sup,push,j}$ $\displaystyle=\alpha\bigg(\frac{\partial t_{sup}}{\partial t^{+}}\bigg)^{T}\bigg(\frac{P_{sup,j}^{-}}{\tau}a^{-}_{j}\bigg)$ (13) $\displaystyle=\alpha\frac{P^{-}_{sup,j}}{\tau}\bigg(a^{-}_{j}-\frac{R}{||a^{+}-t^{+}||}P_{\perp}a^{-}_{j}\bigg)$ $\displaystyle=\alpha\frac{P^{-}_{sup,j}}{\tau}\bigg((a^{-}_{||,j}+a^{-}_{\perp,j})-\frac{R}{||a^{+}-t^{+}||}a^{-}_{\perp,j}\bigg)$ $\displaystyle=\alpha\frac{P^{-}_{sup,j}}{\tau}\bigg(a^{-}_{||,j}+(1-\frac{R}{||a^{+}-t^{+}||})a^{-}_{\perp,j}\bigg),$ $$

where $\alpha$ denotes the weight of the SVR term in the total loss, probability $P_{sup,j}^{-}$ can be calculated in equation [3](https://arxiv.org/html/2509.21033v1#S2.E3) by replacing text embedding $t^{+}$ to text support vector $t_{sup}$. We decompose the negative audio embedding $a^{-}_{j}$ into a parallel component $a^{-}_{||,j}$ and perpendicular component $a^{-}_{\perp,j}$ to the unit vector $\hat{u}$, denoted as $a^{-}_{j}=a^{-}_{||,j}+a^{-}_{\perp,j}$. It should be noted that $P_{\perp}a^{-}_{j}=(I-\hat{u}\hat{u})a^{-}_{j}=a^{-}_{\perp,j}$.

This result is the core of the proof. It shows that for the pushing force generated by the support vector loss, its parallel component $a^{-}_{||,j}$ remains unchanged, while its perpendicular component $a^{-}_{\perp,j}$ is scaled by a factor of $(1-\frac{R}{||a^{+}-t^{+}||})$.

We examine the parallel component of the total pushing force from negative example $j$, denoted as $F_{||,push,j}$:

$$ $\displaystyle F_{||,push,j}$ $\displaystyle=f_{||,j}+f_{sup,||,j}$ (14) $\displaystyle=\frac{P^{-}_{j}}{\tau}a^{-}_{||,j}+\alpha\frac{P^{-}_{sup,j}}{\tau}a^{-}_{||,j}$ $\displaystyle=\bigg(\frac{P^{-}_{j}}{\tau}+\alpha\frac{P^{-}_{sup,j}}{\tau}\bigg)a^{-}_{||,j}.$ $$

And the perpendicular component $F_{\perp,push,j}$ of the total pushing force from negative example $j$ can be computed as:

$$ $\displaystyle F_{\perp,push,j}$ $\displaystyle=f_{\perp,j}+f_{sup,\perp,j}$ (15) $\displaystyle=\frac{P^{-}_{j}}{\tau}a^{-}_{\perp,j}+\alpha\frac{P^{-}_{sup,j}}{\tau}\bigg(1-\frac{R}{||a^{+}-t^{+}||}a^{-}_{\perp,j}\bigg)$ $\displaystyle=\bigg[\frac{P^{-}_{j}}{\tau}+\alpha\frac{P^{-}_{sup,j}}{\tau}\bigg(1-\frac{R}{||a^{+}-t^{+}||}\bigg)\bigg]a^{-}_{\perp,j}.$ $$

The factor $(1-\frac{R}{||a^{+}-t^{+}||})$ plays a key modulating role to control the optimization trajectory drift. For most of the training process, the text embedding $t^{+}$ is at a certain distance from its positive example $a^{+}$, such that $||a^{+}-t^{+}||\textgreater R$. In this scenario, the factor is a positive number less than 1, which means the support vector regularization term actively weakens this drift caused by the perpendicular component. Consequently, the overall gradient direction points more closely toward the pulling force while adaptively maintaining enough additional information in negative samples, making the optimization path from text embedding $t^{+}$ to positive audio embedding $a^{+}$ more direct and stable. This reduces unnecessary drift, thereby accelerating the convergence process and potentially leading to better alignment.

## Appendix D Additional Experiment

### D.1 Experiment Result under Multilingual Scenario

**Table 4: Recall and precision results under multilingual AudioCaps and Clotho dataset**
| Model | AudioCaps | Clotho |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| T2A | A2T | T2A | A2T |  |  |  |  |  |
| R@1 | R@10 | R@1 | R@10 | R@1 | R@10 | R@1 | R@10 |  |
| SigLIP | 34.40 | 82.34 | 44.26 | 86.24 | 13.62 | 50.53 | 17.50 | 53.36 |
| -bi-StaticSVR | 36.87 | 84.23 | 49.54 | 89.08 | 15.17 | 52.32 | 18.70 | 57.00 |
| -bi-DynamicSVR | 38.56 | 85.20 | 51.46 | 90.68 | 15.82 | 54.36 | 21.22 | 58.20 |
| InfoNCE (ML-CLAP) | 37.20 | 84.79 | 50.20 | 90.68 | 17.10 | 55.79 | 21.50 | 58.80 |
| -bi-StaticSVR | 39.60 | 84.98 | 52.36 | 91.05 | 17.28 | 56.06 | 22.37 | 59.42 |
| -bi-DynamicSVR | 39.75 | 85.52 | 53.99 | 91.22 | 17.49 | 56.35 | 22.43 | 60.04 |

Multilingual Retrieval: To further validate the effectiveness of our proposed framework against the standard InfoNCE and SigLIP Loss, we conducted an experimental evaluation in a multilingual scenario. Following the scheme of ML-CLAP Yan et al. (2024), we dynamically pair each audio sample with a random text caption from one of eight languages during multilingual training. For the sake of conciseness, the metrics we present are averaged across all eight languages. Consistent with our results in the monolingual scenario, the results in Table [4](https://arxiv.org/html/2509.21033v1#A4.T4) show that both the bi-StaticSVR and bi-DynamicSVR methods can still effectively improve the performance of standard contrastive learning and bi-DynamicSVR achieves the best performance. These results validate the generalizability of our SVR term in complex multilingual scenarios. We also evaluate the quality of the translated multilingual test set in Appendix [D.2](https://arxiv.org/html/2509.21033v1#A4.SS2).

### D.2 Quality Assessment of Multilingual Test Set

**Table 5: Semantic Similarity between Re-translated Text and Original English Text on Test Set**
| Language | AudioCaps | Clotho |
| --- | --- | --- |
| French | 94.5 | 93.5 |
| Dutch | 95.0 | 94.2 |
| Spanish | 94.8 | 94.0 |
| German | 95.6 | 94.2 |
| Catalan | 91.1 | 92.2 |
| Japanese | 91.7 | 90.9 |
| Chinese | 90.7 | 90.3 |

To verify the semantic fidelity of our LLM-translated multilingual Audiocap and Clotho test set and thus substantiate our experimental results, we performed a back-translation analysis. We first translated the test captions back to English using Deepseek V3. We then computed the average cosine similarity between the embeddings of the back-translated and original English texts via the Roberta-large model. The results, detailed in Table [5](https://arxiv.org/html/2509.21033v1#A4.T5), reveal an average similarity exceeding 90% for all languages. This high fidelity demonstrates strong semantic preservation and suggests that translation errors in our test set are sufficiently small.

### D.3 Additional Ablation Study

**Table 6: Ablation Study of SVR Variants on Multilingual Text-Audio Retrieval.**
| ID | Model | T2A | A2T |  |  |
| --- | --- | --- | --- | --- | --- |
| R@1 | mAP10 | R@1 | mAP10 |  |  |
| 0 | InfoNCE | 37.20 | 52.19 | 50.20 | 28.28 |
| 1 | -bi-DynamicSVR | 39.75 | 54.22 | 53.99 | 32.82 |
| 2 | -bi-DynamicSVR wo/ constraints | 39.59 | 54.16 | 53.39 | 32.42 |
| 3 | -uni-DynamicSVR | 39.52 | 54.08 | 53.26 | 32.26 |
| 4 | -uni-DynamicSVR wo/ constraints | 39.37 | 53.81 | 53.07 | 31.78 |
| 5 | -bi-StaticSVR | 39.60 | 53.99 | 52.36 | 31.59 |
| 6 | -uni-StaticSVR | 39.51 | 53.77 | 52.06 | 30.70 |

Effectiveness of SVR Components: The multilingual ablation study results in Table [6](https://arxiv.org/html/2509.21033v1#A4.T6) are consistent with the monolingual scenario in Table [3](https://arxiv.org/html/2509.21033v1#S4.T3); in both cases, the bidirectional DynamicSVR model with constraints (bi-DynamicSVR) achieves the best results. The performance ranking from highest to lowest is bidirectional SVR, unidirectional SVR, and InfoNCE.

**Table 7: Ablation study on different batch sizes for T2A and A2T tasks**
| Model | Batch Size | T2A | A2T |  |  |
| --- | --- | --- | --- | --- | --- |
| R@1 | mAP10 | R@1 | mAP10 |  |  |
| InfoNCE | 24 | 37.20 | 52.19 | 50.20 | 28.28 |
| 48 | 39.32 | 53.95 | 53.28 | 31.99 |  |
| 72 | 40.22 | 54.85 | 54.34 | 32.93 |  |
| -bi-StaticSVR | 24 | 39.60 | 53.99 | 52.36 | 31.59 |
| 48 | 40.54 | 55.01 | 54.21 | 32.92 |  |
| 72 | 40.89 | 55.40 | 54.87 | 33.47 |  |
| -bi-DynamicSVR | 24 | 39.75 | 54.22 | 53.99 | 32.82 |
| 48 | 40.83 | 55.20 | 54.31 | 33.25 |  |
| 72 | 40.92 | 55.57 | 55.00 | 33.82 |  |

Impact of Batch Size: Table [7](https://arxiv.org/html/2509.21033v1#A4.T7) shows that the performances of both InfoNCE, bi-StaticSVR, and bi-DynamicSVR improve as the batch size increases (24, 48, 72). This is consistent with the principle that larger batches provide more diverse negative samples for contrastive learning, which in turn leads to better performance. Both bi-StaticSVR and bi-DynamicSVR consistently maintain a stable performance advantage over InfoNCE, indicating that our methods can further enhance the performance of InfoNCE with larger batch sizes.

**Table 8: Ablation of SVR weight $\alpha$**
| $\alpha$ | T2A | A2T |  |  |
| --- | --- | --- | --- | --- |
| R@1 | mAP10 | R@1 | mAP10 |  |
| 0.1 | 38.69 | 53.38 | 52.05 | 30.48 |
| 0.5 | 39.42 | 53.90 | 52.28 | 31.43 |
| 1.0 | 39.75 | 54.22 | 53.99 | 32.82 |

Impact of SVR Weight $\alpha$: The results in Table [8](https://arxiv.org/html/2509.21033v1#A4.T8) indicate that model performance consistently improves as $\alpha$ increases from 0.1 to 1.0. This suggests that a greater weight for the SVR module significantly enhances the model’s feature representation or matching capabilities. Therefore, the experiment identifies $\alpha=1.0$ as the optimal hyperparameter setting for the current configuration.

### D.4 Overhead Analysis

**Table 9: Evaluation results in GPU memory overheads and time overheads**
| Scheme | AudioCaps | Clotho |  |  |
| --- | --- | --- | --- | --- |
| GMO(MB) | TO(s) | GMO(MB) | TO(s) |  |
| InfoNCE | 22724 | 2479 | 30318 | 1102 |
| -bi-StaticSVR | 22732 | 2438 | 30368 | 1127 |
| -bi-DynamicSVR | 22834 | 2519 | 30398 | 1116 |

In Table [9](https://arxiv.org/html/2509.21033v1#A4.T9), we evaluate the time overhead (TO) and GPU memory overhead (GMO) of SupCLAP compared to InfoNCE. TO denotes the average training time over 10 epochs, and GMO denotes the peak GPU memory usage during the training process. The results show that the two SupCLAP variants achieve performance gains with almost no additional computational overhead, possessing both high efficiency and practical viability.

## Appendix E Statement on The Use of Large Language Models

We utilized a large language model (LLM) as a general-purpose writing assistant to enhance the clarity and logical structure of the manuscript. This included polishing sentences, improving transitions, and refining the overall flow. All LLM-assisted edits in this paper were meticulously checked and validated by us.

Furthermore, we employed an LLM to construct a multilingual AudioCaps and Clotho dataset. To validate the quality of the translations, we evaluated the semantic similarity between the original English texts and their translated counterparts in Appendix [D.2](https://arxiv.org/html/2509.21033v1#A4.SS2). This was accomplished by calculating the cosine similarity of their embeddings, which were generated using a RoBERTa-large model.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Given that the SVR method introduces additional parameters that require careful tuning, how does this impact the overall efficiency of the method during training and implementation? Specifically, what are the trade-offs between the improved performance and the potential increases in optimization time?
soundness: 3
strengths: Addresses Optimization Trajectory Drift: The proposed Support Vector Regularization (SVR) method effectively controls optimization trajectory drift caused by the influence of negative samples, enhancing training stability.

Performance Improvement: SupCLAP significantly outperforms various existing baselines across tasks such as classification and monolingual/multilingual retrieval, demonstrating its superior performance in multimodal learning.

Exploration of Unsupervised Modeling Strategies: The study explores both static parameterization and adaptive radius prediction as unsupervised modeling strategies, providing new methodological insights for future research.
summary: The paper addresses the issue of optimization trajectory drift during contrastive learning due to the perpendicular component of pushing force from negative samples. It highlights how this aspect can lead to instability in training and proposes a novel solution called Support Vector Regularization (SVR).

SVR introduces an auxiliary support vector that allows for better control of negative sample influence, leveraging their informative content while mitigating trajectory drift. The effectiveness of SVR is critically tied to the selection of the semantic radius, and the paper explores two unsupervised modeling strategies: static direct parameterization and an adaptive radius predictor module with constraints.

Extensive experimental results demonstrate that SupCLAP significantly outperforms various baseline methods across tasks such as classification and both monolingual and multilingual retrieval, validating its potential in stabilizing optimization paths and improving model performance.
weaknesses: Limited Generalizability Beyond Benchmarks: While the performance in the tested tasks is impressive, the study may have limitations in generalizability. The results may not translate well to all real-world applications, particularly those outside the tested domains or with different data characteristics.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: - Motivation is reasonably strong, and mathematical rigor to the proposed method is appreciated.
- Despite the overall mathematical maturity of SupCLAP, the proposed method is still intuitive and straightforward, aiding the clarity of the paper.
- The experimental results, while generally modest, show clear gains to the using the proposed method relative to the baselines.
summary: This paper presents SupCLAP, a method for regularizing the contrastive learning optimization process of standard objectives (InfoNCE, SigLIP) to improve performance on downstream retrieval and 0-shot classification tasks.
weaknesses: In general, there is not much massively wrong with the present manuscript, my two biggest concerns are:
- It would be useful to report confidence intervals for the retrieval / classification experiments, as the results are generally modest, and it is hard to tell whether this gain in performance is truly statistically significant.
- Overall, there is not much clear evidence as to *why* SupCLAP seems to work better empirically. While there is ample connection to *in theory* what might be going on during optimization, this is never attempted to be empirically confirmed. It would be very useful to analyze parts of the optimization process for SupCLAP (vs InfoNCE or SigLIP) such as gradient norm, gradient variance, or simply convergence speed to see whether the surmised theoretical reasons for its performance actually translate in practice.

---

confidence: 2
contribution: 2
initial_rating: 4
presentation: 3
questions: See weakness.
soundness: 2
strengths: This paper gives a novel regularization term on the InfoNCE loss,  their experiments show that this regularization indeed bring benefits. They also provide their intuition in section 2.  And The writing of this paper is good.
summary: The paper proposes SupCLAP with a Support Vector Regularization (SVR) that shifts the text embedding towards its positive audio.They add an extra contrastive term, claiming to suppress the perpendicular component of the negative “pushing force” and reduce optimization trajectory drift.  And this will help improve performance.
weaknesses: 1. In the section 2, the authors discuss the optimization trajectory drift. However, the causal link “lower perpendicular component → better performance” is not theoretically or empirically established. So this section is not very convincing to me. The author may provide theoretical or empirical evidence.
2. For the InfoNCE loss, we originally want the text embedding being away from the negative sample. So why we need to reduce the perpendicular component of the negative “pushing force”? 
3. Moreover, the proposed Support-Vector Regularization (SVR) seems closely related to simply re-scaling or re-weighting the contrastive gradients. It would strengthen the paper if the authors compared their method to simpler baselines—e.g., emphasizing the parallel component only or giving positive sample gradient more weights—to demonstrate that the observed improvement is not merely a side-effect of gradient magnitude control.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. The dynamic semantic radius relies on an unsupervised predictor. How robust is it under noisy embeddings (e.g., low-resource or weak encoders)? Can the authors show sensitivity or variance analysis to demonstrate stability?

2. The radius shrinks during training, but the explanation is heuristic. Can the authors provide clearer evidence that its values correlate with semantic difficulty (e.g., hard-negative density or retrieval errors)?
soundness: 2
strengths: 1. The work offers a fresh angle by decomposing contrastive gradients into pulling and perpendicular pushing components, which provides new insights into training instability in CLAP.
2. The SVR formulation is presented with solid mathematical justification, and the derivations clearly show how trajectory drift is effectively suppressed.
3. Experiments across retrieval, classification, and multilingual tasks, along with ablations, demonstrate consistent improvements while keeping the computational cost minimal.
summary: This paper studies contrastive language–audio pretraining (CLAP) and identifies a phenomenon called optimization trajectory drift, caused by the perpendicular component of the negative-sample pushing force in contrastive learning. The authors propose Support Vector Regularization (SVR), which introduces auxiliary support vectors and a semantic radius to selectively suppress this perpendicular component while preserving useful negative-sample information. They present both static and dynamic strategies for modeling the semantic radius. Experiments across monolingual and multilingual audio–text retrieval, as well as zero-shot classification, show consistent improvements over InfoNCE and SigLIP with negligible overhead.
weaknesses: 1. While the paper offers a new analytical angle, the proposed SVR is similar to a regularization tweak of InfoNCE and conceptually close to earlier margin-based or support-vector–style methods.

2. The dynamic semantic radius predictor is unsupervised and fragile under noise. Its meaning is not clearly explained, and why it shrinks during training or how it reflects semantic difficulty is described more heuristically than rigorously.

3. The baseline setup makes comparisons harder to follow. In Tables 1 and 2, many models are cited only by reference instead of name. Some strong recent baselines are also missing, including DS-CLAP (Liu et al., 2024), Cacophony (Zhu et al., 2024), and ATRI (Yin et al., 2025).

4. The experiments are limited to AudioCaps and Clotho, with multilingual results relying on machine translation. Compared with recent work that uses larger datasets such as AudioSet, WavCaps, and LAION-Audio and covers broader tasks like captioning or grounding, the generalization claims are less convincing.


[a] Liu et al., 2024, DSCLAP: Domain-specific contrastive language-audio pre-training

[b] Zhu et al., 2024, Cacophony: An improved contrastive audio-text model

[c] Yin et al., 2025, ATRI: Mitigating multilingual audio-text retrieval inconsistencies by reducing data distribution errors


Decision: Accept (Poster)