SupCLAP: Controlling Optimization Trajectory Drift in Audio-Text Contrastive Learning with Support Vector Regularization
Authors: Jiehui Luo
Yuguo Yin
Yuxin Xie
Jinghan Ru
Xianwei Zhuang
Minghua He
Aofan Liu
Zihan Xiong
Dongchao Yang
Venue: ICLR 2026 Poster (2026)
Keywords: contrastive learning; audio-text retrieval; representation learning
Primary area: learning theory

Abstract: Contrastive language-audio pretraining, which aims to unify multimodal representations in a shared embedding space, serves as a cornerstone for building a wide range of applications, from cross-modal retrieval to cutting-edge multimodal large language models. However, we find that the perpendicular component of the pushing force from negative samples in contrastive learning is a double-edged sword: it contains rich supplementary information from negative samples, yet its unconstrained nature causes optimization trajectory drift and training instability. To address this, we propose Support Vector Regularization (SVR), a method that introduces an auxiliary support vector to control this perpendicular component, aiming to harness its rich information while mitigating the associated trajectory drift. The efficacy of SVR is critically governed by its semantic radius, for which we explore two unsupervised modeling strategies: direct parameterization and an adaptive radius predictor module enhanced with constraints to improve its predicting accuracy. Extensive experimental results demonstrate that our method surpasses widely used baselines like InfoNCE and SigLIP loss across classification, monolingual retrieval, and multilingual retrieval on standard audio-text datasets. Both the theoretical analysis and the experimental results on optimizing trajectory drift validate the correctness and effectiveness of our SVR method. Notably, our method is highly efficient, it operates without the need for extra training data or inference computation, and adds only a negligible overhead to the training.

Published as a conference paper at ICLR 2026
SUPCLAP: CONTROLLING OPTIMIZATION TRAJEC- TORY DRIFT IN AUDIO-TEXT CONTRASTIVE LEARN- ING WITH SUPPORT VECTOR REGULARIZATION
Jiehui Luo"; Yuguo Yin Yuxin Xie', Jinghan Ru', Xianwei Zhuang', Minghua He Aofan Liu', Zihan Xiong', Dongchao Yang*
!Peking University, CCntral Conservatory of Music, Thhe Chinese University of Hong Kong, UUniversity of Electronic Science and Technology of China luo luojiehul @mail ccom. edu. en, yuguoy wwwplyyinn0220ggm @rno022cmmaiio com
ABSTRACT
Contrastive language-audio pretraining, which aims to unify multimodal repre- sentations in shared embedding space, serves as comerstone for building a wide range of applications, from cross-modal retrieval to cutting-edge multimodal large language models. However, we find that the perpendicular component of the pushing force from negative samples contrastive learning is a double-edged sword: it contains rich supplementary information from negative samples, yet its unconstrained nature causes optimization trajectory drift and training instability. To address this, we propose Support Vector Regularization (SVR). method that introduces an auxiliary support vector to control this perpendicular component, aiming harness its rich information while mitigating the associated trajectory drift. The efficacy SVR is critically governed by its semantic radius, for which we explore two unsupervised modeling strategies: direct parameterization and an adaptive radius predictor module enhanced with constraints to improve its pre- dicting accuracy. Extensive experimental results demonstrate that our method sur- passes widely used baselines like InfoNCE and SigLIP loss across classification, monolingual retrieval, and multilingual retrieval on standard audio-text datasets. Both the theoretical analysis and the experimental results optimizing trajectory drift validate the correctness and effectiveness of our SVR method. Notably, our method is highly efficient, it operates without the need for extra training data inference computation, and adds only a negligible overhead to the training.
INTRODUCTION
Contrastive Language-Audio Pretraining (CLAP) Wu al. (2023); Ghosh et al. (025 aims to learn unified audio-text embedding space by pulling corresponding pairs closer and pushing others apart. This paradigm. which powers applications like cross-modal retrieval Xie al. (2024) and multimodal L. Xue et al. (2024); Lam et al. (2025). has achieved great empirical success. How- ever, standard InfoNCE- CLAP methods still struggle to learn ideal representations, facing limitations such as poor temporal alignment of audio events Yuan et al. (2024) and inconsistent mul- tilingual alignment Yin et al. (2025). Therefore, achieving optimal alignment between the language and audio representation spaces remains open challenge.
In this paper, we uncover a complex yet overlooked dynamic in the optimization process standard InfoNCE-based contrastive learning Wu et al. (2021): optimization trajectory drift. We conceptu- alize the contrastive learning process as an interplay between a "pulling force" from positive pairs and "pushing force" from negative pairs within the embedding space. According to the analysis in Section
D and the experimental results in Figure T we find that this pushing force is generally not collinear with the pulling force. This phenomenon stems from both the inherent structure of the data distribution and the stochasticity of mini-batch sampling. The resulting perpendicular component of
Jiehui Luo and Yuguo Yin contributed equally.
Yuguo Yin is the corresponding author.

---

Published as a conference paper at ICLR 2026
InfeNCE MONNCE-bl Dynamc IntorCE--bi Static
Cosine Simiaaity
2.38
Fooch
Figure Optimization Trajectory Drift Analysis. Drift is measured by the cosine similarity between the update vector and the 'pulling force' vector; a higher similarity indicates lower drift. Compared to InfoNCE loss, our SVR method effectively mitigates this drift. This result confirms the existence of optimization trajectory drift.
the pushing force is a double-edged sword. On the one hand, it contains supplementary information from negative samples with rich learning signals. On the other hand, its uncontrolled and fluctuating nature exerts sideways force, causing the optimization trajectory to drift. This instability not only slows convergence but also hinders the final alignment quality.
To address the optimization trajectory drift problem, we perform detailed theoretical and quali- tative analysis of the optimization direction of standard InfoNCE-based CLAP methods. Based on this analysis, we design a CLAP framework called SupCLAP, which leverages proposed Sup- port Vector Regularization (SVR) method. The SVR method introduces a new regularization term to the training objective, formulated as additional contrastive loss computed between the audio embeddings and newly constructed text support vectors. These support vectors are created by dis- placing original text embeddings toward their positive audio embedding. The magnitude of this displacement is determined by a semantic radius R, which we model in an unsupervised manner. Our analysis demonstrates that SVR effectively reshapes the gradient space. adaptively suppressing the perpendicular component while retaining sufficient supplementary information from negative samples. This yields more stable optimization trajectory, leading to improved alignment quality.
Building on our analysis for SVR, we further posit that more precise semantic radius R enables finer control over the perpendicular component, directly enhancing SVR's efficacy. To this end, we explore its unsupervised modeling through two primary strategies. The first strategy is StaticSVR, where the radius is treated as a learnable parameter. The other the DynamicSVR, which uses an adaptive radius predictor module that utilizes embedding similarity information in the mini-batch to predict the semantic radius R. We further propose constraints for the radius predictor module to enhance the quality and stability of the modeled semantic radius. Our main contributions are listed as follows:
We find and analyze the optimization trajectory drift problem in contrastive learning from force decomposition perspective, linking it to the perpendicular component of the pushing force from negative samples.
We propose the SupCLAP scheme with Support Vector Regularization (SVR) and rigor- ously justify its ability to control optimization trajectory drift. By reshaping the gradient landscape, SVR leads to more stable and effective audio-text contrastive lcarning.
We conduct a systematic exploration into unsupervised semantic radius modeling, propos- ing two strategies: StaticSVR and DynamicSVR. Furthermore, we propose constraints for the DynamicSVE to improve the quality the predicted radius, validating our hypothesis the importance of radius precision.
Extensive experiments diverse audio-text benchmarks demonstrate that SupCLAP sig- nificantly outperforms baseline methods such as InfoNCE and SigLIP loss, notably requir- ing additional training data or inference overhead while incurring only negligible train- ing costs. Furthermore, these results provide strong validation for our theoretical analysis of optimization trajectory drift and the effectiveness of SupCLAP..
2

---

Published as a conference paper at ICLR 2026
2 ANALYSIS OF OPTIMIZATION TRAJECTORY DRIFT
In this section, analyze the gradient space of contrastive learning by decomposing its gradient into a pulling force from the positive audio embedding and pushing force from the negative audio embeddings. We then further analyze how the component of the pushing force perpendicular to the pulling force affects model optimization, to clearly illustrate the optimization trajectory drift problem.
2.1 CONTRASTIVE LEARNING LOSS FUNCTION AND GRADIENT
First, we define the basic framework for contrastive learning. Assume we have text embedding + E IRd matching positive audio embedding at E Rd, where d is the size of embedding, and
N-11 To simplify the derivation, batch of N - 1 mismatched negative audio embeddings {az fj=1
assume all embedding vectors 1.2-normalized, i.e., ||||| ||a|| = 1. The similarity function used
the scaled dot product (Scaled Cosine Similarity): s(a, t) = cos(a, t)/T e"tt where is the temperature hyperparameter.
The standard InfoNCE loss function Koromilas et al. 2024) (for text-to-audio, as an example) is:
exp(s(t*, a)))
Aotsg(tt.on (a, }) log
exp(s(tt, at)) Σ exp(s(t*.a7)
To analyze the optimization direction, we calculate the gradient of the loss function to the text embedding tt, denoted as VrCorig. We define:
exp(s(t*,.''
pt exp(s(tt,at)) ΣØ= (xp(s((t,.5))
(2)
(ooss(tta,))) PJ exp(s(tt at)) ΣØ= exp(s(tt, ,77))
(3)
where pt E
According to the chain rule, V,s(t, a/T. The gradient of loss function can be derived as:
N+1
VeLorio
Σ
Pjaj + ptatt)
-at
7
(4)
N-1
1
(pt 1)at Σ P7 aj
(pt
7
j=l
We decompose the gradient into two parts:
Pulling Force:
- 1)a+. Since 0<P+<1. the term (P+ 1)) is negative. In gradient descent, we update text embedding + in the direction of VV (i.e., t+ +- ++ V Lorigg). Therefore, this term is equivalent to force that pulls text embedding + towards positive audio embedding at.
Fpuli = (Pp
Pushing Force:
Fpush , 1 a5 This term weighted average of all negative example embeddings, and its effect is to push the text emhedding ++ away from all negative audio embeddings a, It should also be noted that negative audio embedding dg with high similarity to the text embedding t will yield a larger probability P thereby exerting greater pushing force on the text embedding.

---

Published as a conference paper at ICLR 2026
2.1.1 SUPPRESS TRAJECTORY DRIFT PROBLEM
We find that the pushing force Frush is generally not parallel the direction of the pulling force F Pushh We decompose the pushing force from single negative example aj which is given by pushing force fpush. 4 Let us define the unit vector in the pulling direction as â at--t
- The negative pushing force fpush.s can be decomposed into component parallel fil.i û and component perpendicular fLtj û, which denoted spash.j fiI. + ر.ي The parallel component is computed fII.s = (ppush.j üju while the perpendicular component fi. fpnch. fils- = fpush. ûū).
During the optimization process, the parallel component fit, shares the same direction as the pulling force from the positive sample, differing only in magnitude. It thus primarily affects convergence speed and contains little additional information from the negative samples beyond what is already present in the pulling force. In contrast, the gradient direction of the perpendicular component fi.j rich with supplementary information from the negative samples, which is distinct from that of the positive sample. But this perpendicular component acts double-edged sword: while its direction provides additional information, its uncontrolled magnitude can cause the optimization path to drift continuously. The nature of this drift can be understood on both global and local levels, which is illustrated in Figure 2
We put the detailed analysis the trajectory drift problem in Appendix C
Negative Andio Embrdding in Diffearent Batches
Text Embodding
Pushing Force
Perpendicular Component
Positive Audio Embedding
Pulling Force
Optimization Trajectory
Global Perpendicular Component
Local Perpendicular Component
Figure 2: Illustration of Global and Local Perpendicular Components. The left subfigure depicts the global perpendicular component. The subfigure on the right illustrates the local perpendicular component. For clarity in demonstrating the local perpendicular component, the negative audio embeddings are shown in the right subfigure with distinct distributions across batches. In practice, the negative distributions across batches are more likely to overlap, as shown in the left subfigure.
Global Perpendicular Component: From a global perspective, even if we could access all negative examples in the dataset, the direction of their weighted resultant force would seldom collinear with the direction the pulling force from a specific positive example. This creates systematic, global perpendicular component. Throughout the entire model training process, this global component continuously pushes the optimization path of the text embedding away from the ideal straight-line trajectory, resulting in systematic drift in the entire alignment path.
Local Perpendicular Component: In practice, training performed using a mini-hatch strategy, which means that at each update step, the negative examples seen by the model are only a random subset of the total population of negative examples. The distribution of negative examples varies between batches, causing the direction and magnitude of the per- pendicular component to change drastically with each update. This uncertain and random sideways push is the direct cause of the local, high-frequency oscillations observed in the optimization path.
Acting in concert, these two perpendicular components give rise to the problem of optimization tra- jectory drift in audio-text contrastive learning. This reduces the model's convergence efficiency and
4

---

Published as a conference paper at ICLR 2026
ultimately limits the alignment accuracy of the learned representations if the perpendieular compo- nents are not well controlled.
METHOD
To address the optimization trajectory drift identified in our analysis, we propose the SupCLAP framework built around the Support Vector Regularization (SVR) method. The main idea of SVR is to control the perpendicular component of pushing force by using an auxiliary regularization term to align text support vectors and audio embeddings, thereby guiding the optimization towards more stable and direct trajectory. The construction this support vector is critically dependent on semantic radius R. which dictates the magnitude of the regularization. As the semantic radius R lacks direct supervision, we explore two unsupervised strategies for modeling the semantic radius: a straightforward direct parameterization (StaticSVR) and more sophisticated adaptive radius pre- dictor module (DynamicSVR), which is further enhanced with a constraint term to improve the quality of the predicted semantic radius. We analyze SVR's effect mathematically in Appendix D
3.1 FRAMEWORK OF SUPCLAP
The SupCLAP framework is built upon the standard symmetric contrastive learning objective com- mon in CLAP architectures. We begin by defining this baseline. For a given batch, we have a positive audio-text pair (a ++) and N negative pairs {(a7 t, }} 1. all represented in d-dimensional em- bedding space. The affinity between audio and text embeddings measured by the scaled cosine similarity s(a,t) cos(a, f)/r, where is temperature hyperparameter. The standard training objective Lorig consists of two InfoNCE loss terms: one for text-to-audio alignment loss Lorig and one for audio-to-text alignment Lorig,2t. The total original loss is formulated follows:
Lorig Lorigg + Lorig.
pools(t,att///
- =olog
exp(s(tt,
aaas(tta+)//n aa))/)) ΣSn exp(s(tt,a5)) exp(s(at (///))))))))))))))
(5)
log
exp(s(a* (xp(s(attt+//)) + Σ exp(s(at, xxx(s(at,t7)/f))
To address the trajectory drift problem, we design the Support Vector Regularization (SVR) term, denoted Lovr- The final training objective of SupCLAP weighted sum of the original loss and our SVR regularization term:
L = Loork + aLLur
=Torig a log exp(s(Laup: a a+))
(6)
Σj= exp(s (lapps(supp:1 0,7
The text support vector tsup ++ Rd is constructed by displacing the original text embedding
I+ by the semantic radius R IR along the unit vector â = To+--+T controls the influence of this SVR term in the final loss function. We will detail the unsupervised strategies for modeling the semantic radius R in the following subsection. The intuition behind this design is that by creating auxiliary SVR term, it reshapes the gradient space. It is engineered to selectively control the perpendicular component of the pushing force from negative samples, which the primary source of trajectory drift. A detailed mathematical analysis of SVR's mechanism is provided in Appendix D We conclude that after adding the SVR term, the parallel component of pushing force from the j-th negative audio embedding 0, is
-+ The hyperparameter
PJ
sup.j
Fll.push..
+
while the perpendicular component of the pushing force is
5

---

Published as a conference paper at ICLR 2026
P sups
R
F Fi.phsh
1
1
lat 1+1 ai.s"
(8)
The probability Pj and sup.j can be calculated following equation using text embedding /+ and text support vector Laup separately. a denotes the parallel component of audio embedding oj to the unit vector embedding a to the unit vector The SVR term works by reshaping the total pushing force: while the parallel component is preserved, the perpendicular component is uniquely scaled by the factor (1- I|a TT) This factor allows SVR to selectively suppress the magnitude of the perpendicular component of the pushing force. A larger semantic radius leads to a higher degree of suppression on this component. By exerting this precise control, SVR effectively controls the trajectory drift while harnessing the rich information from negative samples.
at ||a+- while "i.j denotes the perpendicular component of audio
R
Note: Although our illustration and analysis of SVR are focused on the unidirectional (text-to-audee tudio alignment) SVR for clarity, the principle is not exclusive to this direction. As demonstrated by the experimental results in Subsection 4.2 incorporating bidirectional SVR in both text-to-audio and audio-to-text alignments yields superior performance.
Inference pipeline: SupCLAP's inference pipeline is identical to that of standard CLAP methods Elizalde et al. (2023). Retrieval and classification are performed solely by ranking the embedding similarity between audio and text, without the need compute auxiliary support vectors like laup-
3.2 UNSUPERVISED SEMANTIC RADIUS MODELING
The effectiveness of proposed SVR critically depends on the semantic radius R E R. This parameter is central to our method as it directly governs the factor (1 - | - which scales the perpendicular component of the pushing force. Since datasets provide ground-truth values for the semantic radius R, we frame its estimation as an unsupervised modeling problem. To this end, propose and investigate two distinct strategies:
Static modeling (StaticSVR): The Sta a modeling strategy focused on suppress- ing the global perpendicular component. We model the semantic radius R as a single, globally shared, learnable scalar. This scalar jointly optimized with other model parame- ters to minimize the total loss LSaPCLAP- For any text embedding t+, this constant radius is used to construct the support vector tupp = + R. û, where á is the unit vector of the pulling force.
This method's primary advantages are its simplicity and the stability of the global radius R. However, its limitation is the idealized assumption that single, constant radius is optimal for all instances. This static approach lacks the flexibility to handle the varying complexity and alignment difficulty among different audio-text pairs.
Dynamic modeling (DynamicSVR): The DynamicSVR is modeling strategy focused on suppressing the local perpendicular component. For adaptive, instance-specific con- trol, we propose a radius predictor. We hypothesize the optimal semantic radius R is not fixed; it should adapt dynamically to the local embedding geometry, defined by the re- lationship between text embedding &+ and its positive and negative audio counterparts. To learn this relationship, we design a radius predictor-a lightweight Multi-Layer Per- ceptron (MLP) So which learns to output suitable radius in fully unsupervised man- ner. Formally, the predictor fe R + I maps a local similarity vector S to an instance-specific semantic radius R = fo(S). The similarity vector S is composed of s = [s(t+, ; aa)), s(+ ((tt))).(st.,,
The similarity vector informs the radius predictor by capturing local embedding ge- ometry. Each values in S measures the proximity between text embedding + and audio embeddings, collectively signaling potential trajectory drift; for instance, high similarity to negative sample implies higher drift risk. Learning these patterns allows the predictor to estimate a custom semantic radius R for precise control. This adaptive approach's main ad- vantage is its flexibility to adjust the optimization trajectory to each mini-batch 's alignment

---

Published as a conference paper at ICLR 2026
difficulty. However, DynamicSVR's performance is heavily dependent the prediction accuracy of the batch-level radius R. Inaccurate predictions R caused by noisy data, noisy embedding space, weak pretrained model-may may make DynamicSVR perfor- mance inferior to that of the simpler StaticSVR.
We propose constraint term Leons. for the radius predictor module to mitigate the problem of the predicted semantic radius R being unstable inaccurate. Due to space limitations, the introduction of this constraint term is provided in Appendix
EXPERIMENT
In this section, present comprehensive evaluation of our proposed SupCLAP framework. We begin by detailing the datasets, models, evaluation metrics, and experimental setup. We then bench- mark our main results monolingual and multilingual audio-text retrieval, well shot classification tasks. Subsequently, we conduct detailed ablation studies to dissect the contri- bution of each component in our method, the effectiveness of SVR under different batch sizes, and the impact of the SVR term's 's weight Finally, we evaluate the additional time and GPU mem- ory overhead and show that they are negligible. In addition, we experimentally analyze how the modeled semantic radius changes with training epochs. These experiments further demonstrate the theory of trajectory drift and the effectiveness of our proposed approach. Due to space limitations, the evaluation results and corresponding analysis on the multilingual dataset, partial ablation study results, and the analysis of SVR's 's overhead are presented in Appendix
the zero-
4.1 EXPERIMENTAL SETUP
4.1.1 DATASETS AND METRICS
Our experiments are conducted on the AudioCaps Kim (2019) and Clotho Drossos Drossos et al. (0220)
datasets. AudioCaps consists of approximately 49k 10-second audio clips, where each training sample is paired with single caption, and each validation/test sample has five captions. Clotho contains 6974 clips of 15-30 seconds in duration, all of which are annotated with five captions. For our multilingual scenario, we augment these datasets by translating all English captions into seven other languages (fra, deu, spa, nid, cat, jpn, zho). All audio clips are resampled to 16 kHz.
We evaluate retrieval performance in both monolingual and multilingual settings using Recall at rank k (R @ k), which is I if the target is found in the top-k results, and mean Average Precision at 10 (mAP10), which evaluates precision scores across all queries for the top 10 retrieved items.
4.1.2 MODELS AND IMPLEMENTATION DETAILS
Model Architecture: Our framework for audio-text retrieval leverages two powerful pretrained encoders to handle the respective modalities. We adopt the CED-Base model Dinkel al. (2024) as our audio encoder. We employ the multilingual SONAR-TE model Duquenne et al. (2023) as text encoder, which has demonstrated good performance on cross-lingual similarity benchmarks like xsim and xsim++, making it well-suited as the text encoder for this paper. The sentence embeddings are computed by pooling the encoder's token-level hidden states. The semantic radius predictor implemented as an MLP with 3 layers.
To ensure a fair and controlled comparison, all models are initialized with weights from CED-Base and SONAR-TE and trained for 10 epochs on a single NVIDIA H800 GPU. The training employs the Adam optimizer with a learning rate of 10-5 The batch size is set to 24, and temperature of r=0.07 for the contrastive loss. The SVR weight a is set to 1. We investigate two distinct audio-text retrieval scenarios: a monolingual scenario, utilizing only English captions, and multilingual scenario. The model checkpoint yielding the highest recall the test set is selected for final evaluation. We name our SVR methods like "bi-DynamicSVE and "uni- StaticSVR", The prefix "bi" or "uni" indicates bidirectional (both audio-to-text and text-to-audio) or unidirectional (text-to-audio) SVR, respectively. The suffix "StaticSVR" or "DynamicSV VVR" denotes the different ways of modeling the semantic radius R mentioned in 3.2
Implementation Details:
7

---

Published as a conference paper at ICLR 2026
4.2
MAIN RESULTS
4.2.1
AUDIO-TEXT RETRIEVAL
To evaluate the effectiveness of our proposed framework, we compare four methods: (1) the standard InfoNCE loss; (2) the standard SigLIP loss Zhai et al. (2023); (3) our bi-StaticSVR, where the semantic radius R is a single, globally shared, learnable parameter; and (4) our bi-DynamieSVR. which utilizes an adaptive predictor to estimate an instance-specific radius R.
Table Recall and precision results under monolingual AudioCaps and Clotho dataset
AudioCaps
Clotho
Model
T2A
A2T
T2A
A2T
R@D
R@ 10
R@
R@
R@
R@
MMT
36.10
84,50
39,60
86.70
6.70
33.20
7.00
34.60
ML-ACT
33,90
82.60
39.40
83.90
14.40
49.90
16.20
50.20
CLAP
34.60
82.00
41.90
84.60
16.70
54.10
20.00
58.70
CompA-CLAP
36.10
81.60
45.20
86.70
16.80
56.10
19.70
55.60
LAION-CLAP
34.50
80.20
42.50
87.40
15.80
52.90
19.10
54.90
GPA
36.20
82.90
44.20
86.70
15.70
50.90
18.60
55.30
T-CLAP
39.70
86.90
49.80
91.90
17.30
53.60
21.80
57.40
ReCLAP
37.10
85.00
48.00
90.80
18,90
59.00
20.50
58.90
Cacophony
41.00
86.40
55.30
92.40
20,20
58.80
26.50
67.30
SigLIP
36.74
85.71
48.00
88.03
13.58
51.21
17.10
52.56
-bi-StaticSVR
42.54
87.61
55.25
90.55
16.21
53.60
21.26
59.13
43.09
89.26
56.30
92.67
17.51
56.85
22.71
60.87
InfoNCE
41.87
87.69
56.72
92.33
18.67
58.42
22.61
63.09
-bi- bi-SStatieSVR
43.89
88.78
57.77
92.75
19.50
58.86
24.93
63.19
-i-DDynamiSSVR
44.16
89.24
59.66
93.49
19.75
59.13
25.31
63.29
Results from the monolingual audio-text retrieval task, as shown in Table demonstrate that both bi-StaticSVR and bi-DynamieSVR effectively enhance the performance of InfoNCE and SigLIP. This validates our analysis of the perpendicular component in contrastive learning and the effectiveness of our proposed SVR term in resolving the trajectory drift problem. The superior performance of InfoNCE compared to SigLIP stems from its Softmax based compet- itive mechanism, which provides stronger gradient signal for effective discrimination on diverse audio datasets like AudioCaps and Clotho that contain numerous hard negatives.
Monolingual Retrieval:
4.2.2 ZERO-SHOT AUDIO CLASSIFICATION
We also assess zero-shot audio classification on the ESC-50 and US8K benchmarks using our mono- lingual model loaded with pretrained weights provided in ML-CLAP Yan et al. (2024). Text labels are constructed using the template "This is sound of [classlabel]" and we report top-1 accuracy based on the highest cosine similarity between audio and text embeddings.
Table 2: Zero-shot Audio Classification Performance of the CLAP Model
Audio Clas lassification Dataset & Setting
Model
ESC-50
US8K
VGGSound
Wav2CLIP
41.4
40.4
10.0
AudioCLIP
69.4
65.3
CLAP
82.6
73.2
CLAP
89.5
76.3
23.1
Collat
84.0
77.0
InfoNCE
89.6
81.63
24.57
bi Ri-SStaiceSR
90.7
83.63
24.65
Bi-DDynamieSVR
92.1
83.74
25.11
As shown Table the bidirectional dynamic SVR achieves the highest classification accuracy. This further demonstrates the generalization capability of SVR, which can learn more robust and semantically meaningful feature representations by effectively suppressing trajectory drift, thereby enhancing the model's performance in both retrieval and classification tasks.
8

---

Published as a conference paper at ICLR 2026
4.2.3 ABLATION STUDIES
We conducted extensive ablation studies on the AudioCaps dataset. These studies include evaluating the impact of SVR components with different directions and modeling strategies on model perfor- mance, the generalization of SVR under different batch sizes, and the effect of the SVR weight a on model performance. Due to space limitations, the ablation experiment results and analysis for different batch sizes and the SVR weight are presented Appendix [0. The results show that both SV and DynamicSVR can effectively improve the performance of contrastive learning across various batch sizes, and the best results are achieved when the SVR weight a is set to
Table 3: Ablation Study of SVR Variants on Monolingual Text-Audio Retrieval.
T2A
A2T
ID
Model
R@
mAPI0
R@
mAPIO
0
InfoNCE
41.87
56.74 74
56.72
35.36
-bi- Si-DDynamicSRR
44.16
58.79
59.66
36.69
Ibi-DynamiSSRR wo/ constraints
44.01
58.47
59.24
36.64
3
-uii-DynamicSVR
43.63
58.16
58.51
36.00
4
unni-DynmiiSSV wo/ constraints
43.53
58.11
57.67
35.96
5
-bi-StaticSVR
43.89
58.36
57.77
35.72
uni-SSaaicSSR
43.28
57.95
57.56
34.62
Effectiveness SVR Components: The results in sybbeesyyttmmaticaall demonstrate the effective- ness of our proposed components for SVR. The fully-equipped bidirectional DynamicSVR model with constraints (bi-DynamieSVR) achieves the best results. We find that the unidirectional SVR outperforms the baseline model. This performance gain is further amplified when bidirectional SVR simultancously applied to both audio-to-text and text-to-audio directions. Furthermore, the exper- Sarriioniimmrree imppoves the accuracy of DynamicSVF modeling the semantic radius RG further enhancing the effectiveness SVR.
imental results indicate that introducing the constraint term in equation
4.3 SEMANTIC RADIUS ANALYSIS
EVORCCE Mott
Opsch
Figure 3: Results of Semantic Radius Changes
We extract the parameterized radius for bi-StaticSVR at each epoch and compute the average pre- dicted radius for bi-DynamicSVR. As depicted in Figure 3 the semantic radius R decreases as training progresses. Based on our analysis in Appendix E.8 this phenomenon provides evidence that self- supervieed modeling effectively learns the trade-off between suppressing the perpendicular component and preserving information from negative samples. Furthermore, bi-StaticSVR s radius curve is smoother, reflecting its stable, global modeling of the perpendicular component, whereas bi-DynamicSVR's local approach results in greater fluctuations across batches.
5 CONCLUSION
This work addresses optimization trajectory drift in audio-text contrastive learning, an instability caused by the perpendicular component of the pushing force from negative samples. We propose SupCLAP, framework that incorporates Support Vector Regularization (SVR) mitigate this drift. SVR introduces an auxiliary support vector to reshape the gradient space, controlling the perpendic- ular force and stabilizing the optimization path. We explore two unsupervised strategies for its key

---

Published as a conference paper at ICLR 2026
parameter, the semantic radius: StaticSVR and DynamicSVR. Experiments show that SupCLAP significantly outperforms InfoNCE baselines on retrieval and classification tasks with negligible computational overhead, validating approach's 's effectiveness and practical viability.
REFERENCES
Heinrich Dinkel, Yongqing Wang, Zhiyong Yan, Junbo Zhang, and Yujun Wang. Ced: Consistent ensemble distillation for audio tagging. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 291-295. IEEE, 2024.
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 736-740. IEEE, 2020.
Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. Sonar: sentence-level multimodal and language-agnostic representations. arXiv e-prints, pp. arXiv-2308, 2023.
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1-5. IEEE, 2023.
Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Evuru, S Ramaneswaran, Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Compa: Addressing the gap in compositional reasoning lanioala models. arXiv preprint arXiv:2310.08753, 2023.
Sreyan Ghosh, Sonal Kumar, Chandra Kiran Reddy Evuru, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Reclap: Improving zero shot audio classification by describing sounds. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2025.
Andrey Guzhov, Federico Raue, Jöm Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 976-980. IEEE, 2022.
Marti A. Hearst, Susan Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18-25) 1998.
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume (Long and Short Papers), pp. 119-132, 2019.
Panagiotis Koromilas, Giorgos Bouritsas, Theodoros Giannakopoulos, Mihalis Nicolaou, and Yan- nis Panagakis. Bridging mini-batch and asymptotic analysis in contrastive learning: From infonce to basee losses. arXiv preprint arXiv:2405. 18045, 2024.
Max WY Lam, Yijin Xing. Weiya You, Jingcheng Wu, Zongyu Yin, Fuqiang Jiang, Hangyu Liu, Feng Liu, Xingda Li, Wei-Tsung Lu, et al. Analyzable sfain-of-musicall -thouhh prompting for high-fidelity music generation. arXiv preprint arXiv:2503. 1961 2025.
Xin Liang, Yanli Ji, Wei-Shi Zheng, Wangmeng Zuo, and Xiaofeng Zhu. Sv-learner: Support- contrastive learning for robust learning with noisy labels. IEEE Transactions on Knowledge and Data Engineering, 36(10):5409-5422 2024.
Shengqiang Liu, Da Liu, Anna Wang, Zhiyu Zhang, Jie Gao, and Yali Li. Dsclap: Domain-specific contrastive language-aude audio pre-training. arXiv preprint arXiv:2409.09238 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 1748-8763 PmLR, 2021.
10

---

Published as a conference paper at ICLR 2026
Anshul Shah, Suvrit Sra, Rama Chellappa. and Anoop Cherian. Max- margin contrastive learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2220-8230) 2022.
Dadallage AR Silva, Spencer Whitehead, Christopher Lengerich, and Hugh Leather. Collat: on adding fine-grained audio understanding to language models using token-level locked-language tuning. Advances in Neural Information Processing Systems, 36:63197-63227 63209,,,, 2023.
Jiamian Wang, Guohao Sun, Pichao Wang, Dongfang Liu, Sohail Dianat, Majid Rabbani, Raghuveer Rao, and Zhiqiang Tao. Text is mass: Modeling stochastic embedding for text-video retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 16551-16560, 2024.
Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. Rethinking infonce: How many negative samples do you need? arXiv preprint arXiv:2105. 130033 2021.
Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4563-4567. IEEE, 2022.
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keypord-to-captio augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Yuxin Xie, Zhihong Zhu, Xianwei Zhuang, Liming Liang, Zhichang Wang, and Yuexian Zou. Gpa: Global and prototype alignment for audio-text retrieval. In Proc. Interspeech, volume 2024, pp- 5078-5082, 2024.
Jinlong Xue, Yayue Deng, Yingming Gao, and Ya Li. Retrieval augmented generation in prompt- based text-to-speeed synthesis with context-aware contrastive sunuuage-audin pretraining. arXiv preprint arXiv:2406.037/1 2024.
Zhiyong Yan, Heinrich Dinkel, Yongqing Wang, Jizhong Liu, Junbo Zhang, Yujun Wang. and Bin Wang. Bridging language gaps in audio-text retrieval. arXiv preprint arXiv:2406.070/2 2024.
Yuguo Yin, Yuxin Xie, Wenyuan Yang. Dongchao Yang, Jinghan Ru, Xianwei Zhuang, Liming Liang, and Yuexian Zou. Atri: Mitigating multilingual audio text retrieval inconsistencies by reducing data distribution errors. arXiv preprint arXiv:2502.14621 2025.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners image-text foundation models. arXiv preprint 2rX:v:2005.01917. 2022.
Ruibin Yuan, Hanfeng Lin, Shuyue Guo, Ge Zhang, Jiahao Pan, Yongyi Zang, Haohe Liu, Yiming Liang, Wenye Ma, Xingjian Du, et al. Yue: Scaling open foundation models for long-form music generation. arXiv preprint 2xXiv:2503.0863: 2025.
Yi Yuan, Zhuo Chen, Xubo Liu, Haohe Liu, Xuenan Xu, Dongya Jia, Yuanzhe Chen, Mark D Plumbley, and Wenwu Wang. T-clap: Cemporal-enhancee contrastive language-audio pretraining. In 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1-6. IEEE, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. Proceedings of the IEEF/CVF international conference on computer vision, pp. 11975-11986, 2023.
Ge Zhu, Jordan Darefsky, and Zhiyao Duan. Cacophony: An improved contrastive audio-text model. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
Xianwei Zhuang, Yuxin Xie, Yufan Deng, Liming Liang, Jinghan Ru, Yuguo Yin, and Yuexian Zou. Vargpt: Unified understanding and generation in visual autoregressive multimodal large language model. arXiv preprint arXiv:2501. 12327, 2025a.
Xianwei Zhuang, Yuxin Xie, Yufan Deng, Dongchao Yang. Liming Liang, Jinghan Ru, Yuguo Yin, and Yuexian Zou. Vargpt-v1. 1: Improve visual autoregressive large unified model via iterative instruction tuning and reinforcement learning. arXiv preprint arXiv:2504, 02999, 2025b.
11

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Learning joint representations for audio and text has become a pivotal area of research, enabling applications from cross cross-modal retzaldalElizzlde Elizalde et al. (2023) to text-to-audio generation Yuan et al. (2025). A dominant paradigm this domain contrastive learning, which aims to create a unified embedding space where audio and its corresponding text description are brought closer together, while dissimilar pairs are pushed apart. Drawing inspiration from the success of CL.IP Radford et al. (2021); Zhuang et al. (2025b) in the vision-language domain Yu et al. (2022); Zhuang ct al. (2025a), the Contrastive Language- Audio Pretraining (CLAP) framework based on InfoNCE loss has emerged as a foundational approach Wu et al. (2022); Guzhov et al. (2022). Researchers have since focused on refining and extending the capabilities of CLAP. For instance, to better manage au- dio inputs of varying durations and boost overall performance, /Wu et al. (2023) integrated a feature fusion mechanism with a keyword-based description enhancement strategy. Others have focused on improving the model's grained understanding. Siiva et al. (2023) introduced COLLAT, frame- work that achieves nuanced audio comprehension by freezing the language model's parameters and training the audio encoder with a specialized audio-text alignment objective. Similarly, address the challenge of distinguishing between closely related but distinct sounds, such hard negative samples. Ghosh et al. (2023) developed a modular contrastive loss designed to improve the model's discriminative capabilities. In subsequent work, Ghosh et al. (2025) further explored the impact of textual data quality, demonstrating that rewriting audio captions to be more descriptive significantly enhances the model's 's comprehension of real-war world acoustic scenes. Beyond these, domain special- ization has gained attention. [Liu et al.] (2024) proposed DSCLAP for domain -specifi audio-text pre-training, emphasizing tailored representations for specialized contexts. Meanwhile, /huu et al. 2024) introduced Cacophony, which strengthens retrieval with auxiliary objectives.
Existing works have explored integrating principles from Support Vector Machine (SVM) Hearst et (1998) into contrastive learning from different perspectives. Existing approaches include Max- Margin Contrastive Learning (Shah et al. (2022), which adapts the SVM max-margin principle to identify and push away hard negative samples treated as support vectors. SV-Learner Liang et al.) (2024) is another approach that employs SVM to select reliable data pairs for contrastive learn- ing, thereby enhancing robustness in noisy-label scenarios. Concurrently, the T-MASS Wang et al. (2024) approach also utilizes a support vector to model text as a stochastic mass, aiming to en- rich text's semantic representation for text-video retrieval. Compared to the above approaches, our work SupCLAP introduces Support Vector Regularization (SVR) to address a different core prob- lem: controlling the optimization trajectory drift caused by the perpendicular pushing force from negative samples in contrastive learning.
B CONSTRAINT TERM IN DYNAMICSVR
To improve the stability and precision of the predicted semantic radius R. propose a constraint term Leoms on the radius predictor module. While the adaptive predictor offers flexibility, an uncon- strained radius vector R could lead two potential failure modes:
Excessive Magnitude: A semantic radius R with an excessively large magnitude (i.c., R. >>>>> ||at ID) causes the factor (1 -o+ ) to become negative. This may lead a directional inversion of the perpendicular component representing the additional infor- mation of the negative samples. As a result, the effective utilization of this information is impeded, which turn leads to optimization instability.
Counterproductive Direction: Without constraints the semantic radius R during un- supervised modeling. the radius predictor may output R<0, which results in the factor (1- - >>> This adversely magnifies the perpendicular component, thereby aggra- vating the problem of optimization trajectory drift.
To simultaneously address both issues, we formulate the constraint term Loons = Relu R ||at [+) + Relu(-R) This term is composed of two components, each targeting one of the aforemen- tioned failure modes. The first Relu( R lIaaa -I1 directly penalizes large magnitudes to prevent overshooting. The second, Relu(- R), explicitly encourages directional alignment with the pulling
12

---

Published as a conference paper at ICLR 2026
force. By incorporating this constraint, the total loss function becomes:
LSupCLAP-Cond = Lorig + aLLevr + Lecons
These constraints ensure that the support vector provides a stable and meaningful regularization signal, transforming SVR into more robust and effective fine-tuning mechanism. We use a default constraint weight of B = 0.01, which is intended to impose slight penalty, ensuring the semantic radius remains within a reasonable range without dominating the loss function.
C DETAILED THEORETICAL AND EMPIRICAL ANALYSIS OF OPTIMIZATION TRAJECTORY DRIFT
In this section, elucidate the causal relationship between controlling the perpendicular compo- nent and performance improvement by examining the two fundamental optimization objectives of contrastive learning: pulling positive pairs closer and pushing negative pairs apart.
C.I THBORETICAL PROOF CONVERGENCE SPEED OF POSITIVE PAIRS
To demonstrate that the perpendicular component (F) reduces the optimization speed of represen- tation alignment, we analyze the change the squared Euclidean distance between the anchor text embedding t and the positive audio embedding at .et (zezt denote the text embedding in the next update step. The update rule is given by + next = 1+ n,F, where is the learning rate, and the total gradient force decomposed into parallel and perpendicular components relative to the pulling di- rection: F = Fii + F1. We quantify the convergence speed by the reduction in the squared distance the target at_ A smaller value of the difference ||'text a=1 || a+ 112 implies larger reduction in distance, and thus, a faster convergence speed. The derivation is as follows:
|| a a+112= ||(t= a+* -at)-
( + FI)112
(10)
= ||t - --112-- 2n((t - a (F) +1 + FL11..
Since F1 is orthogonal to both the pulling direction (t+ a+) and the parallel component FJJ,
have F and || + F1 = ||21 +| Substituting these into the equation:
|| Ieeatt
a+113 ||t a 1|2 2n(t a (11 +|
= ||t - a 1|2 - 2n(t - at - + 1/
(11)
||t a* 112 - -)((1 2a - n111)" > F1i + 1
The change in distance can thus be decomposed into two distinct terms:
|
next a 1| || a at|l1 112=
(2t 2a - 11F F
F1
(12)
Optimization dynamics (convergence speed of positive pairs)
Optimization dynamics along the idcal direction
penalty
Optimization along the ideal direction: In the first term, because the learning rate r) typi- cally small, the hindrance caused by the second-order parallel component (r is far smaller than the promotion of convergence speed provided by the first-order parallel com- ponent (2ny(t* a a+ Fi). This term represents the effective progress along the geodesic path.
Noise penalty: The second term 1f1 I| depends solely on the perpendicular component. Since this term is strictly greater than 0 if the perpendicular component is not zero, it acts as a penalty that increases the final distance. Therefore, a larger perpendicular component imposes greater hindrance to the convergence speed of positive pairs. This mathematical conclusion validates our hypothesis that suppressing the perpendicular component acceler- ates alignment.
13

---

Published as a conference paper at ICLR 2026
C.2 ANALYSIS THE PUSHING OBJECTIVE AND STOCHASTIC DEVIATION
Regarding the objective of pushing negative pairs apart, we analyze the impact of F1 from an empirical and global perspective. While InfoNCE aims to push text embeddings away from negative samples, two key factors suggest that the perpendicular component suboptimal for this goal:
Mini-Batch Stochasticity: Current InfoNCE implementations rely on Mini-Batch training. Consequently, the direction of the pushing force within specific mini-batch inevitably deviates from the true gradient direction of the entire dataset. This stochastic deviation acts noise in cross-modal representation alignment. This is supported by the consen- sus contrastive learning research that larger batch sizes-which reduce gradient vari- ance consistently yield better alignment performance.
Global Trajectory Drift: Even from global perspective, the aggregate pushing force gen- erated by the distribution of all negative samples seldom perfectly collinear with the pulling force from the positive sample. This generates a Global Perpendicular Compo- nent 1. Although this component geometrically pushes the embedding away from negative samples, it simultaneously pushes the optimization path sideways, deviating from the ideal straight-line trajectory required to align with the positive sample.
D ANALYSIS OF SVR's MECHANISM
In this section, we analyze the gradient space of the SVR to show that SVR effectively suppress the perpendicular component and improve the accuracy of the learned representations in theory.
The support vector is located in the direction pointing from text embedding t to positive audio embedding at and lies on the surface of the text embedding distribution centered text embedding + Its mathematical expression tsup = ++ + R --- - ++ + Rú, where = a -|-1-1111 the unit vector in the pulling direction and R is an adaptive and learnable radius.
The gradient of SVR term in equation 6 computed by the chain rule:
@8tupp Öt
T V
V
taup Love-
(13)
The Jacobian matrix dr*
Staup is computed
Ötsup
1t + R at -t
0tt
Ot
laat - -+1l
(14)
at ++
+ R Ot+
|aat - 1+
After derivation, get 06 that projects any vector onto the hyperplane orthogonal û. We denote P1 I ûû. Therefore, the Jacobian matrix is denoted as
| (bù- 1). The matrix (ūù I) is projection operation
Ht Öteu dt+
R
=1
P1.
(15)
|aat
++
We further analyze the pushing force component from a single negative example aj denoted as jusp.push... within the gradient generated by Tagar term:
14

---

Published as a conference paper at ICLR 2026
xup Ut+
T
Psupl
fsup.push.j = 0:
T
P
sup.j
lat - |+| P_aj)
R
a T
aj
(16)
P Poup.
R
( +
1aat ++
7
P
sap.
R
a
"Ii. + (1
1aat - 1 (aLi))
where denotes the weight of the SVR term in the total loss, probability Pupp. can be calculated in equation 3 by replacing text embedding t to text support vector (sup- We decompose the negative audio embedding aj into parallel component iii and perpendicular component a aI. the unit vector ú, denoted as aj Ii.5 0 "i. It should be noted that Piaj (1 uujaaj a..""
This result is the core of the proof. It shows that for the pushing force generated by the support vector loss, its parallel component || remains unchanged, while its perpendicular component eI. scaled by a factor of |---
We examine the parallel component of the total pushing force from negative example j. denoted as F f.
Fil.push.. S1i.s + Seaupllis
Pj
supis
"1.
11.5
(17)
P
P Paupus
+
And the perpendicular component F, push, of the total pushing force from negative example j can be computed
Fu.phshh.j S1.j Ssup.
P.
Poups
R
aig +
(1
1
|aa - tt PI.j
(18)
P7
+
p Props
1-
||at - tt|l (
R
aI.
The factor (1 (1- [1-1a=-(+i] plays a key modulating role to control the optimization trajectory drift. For most of the training process, the text embedding /+ is at certain distance from its positive example at, such that ||at II... In this scenario, the factor is positive number less than 1, which means the support vector regularization term actively weakens this drift caused by the perpendicular component. Consequently, the overall gradient direction points more closely toward the pulling force while adaptively maintaining enough additional information in negative samples, making the optimization path from text embedding t" positive audio embedding a more direct and stable. This reduces unnecessary drift, thereby accelerating the convergence process and potentially leading to better alignment.
Rt
15

---

Published as a conference paper at ICLR 2026
E ADDITIONAL EXPERIMENT
E.1 STATISTICAL SIGNIFICANCE EVALUATION
To ensure the robustness of our results and verify that the performance gains are statistically signif- icant rather than artifacts of random initialization, we performed 5 independent training runs using different random seeds for the InfoNCE, SigLIP, and our proposed variants. We report the mean and standard deviation of the retrieval metrics on the AudioCaps dataset.
As presented in Table 4 our SVR-based methods consistently outperform the baselines with low variance. For instance, compared to the strong InfoNCE baseline (41.79 + 0.14), our bi- DynamicSVR achieves a T2A R@I I of 44.29 = 0.61. Notably, the lower bound of our method's confidence interval is strictly higher than the upper bound of the baseline, indicating that the im- provement statistically significant. A similar trend observed when comparing our method against SigLIP, where bi-DynamicSVR yields substantial improvement of over 6% in T2A R@) 1. These results confirm that controlling the perpendicular component of the pushing force leads to stable and reproducible performance gains.
Table 4: Performance comparison AudioCaps with mean and std.
Text-to-Audio
Audio-to- Text
Method
R@l
RØ10 10
R@i
R@ 10
SigLIP
16.18
85.83
47.92
88. 14
bi-StaticSVR
42.67
87.55
55.41
90.42
Db-DDynamicSVR
43.21
89.1540.58
56. 56.4420.72
40.61
InfoNCE
41.7940.14
87.75
56.61
92.41 40.18
bi StaticSVR
43.7640.35
88.92 +0.29
57. 57.64
92.88+0.24
bi- Hii-DynamicSSS VR
44.29
89.0810.53
59.83 19.8320.77 2..79
93.3540.66
E.2 EXPERIMENT RESULT UNDER LARGE- SCENARIO AND DISTRIBUTION SHIFTS SCENARIO
To further validate the efficacy of SVR in real-world scenarios, we evaluated it under two settings: data scaling and distribution shift. For the data scaling setting, we utilized the WavCaps dataset, which encompasses a broader range of audio events and comprehensively reflects complex real- world data distributions. Specifically, we held out 5,000 samples as the test set and used the remain- der for training. regarding the distribution shift scenario, we evaluated the performance of models trained on the Clotho dataset using the AudioCaps dataset, and vice versa.
Table 5: Performance of InfoNCE and InfoNCE+SVR under WavCaps dataset
A2T
T2A
Model
R@1
R@1
InfoNCE
20.60
7.76
-bi- StaticSVR
21.00
8.26
-bi- DynamicSVR
21.30
8,30
As shown in 5 both StaticSVR and DynamicSVR consistently outperform the standard In- foNCE baseline. specifically, DynamicSVP VR achieves the highest accuracy with 21.30% (A2T) and 8.30% (T2A) in R@1, This confirms that our SVR method remains effective and beneficial when applied to large-scale, diverse real-world data.
Table 6: Performance of InfoNCE and InfoNCE+SVR under distribution shifts scenario
AudioCaps
Clotho
Clotho
AudioCaps T2A
Model
A2T
T2A
A2T
R@
RØI
R@
R@
InfoNCE
18.1643
14.4541
26.6807
20.6513
-bi-StaticSVR
19,4203
14,8213
28.3613
21.0924
-bi- SVR
20.1932
14.9279
29,6218
21.3655
16

---

Published as a conference paper at ICLR 2026
Furthermore, we assessed the model's robustness to domain differences through cross cross-dataset eval- uation, as shown in Table In this zero-shot and vice versa-oue SVR-based methods demonstrate significant performance gains over the base- line. Notably, DynamicS VR improves A2T @@1 by approximately 2.0% and 2.9% in the two transfer scenarios, respectively. These results suggest that by controlling optimization trajectory drift, our SVR methods learn more universal representations that generalize well to unseen domains with distinct data characteristics.
on AudioCaps and testing on Clotho,
E.3 DISTRIBUTION OF POSITIVE PAIR SIMILARITY
Stankarity
Figure 4: Distribution of Positive Pair Similarity in AudioCaps
To further demonstrate the robustness of our approach, we analyzed the similarity score distribution of positive pairs on the AudioCaps test set, as shown in Figure The blue curve represents the base- line InfoNCE model, while the green and orange curves correspond to our InfoNCE-bi-Station St--StatiSV. and InfoNCE-bi-DynnmmisSW tion peaks earlier (around 0.65-0.70) and decays rapidly in the high-similarity region (> 0.75). In contrast, both SVR-based methods exhibit noticeable rightward shift in the distribution. Specif- ically, our methods maintain a significantly higher density of positive pairs in the high-confidence interval (similarity scores between 0.7 and 0.9) compared to the baseline. This distributional shift indicates that the proposed SVR mechanisms effectively mitigate the semi-hard negative problem by pulling positive pairs closer in the shared embedding space. The increased number of high-similarity pairs provides empirical evidence that the performance improvements are driven by superior and more tightly clustered modal alignment, rather than random variance.
methods, respectively.Ar illustrated, the baseline InfoNCE distribu-
E.4 COMPARISON SVR WITH SIMPLE GRADIENT REWEIGHTING
To address the question of whether the proposed Support- Vector Regularization (SVR) functions merely form of gradient re-scaling, we implemented a baseline method named Simple Reweight under the same setting as SVR for comparison. This analysis serves to demonstrate that the improve- ments yielded by SVR are not side effects of simple magnitude control, but rather the result of its unique geometric properties.
The Simple Reweight baseline attempts to achieve directional correction by amplifying the weight of the positive sample gradient. We introduced a learnable scalar parameter to scale the probability pt computed in equation 2 This method emphasizes the parallel component of the gradient relative to the negative samples, theoretically mimicking a basic form of directional adjustment.
We evaluated this baseline the AudioCaps dataset. As shown in Table the Simple Reweight strategy resulted in performance degradation across both Audio-to-Text (A2T) and Text-to-Audio (T2A) retrieval tasks compared the standard InfoNCE loss.
The performance degradation in the Simple Reweight baseline highlights a critical distinction in optimization dynamics: the coupling between gradient direction and magnitude.
Instability of Simple Reweighting: Simply increasing the weight of the positive gradient causes the learnable scalar to simultaneously alter the gradient direction and significantly
17

---

Published as a conference paper at ICLR 2026
Table 7: Performance Comparsion between InfoNCE and Simple Reweight Baseline.
A2T
T2A
Model
R@1
R6
InfoNCE
56.72
41.87
-Simple Reweight
51.26
39.15
inflate the magnitude of the parallel component. This coupling destabilizes the training process, the scalar itself becomes difficult to model in an unsupervised manner when the gradient magnitudes fluctuate drastically.
Stability of SVR: In contrast, SVR employs the scaling factor (1 R selectively adjust the perpendicular component. Crucially, the unsupervised modeled radius It does not directly inflate the magnitude of the parallel component. This decoupling ensures that the gradients remain stable after regularization, avoiding the magnitude explosion observed in simple re-weighting.
Therefore, while SVR be theoretically viewed sophisticated form of re- weighting, its ability to control trajectory drift without destabilizing optimization is unique and cannot be replicated by simple scalar multiplication.
E.5
EXPERIMENT RESULT UNDER MULTILINGUAL SCENARIO
Table 8: Recall and precision results under multilingual AudioCaps and Clotho dataset.
AudioCaps
Clotho
Model
T2A
A2T
T2A
A2T
R@
R@ 10
R@D
R@10
R@
R@ 10
R@1 1
R@ 10
SigLIP
34.40
82.34
44.26
86.24
13.62
50.53
17.50
53.36
-hi-StaticSVR
36.87
84.23
49.54
89.08
15.17
52.32
18.70
57.00
-bi- bb-DyynamieSVR
38.56
85.20
51.46
90.68
15.82
54.36
21.22
58.20
InfoNCH (MI-CIAP)
37.20
84.79
50.20
90.68
17.10
55.79
21.50
58.80
-hiSStaticSVR
39.60
84.98
52,36
91.05
17.28
56.06
22.37
59.42
bi-DyynmieSSVR
39.75
85.52
53,99
91.22
17.49
56.35
22.43
60.04
ATRI-CACI
39.44
85.36
53.42
91.05
-bi bi-StaaicSVR
40.01
84.98
54.28
92.20
-i-DDynamiSSVR
43.61
89.13
60.08
94.22
Multilingual Retrieval: Multilingual Retrieval: To further validate the effectiveness of our pro- posed framework, we conducted evaluations in a multilingual scenario. Following the scheme of ML-CLAP Yan et al. (2024), we dynamically pair each audio sample with random text caption from one of eight languages during training. The reported metrics are averaged across lan- guages. We compared our approach against standard losses like SigLIP, InfoNCE, and the recent strong baseline ATRI Yin et al. (2025). For ATRI, we reproduced its Audio English Co-Anchor Contrastive Learning (CACL) objective.
As shown in Table g both bi-StaticSVR and bi-DynamicSVR consistently improve performance across all baselines. Notably, incorporating SVR into ATRI yields significant gains, with bi- DynamicSVR achieving the best overall performance. These results confirm that our SVR mecha- nism orthogonal existing advanced schemes and can be effectively integrated to enhance state- of-the-art models, validating its generalizability even in complex multilingual scenarios. We also evaluate the quality of the translated multilingual test in Appendix E.9
E.6 EXPERIMENT RESULT FOR CONVERGENCE SPEED
To empirically validate that SupCLAP establishes a more effective optimization trajectory, we an- alyzed the model's convergence speed. Figure 5 illlustrtess the evolution of the R@ metric across training epochs for both Audio-to-Tex, (A2T) and Text-to-Audio (T2A) retrieval tasks on the Au- dioCaps dataset.
18

---

Published as a conference paper at ICLR 2026
0.4
HAAT
PAACE www.yaaa....om COUR
Spart
Tooch
(a) A2T Convergence Speed under SigLIP.
(b) A2T Convergence Speed under InfoNCE.
081
181
Sg.ll Sy.7 Drure:5
HNNCC SupraN
250
(c) T2A Convergence Speed under SigLIP.
(d) T2A Convergence Speed under InfoNCE.
Figure 5: Comparison of Convergence Speed between Baseline loss and SVR.
As observed, both bi-StaticSVR and bi-DynamieSVR exhibit significantly faster convergence com- pared to the standard InfoNCE and SigLIP baselines. The SVR-enhanced models consistently achieve higher retrieval performance earlier epochs and maintain this advantage throughout the training process. This empirical evidence corroborates our theoretical analysis: by suppressing the perpendicular component of the pushing force, SVR effectively mitigates optimization trajectory drift, resulting in more direct and efficient path toward optimal alignment.
E.7 ROBUSTNESS ANALYSIS OF DYNAMICSVR UNDER NOISY SETTINGS
To further explore the stability of the unsupervised dynamic semantic radius predictor under condi- tions with low-quality or noisy embeddings, we conducted a sensitivity analysis in specific Noisy Setting.
In this setting, both the audio and text encoders were initialized with random weights instead of pre- trained checkpoints. Furthermore, to limit the modeling capacity, we reduced the depth of the text encoder (SONAR-TE) from 24 layers to 12 layers. We compared the performance of the standard method on the AudioCaps dataset.
InfoNCE baseline against our
The retrieval performance under this noisy setting is reported in Table 9 Despite the lack of initial semantic alignment, the CROoNCE-bi-DDnamicc VR method demonstrates significant improvements over the standalone InfoNCE baseline across all metrics.
Table Performance InfoNCE and InfoNCE-bi-DynamiicS VR under noisy setting
R@T
mAP@ 10
Model
A2T
T2A
A2T
T2A
InfoNCE
8.1933
7.4160
5.2340
15.0193
-bi- bbi-DynamieSVR
10.7143
9.4328
7.0139
17.9908
19

---

Published as a conference paper at ICLR 2026
The results indicate that DynamicSVR is robust noisy imput embeddings and does not lead to training collapse or instability. To understand the underlying mechanism, we analyzed the evolution of the predicted semantic radius R during training in the noisy setting. According the result in Figure 6. we observed that the average radius R predicted by the module in the noisy setting is notably larger than that observed when using pre-trained weights.
22
InfoNCE bi StanieSVR
28
InfoNCEE- -i B--DymmmicSV nole setting
InfoNCE DynamicmVii
Somantc
Epoch
Figure 6: Variation of Semantic Radius under Noisy Settings and Pre-trained Weights.
This phenomenon aligns with our theoretical framework regarding optimization trajectory drift. In the noisy setting, there are higher cosine similarities between negative pairs compared to a well- clustered pre-trained space. According to our force decomposition analysis, higher negative similar- ities result in a significantly larger perpendicular component of the pushing force. The DynamicS VR module correctly identifies this instability and adaptively predicts larger radius R. This increased radius exerts a stronger suppression effect on the perpendicular component (via the scaling factor 1 R/ l/ -+ thereby preventing excessive optimization trajectory drift even when the encoder signals are weak. This confirms that the unsupervised predictor functions as an effective adaptive regularizer, scaling its intervention based on the quality of the embedding space.
E.8 IMPACT OF SEMANTIC DIFFICULTY ON SELF-SUPERVISEE SEMANTIC RADIUS MODELING
1.05
1.30
0.20
Sombatic
Epoch
Figure 7: Variation of Semantic Radius under Different Semantic Difficulties.
To elucidate the mechanism of the unsupervised dynamic semantic radius predictor and demonstrate its robustness, we conducted experiment analyzing the variation of the predicted semantic radius R in the presence hard negative samples. Specifically, we randomly sampled 720 negative samples and identified the Top-K (where K E {6, 12, 18)) most similar ones "hard negatives." " These hard negatives were then mixed with general negatives and positive pairs within mini-batch to monitor the changes in the semantic radius R predicted by the DynamieSVR module across different epochs.
As illustrated in Figure 7, we observed a clear positive correlation: the magnitude of the predicted radius R consistently increases the density of hard negative samples (K) rises. This empirical finding provides a precise mechanistic explanation for the predictor's 's behavior, countering concerns regarding its fragility:
20

---

Published as a conference paper at ICLR 2026
Response Semantic Difficulty: The high similarity characteristic of hard negative sam- ples generates a significantly stronger perpendicular component in the gradient space com- pared easy negatives. According to our theoretical analysis, this increases the risk of drift in the optimization trajectory. The radius predictor robustly detects this condition and adaptively increases R. A larger R strengthens the regularization effect (via the scaling factor 1 1 Rt/laat/ttt t (]) to impose tighter control, thereby mitigating the risk of excessive optimization path deviation and ensuring stability.
Temporal Evolution: As training progresses and the model's discriminative capability im- proves, the cosine similarity between negative pairs naturally decreases. This attenuates the pushing force and the associated perpendicular component derived from negative samples. Consequently, the predictor adaptively decreases to avoid over-suppressing component that is already minimal.
These results confirm that the unsupervised radius predictor successfully learns a dynamic trade-off: it prioritizes stability by assigning larger R when the risk of drift is high (e.g., early training stages or hard negative presence) and prioritizes information retention by reducing when the optimization path becomes clearer. This validates that our module is not fragile under noise but rather acts as an adaptive regulator for the perpendicular component.
E.9 QUALITY ASSESSMENT OF MULTILINGUAL TEST SET
Table 10: Semantic Similarity between Re-translated Text and Original English Text on Test Set
Language
AudioCaps
Clotho
French
94.5
93.5
Dutch
95.0
94.2
Spanish
94.8
94.0
German
95.6
94.2
Catalan
91.1
92.2
Japanese
91.7
90.9
Chinese
90.7
90.3
To verify the semantic fidelity of our LLM-translated multilingual Audiocap and Clotho test set and thus substantiate experimental results, we performed back-translation analysis. We first translated the test captions back to English using Deepseek V3. We then computed the average cosine similarity between the embeddings of the back-translated and original English texts via the Roberta-large model. The results, detailed in Table [0] reveal an average similarity exceeding 90% for all languages. This high fidelity demonstrates strong semantic preservation and suggests that translation errors in our test set are sufficiently small.
E.10
ADDITIONAL ABLATION STUDY
Table 11: Ablation Study of SVR Variants on Multilingual Text-Audio Retricval.
T2A
A2T
ID
Model
R@
mAPI0
R@
mAPI0
0
InfoNCE
37.20
52.19
50.20
28.28
-bi-DynamicSVR
39.75
54.22
53.99
32,82
-bi-DynamicSVR wo/ constraints
39.59
54.16
53.39
32.42
unn--ynamicSVR
39.52
54.08
53.26
32.26
Muni-DynamiSSVR wo/ constraints
39.37
53.81
53.07
31.78
5
-bi-StaticSVR
39.60
53.99
52.36
31.59
-nii-StaticSVR
39.51
53.77
52.06
30.70
Effectiveness of SVR Components: The multilingual ablation study results in Table II are consis-
tent with the monolingual scenario in Table 3, in both cases, the bidirectional DynamieSVF VR model with constraints Obi-DynamieSVE achieves the best results. The performance ranking from highest to lowest is bidirectional SVR, unidirectional SVR, and InfoNCE.
21

---

Published as a conference paper at ICLR 2026
Table 12: Ablation study on different batch sizes for T2A and A2T tasks
T2A
A2T
Model
Batch Size
R@1
mAPIO
R@
mAPTO
24
37.20
52.19
50.20
28.28
InfoNCE
48
39.32
53.95
53.28
31.99
72
40.22
54.85
54.34
32.93
24
39.60
53.99
52.36
31.59
-hi-StaticSVR
48
40.54
55.01
54.21
32,92
72
40.89
55.40
54.87
33.47
24
39.75
S4.22
53.99
32.82
bi-DynamicSVR
48
40.83
55.20
54.31
33.25
72
40.92
55.57
55.00
33.82
Impact of Batch Size: Table 12 shows that the performances of both InfoNCE, bi-StaticSVR, and bi-DynamicSVR improve as the batch size increases (24, 48, 72). This is consistent with the principle that larger batches provide more diverse negative samples for contrastive learning. which in turn leads to better performance. Both bi-StaticS and bi-DynamicSVR consistently maintain a stable performance advantage over InfoNCE, indicating that our methods can further enhance the performance of InfoNCE with larger batch sizes.
Table 13: Ablation of SVR weight @
T2A
A2T
Q
R@1
mAPIO
R@1
mAPIO
0.1
38.69
53.38
52.05
30.48
0.5
39.42
53.90
52.28
31.43
1.0
39.75
54.22
53.99
32.82
Impact of SVR Weight a: The results in Table 13 indicate that model performance consistently improves as a increases from 0.1 1.0. This suggests that greater weight for the SVR module significantly enhances the model'ss feature representation matching capabilities. Therefore, the experiment identifies a 1.0 as the optimal hyperparameter setting for the current configuration.
E.11 OVERHEAD ANALYSIS
Table 14: Evaluation results in GPU memory overheads and time overheads
Audio aps GMO(MB)
Clotho
Scheme
TO(s)
GMO(MB)
TO(s)
InfoNCE
22724
2479
30318
1102
bi- StaSitaie
22732
2438
30368
1127
-bi- Di-DDynaieSSVR
22834
2519
30398
1116
In Table 14 we evaluate the time overhead (TO) and GPU memory overhead (GMO) of SupCLAP compared InfoNCE. TO denotes the average training time over 10 epochs, and GMO denotes the peak GPU memory usage during the training process. The results show that the two SupCLAP variants achieve performance gains with almost no additional computational overhead, possessing both high efficiency and practical viability.
F STATEMENT ON THE USE OF LARGE LANGUAGE MODELS
We utilized a large language model (LLM) as a general-puree purpose writing assistant to enhance the clarity and logical structure of the manuscript. This included polishing sentences, improving transi- tions, and refining the overall flow. All LLM-assisted edits in this paper were meticulously checked and validated by us.
Furthermore, we employed an LLM to construct a multilingual AudioCaps and Clotho dataset. To validate the quality of the translations, evaluated the semantic similarity between the original En-
22

---

Published as a conference paper at ICLR 2026
glish texts and their translated counterparts in Appendix E.9 This accomplished by calculating the cosine similarity of their embeddings, which were generated using a RoBERTa-large model.
23


Reviews:
[{"review_id": "7l3lpX6Hlr", "reviewer": "Reviewer_xE2L", "summary": "The paper addresses the issue of optimization trajectory drift during contrastive learning due to the perpendicular component of pushing force from negative samples. It highlights how this aspect can lead to instability in training and proposes a novel solution called Support Vector Regularization (SVR).\n\nSVR introduces an auxiliary support vector that allows for better control of negative sample influence, leveraging their informative content while mitigating trajectory drift. The effectiveness of SVR is critically tied to the selection of the semantic radius, and the paper explores two unsupervised modeling strategies: static direct parameterization and an adaptive radius predictor module with constraints.\n\nExtensive experimental results demonstrate that SupCLAP significantly outperforms various baseline methods across tasks such as classification and both monolingual and multilingual retrieval, validating its potential in stabilizing optimization paths and improving model performance.", "questions": "Given that the SVR method introduces additional parameters that require careful tuning, how does this impact the overall efficiency of the method during training and implementation? Specifically, what are the trade-offs between the improved performance and the potential increases in optimization time?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Addresses Optimization Trajectory Drift: The proposed Support Vector Regularization (SVR) method effectively controls optimization trajectory drift caused by the influence of negative samples, enhancing training stability.\n\nPerformance Improvement: SupCLAP significantly outperforms various existing baselines across tasks such as classification and monolingual/multilingual retrieval, demonstrating its superior performance in multimodal learning.\n\nExploration of Unsupervised Modeling Strategies: The study explores both static parameterization and adaptive radius prediction as unsupervised modeling strategies, providing new methodological insights for future research.", "weaknesses": "Limited Generalizability Beyond Benchmarks: While the performance in the tested tasks is impressive, the study may have limitations in generalizability. The results may not translate well to all real-world applications, particularly those outside the tested domains or with different data characteristics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E88kTX51MR", "reviewer": "Reviewer_nh4f", "summary": "This paper presents SupCLAP, a method for regularizing the contrastive learning optimization process of standard objectives (InfoNCE, SigLIP) to improve performance on downstream retrieval and 0-shot classification tasks.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Motivation is reasonably strong, and mathematical rigor to the proposed method is appreciated.\n- Despite the overall mathematical maturity of SupCLAP, the proposed method is still intuitive and straightforward, aiding the clarity of the paper.\n- The experimental results, while generally modest, show clear gains to the using the proposed method relative to the baselines.", "weaknesses": "In general, there is not much massively wrong with the present manuscript, my two biggest concerns are:\n- It would be useful to report confidence intervals for the retrieval / classification experiments, as the results are generally modest, and it is hard to tell whether this gain in performance is truly statistically significant.\n- Overall, there is not much clear evidence as to *why* SupCLAP seems to work better empirically. While there is ample connection to *in theory* what might be going on during optimization, this is never attempted to be empirically confirmed. It would be very useful to analyze parts of the optimization process for SupCLAP (vs InfoNCE or SigLIP) such as gradient norm, gradient variance, or simply convergence speed to see whether the surmised theoretical reasons for its performance actually translate in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1AE6PAHFfA", "reviewer": "Reviewer_pCqQ", "summary": "The paper proposes SupCLAP with a Support Vector Regularization (SVR) that shifts the text embedding towards its positive audio.They add an extra contrastive term, claiming to suppress the perpendicular component of the negative “pushing force” and reduce optimization trajectory drift.  And this will help improve performance.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper gives a novel regularization term on the InfoNCE loss,  their experiments show that this regularization indeed bring benefits. They also provide their intuition in section 2.  And The writing of this paper is good.", "weaknesses": "1. In the section 2, the authors discuss the optimization trajectory drift. However, the causal link “lower perpendicular component → better performance” is not theoretically or empirically established. So this section is not very convincing to me. The author may provide theoretical or empirical evidence.\n2. For the InfoNCE loss, we originally want the text embedding being away from the negative sample. So why we need to reduce the perpendicular component of the negative “pushing force”? \n3. Moreover, the proposed Support-Vector Regularization (SVR) seems closely related to simply re-scaling or re-weighting the contrastive gradients. It would strengthen the paper if the authors compared their method to simpler baselines—e.g., emphasizing the parallel component only or giving positive sample gradient more weights—to demonstrate that the observed improvement is not merely a side-effect of gradient magnitude control.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iTqOHiwsno", "reviewer": "Reviewer_EVJV", "summary": "This paper studies contrastive language–audio pretraining (CLAP) and identifies a phenomenon called optimization trajectory drift, caused by the perpendicular component of the negative-sample pushing force in contrastive learning. The authors propose Support Vector Regularization (SVR), which introduces auxiliary support vectors and a semantic radius to selectively suppress this perpendicular component while preserving useful negative-sample information. They present both static and dynamic strategies for modeling the semantic radius. Experiments across monolingual and multilingual audio–text retrieval, as well as zero-shot classification, show consistent improvements over InfoNCE and SigLIP with negligible overhead.", "questions": "1. The dynamic semantic radius relies on an unsupervised predictor. How robust is it under noisy embeddings (e.g., low-resource or weak encoders)? Can the authors show sensitivity or variance analysis to demonstrate stability?\n\n2. The radius shrinks during training, but the explanation is heuristic. Can the authors provide clearer evidence that its values correlate with semantic difficulty (e.g., hard-negative density or retrieval errors)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The work offers a fresh angle by decomposing contrastive gradients into pulling and perpendicular pushing components, which provides new insights into training instability in CLAP.\n2. The SVR formulation is presented with solid mathematical justification, and the derivations clearly show how trajectory drift is effectively suppressed.\n3. Experiments across retrieval, classification, and multilingual tasks, along with ablations, demonstrate consistent improvements while keeping the computational cost minimal.", "weaknesses": "1. While the paper offers a new analytical angle, the proposed SVR is similar to a regularization tweak of InfoNCE and conceptually close to earlier margin-based or support-vector–style methods.\n\n2. The dynamic semantic radius predictor is unsupervised and fragile under noise. Its meaning is not clearly explained, and why it shrinks during training or how it reflects semantic difficulty is described more heuristically than rigorously.\n\n3. The baseline setup makes comparisons harder to follow. In Tables 1 and 2, many models are cited only by reference instead of name. Some strong recent baselines are also missing, including DS-CLAP (Liu et al., 2024), Cacophony (Zhu et al., 2024), and ATRI (Yin et al., 2025).\n\n4. The experiments are limited to AudioCaps and Clotho, with multilingual results relying on machine translation. Compared with recent work that uses larger datasets such as AudioSet, WavCaps, and LAION-Audio and covers broader tasks like captioning or grounding, the generalization claims are less convincing.\n\n\n[a] Liu et al., 2024, DSCLAP: Domain-specific contrastive language-audio pre-training\n\n[b] Zhu et al., 2024, Cacophony: An improved contrastive audio-text model\n\n[c] Yin et al., 2025, ATRI: Mitigating multilingual audio-text retrieval inconsistencies by reducing data distribution errors", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)