A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity
Authors: Giordano Cicchetti
Eleonora Grassucci
Danilo Comminiello
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Learning
Contrastive Learning
Multimodal Alignment
Primary area: deep_learning
TL;DR: TRIANGLE is a novel multimodal contrastive method aligning three modalities altogether without needing pairwise similarities and establishing new SOTA in several downstream retrieval tasks.

Abstract: Multimodal learning plays a pivotal role in advancing artificial intelligence systems by incorporating information from multiple modalities to build a more comprehensive representation. Despite its importance, current state-of-the-art models still suffer from severe limitations that prevent the successful development of a fully multimodal model. Such methods do not provide indicators that all the involved modalities are effectively aligned. As a result, a set of modalities may not be aligned, undermining the effectiveness of the model in downstream tasks where multiple modalities should provide additional information that the model fails to exploit. 
In this paper, we present TRIANGLE: TRI-modAl Neural Geometric LEarning, the novel proposed similarity measure that is directly computed in the higher-dimensional space spanned by the modality embeddings. TRIANGLE improves the joint alignment of three modalities via a triangle‑area similarity, avoiding additional fusion layers.
When incorporated in contrastive losses replacing cosine similarity, TRIANGLE significantly boosts the performance of multimodal modeling, while yielding interpretable alignment rationales. Extensive evaluation in three-modal tasks such as video-text and audio-text retrieval or audio-video classification, demonstrates that TRIANGLE achieves state-of-the-art results across different datasets improving the performance of cosine-based methods up to 9 points of Recall@1.

A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity
Giordano Cicchetti, Eleonora Grassucci, Danilo Comminiello Department of Information Engineering, Electronics, and Telecommunications Sapienza University of Rome, Italy
{name. surname it
Abstract
Multimodal learning plays pivotal role in advancing artificial intelligence systems by incorporating information from multiple modalities to build a more compre- hensive representation. Despite its importance, current tfate-of-the-rt models still suffer from severe limitations that prevent the successful development of a fully multimodal model. Such methods may provide indicators that all the involved modalities are effectively aligned. As a result, some modalities may not be aligned, undermining the effectiveness of the model in downstream tasks where multiple modalities should provide additional information that the model fails to exploit. In this paper, present TRIANGLE: TRI-modAl Neural Geometric LEarning, the novel proposed similarity measure that is directly computed in the higher-dimennional space spanned by the modality embeddings. TRIANGLE improves the joint alignment of three modalities via triangle-area similarity, avoiding additional fusion layers or pairwise similarities. When incorporated contrastive losses replacing cosine similarity, TRIANGLE significantly boosts the performance of multimodal modeling, while yielding interpretable alignment ratio- nales. Extensive evaluation in three-modal tasks such video-text and audio-text retrieval or audio-video classification, demonstrates that TRIANGLE achieves state-of-the-art results across different datasets improving the performance of cosine-based methods up to 9 points of Recall@1 Code and checkpoints available at hhttps://gitte com/ http://ssammm/A/AA// TRIAAGLE/////
1 Introduction
Humans perceive reality mixture of signals coming from different modalities registered through diverse senses, such as sounds perceived by the ears or vision by the eyes, and process these multimodal inputs understand the scene. In the last few years, foundational models have attempted to emulate this understanding system by aligning pairs of modalities using contrastive learning approaches. The first valuable contribution in this sense, CLIP Radford al. (2021), aligns image and textual latent representation with contrastive loss based on the cosine similarity between the two vectors. CLIP established the de-facto receipt for aligning multimodal latent features and the subsequent methods, such as CLAP for audio-text alignment Elizalde et al. (023), have followed the same scheme.
Later, several works focused on extending the CLIP approach to more than two modalities, incorpo- rating audio, depth, or other modalities to build more representative embedding space Girdhar et al. (2023): 7hu et al. (2024); Ruan et al. (202)): Wang al. (2024). All these methods still rely on the pairwise cosine similarity between modalities. Specifically, they select anchor modality, which may be image Girdhar et al. (2023), audio Ruan et al.) (2023), or language Zhu et al. (2024), and then align the embeddings of all the other modalities oye-by-one to the anchor embedding. However,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

"A dog is growling" 0.02
SOTA (V+T)
x
0.03
"A dog is howling"
TRIANGLE (VV - ours)
0.66
0.14
"A dog is barking"
0.32 0.83
Figure I: Current ofate-of-the-aa (SOTA) methods struggle incorporate the third audio modality and fail in video-to-text tasks, incorrectly assigning captions such as "A dog is howling", as the frames may be similar. On the contrary, the proposed TRIANGLE method effectively leverages all modalities together, bringing crucial information to discriminate among the captions, and retrieving the correct caption "A dog is barking".
this alignment only ensures that all modalities are aligned with the anchor, without providing any indicators for the aligmment between non-anchor modalities. Therefore, if the video aligned with the language the anchor modality, and the audio is also aligned with the language, there are no geometric indicators that the audio and video modalities are aligned with each other. Such a limitation may undermine the effective upplicability of these models in real-world multimodal scenarios. Indeed, they mostly perform experiments on two-modal tasks where they just test the alignment between one modality and the anchor one, as their pairwise similarity loss does not allow them to test three modalities at once. However, even in these scenarios, such models often fail when sample data includes third modality that may be crucial to accurately discriminate and truly understand the content Yoon et al.) (202)). As an example, in widespread video-text retrieval tasks, conventional state-of-the-ar (SOTA) models focus on video frames to solve the task, while the audio information is instead often crucial to properly retrieve the correct caption. Figure shows an example of this task in which the SOTA model fails it is not able leverage the audio information, while the proposed one exploits both the modalities and correctly retrieves the caption. To partially overcome the limitation due to the use of only two modalities, other works have proposed to fuse multiple modalities via MLP layers Chen et al. (2023b), supplementary loss functions Wang et al. (2024); Chen al. (2023b) or additional architectural strategies Li et al. (202T). Very recently, GRAMCCiccher et al. (025) and Symile Saporta et al. (2024) proposed general alternative similarity objectives that scale to an arbitrary number n of modalities by minimizing. respectively, the parallelotope volume and corr-loorrlltion bound.
Unfortunately, while these methods obtain improved alignment performance, some of them still lack geometrical indicators and fail to provide insights on how each modality impacts the final outcome, thereby reducing the interpretability of the results as well. Additionally, the majority of the recent works come up with their own datasets Zhu et al. (0024) Chen et al. (2023b a): Saporta et al. (2024). placing strong emphasis on data quality and on scaling up neural models to billions of parameters Wang et al. (2024), yet still relying on the same similarity measure.
In this paper, we present TRIANGLE: TRI-modAl Neural Geometric LEarning, novel method capable addressing the aforementioned limitations. The proposed method directly works within the space spanned by all embedding vectors, aligning them altogether. In this space, the extremities of the vectors form the vertices of a 2D polygon, whose shape and area show interesting insights into the semantic affinity among modalities. In the most common case of video, audio, and language modalities, such a polygon is a triangle, whose area is strictly related to the alignment of all the modalities. Geometrically, the smaller the area of the triangle, the closer (i.e., the more aligned) the vectors of the three modalities are to each other. Conversely, a larger area indicates that the vectors are more orthogonal, suggesting they point in opposite directions and probably represent misaligned data. Therefore, TRIANGLE proposes using triangle area minimization as the similarity metric to align three-modal vectors their higher- dimensional space, without relying on pairwise comparisons. This method effectively integrates the information from the three modalities, as shown in Fig. [] where TRIANGLE exploits both the video and audio information to retrieve the correct caption. TRIANGLE sets new state of the art in video-text retrieval across datasets such as MSR-VTT, DiDeMo, ActivityNet, and VATEX, as well as in audio-text retrieval and classification in AudioCaps and VGGSound, all without requiring new datasets additional layers, underscoring its superiority

---

over conventional methods relying on cosine similarities between pairs of modalities and over generic methods designed for joint alignment.
In summary, our contributions are:
1. TRIANGLE, a three-modal alignment method is introduced. TRIANGLE encourages align- ment among three modalities, leveraging all the modalities together to perform downstream tasks.
2. TRIANGLE offers an easily interpretable measure of alignment of three modalities, feature lacking in previous models.
3. TRIANGLE establishes new state-of-the-ee results in video-text and audio-text retrieval across diverse datasets and scenarios.
2 Related Work
Two-modal Alignment. In 2021, CLIP Radford CLIP|Radford et al. (2021) and ALIGN Jia et al. (2021) paved the way for foundational models capable of aligning two modalities, specifically focusing on images and text. From that framework, several works followed improving the performance and the alignment Uesaka et al. (2024); [harco et al. (202T): Zhai et al. (2023); Grassucci et al. (2025). CLIP has also served the backbone for extending such alignment capabilities to different modalities such as audio and text as in CLAP|Elizala et al. (2023), video and text in CLIP4Clip Luo et al. (2021),
point clouds and text in Zhang al. (202T). Each of these models is based on contrastive learning strategies, which bring similar vectors closer together while pushing dissimilar embeddings further apart. They all rely on the pairwise cosine similarity in the contrastive loss.
Three-modal Alignment. More recently, several attempts have been made to better capture the complexity of the reality around us, which usually involves more than two modalities. CLIP4VLA Ruan et al. (2023) proposed to incorporate the audio modality in the CLIP framework, aligning video, audio, and text modalities in pairs using the audio embedding anchor. Lately, ImageBind Girdhar jet al. (2023) introduced a multimodal pre-trained framework that includes multiple modalities such as depth and infrared, considering the image modality bridge to all others. Building on this approach, LanguageBind Zhu et al. (2024) showed that using the text modality as the anchor is more effective than using the image modality. Concurrently, different approaches for building large three- modal pretrained embedding models have emerged, including VALOR Chen et al. (2023a). VAST Chen et al.) (2023b), mPLUG-2 Xu et al. (2023), and Intern Video2 Wang al. (2024), all collecting large pretraining datasets and bringing architectural improvements to enhance model performance. However, none of these methods work on the higher-dimensional space spanned by the multimodal vectors, as they primarily rely on the cosine similarity computed on the 2D plane defined by two modalities, on architectural fusion strategies for multiple modalities. In practice, these approaches are unable to fully leverage the effective multimodal information needed to solve downstream tasks comprehensively. More recently, GRAM Cicchetti al.) (2025) and Symile Saaporta al. (2024) proposed two loss functions that aim to better capture the overall aligmment of n modalities. However, despite their generalizability, we will show that they underperform in three-modal (video-audio- text) tasks, showing that an objective tailored to triplets can exploit modality- features more effectively than general losses.
3 TRIANGLE: TRI-modAl Neural Geometric LEarning
3.1 Problem Formulation
Multimodal representation learning aims to derive latent representations from co-occurrent input data modalities. The i-th modality encoded in a latent representation in an n-dimensional space using an encoding function M, - Z, with 2 E R². In the case of video-audio-text representation we have a tri-modal representation with three encoding functions: ey : V - 2 visual encoder, e Au Z audio encoder, er T - Z text encoder.
3

---

EVAClip- VIT-G
Video
Video Embed
Video
grey dolphin comes weter gel blve toy in poor
Taxt
Text Embed
Clasa
Toxt
Token
BEATs
Embed
Audio
Audio
Token
Class
TRIANGLE
Figure 2: TRIANGLE builds the high-dimensional space spanned by the embeddings generated by the encoders. The embedding vectors of the three modalities lie within a unit hypersphere, where they form triangle. The area of this triangle is unambiguous measure of their alignment.
Conventionally, the similarity between two modalities M,, Mj is obtained by computing the cosine of the angle fig between them:
(,, M,) cos (0.)) |M,l| ||,]]|
(1)
where (M, My) is the dot product between modality M, and modality Mj. and |M,]] is the norm.
Intuitively, the closer the cosine value is to 1, the closer the two vectors are in the hyperdimensionall space, meaning that the two embeddings represent similar original content. CLIP Radford al. (2021) introduces this intuition in the multimodal contrastive loss function, which, given the textual representation t and the image one i, the number of elements and the temperature 7 takes the form:
exp(i, L//T)
L12T
log
(2)
B
Σ 5 exp(i, t//T)
i=1
B
(tp(t,Ti./ Σ exp(t) i5/T)
CT21
log
(3)
B
1=1
Whenever the input includes third modality that needs to be considered by the model, they cannot be naturally compared using cosine similarity. Indeed, cosine similarity is not defined for higher- dimensional spaces and instead, it applies a projection into 2D space. This issue occurs in both the training and inference phases. During training, the most common solution cireumvent this problem is the anchor selection Girdhar et al. (2023); Zhu et al. (2024): one modality is chosen as the anchor and the other modalities are pairwisely aligned the anchor. This approach guarantees alignment between any modality M, and the anchor. However, nothing can be inferred about the alignment between the other modalities. During inference, the problem becomes even worse since conventional methods lack a direct mechanism to compute similarity among three embedding vectors. Therefore, they are forced to rely on only two modalities or develop suboptimal neural fusing strategies. For instance, LanguageBind Zhu et al.) (2024) attempts to linearly combine two modalities and then compute similarity with third one, while methods like UMT et al. (2022), m- m-PLUG2 Xu et al.) (2023) and VAST Chen et al. (2023b) introduce layers that fuse two or more modality embeddings before computing cosine similarity with the remaining one.
Although these suboptimal multimodal strategies show slight improvements in metrics, they fail to fully and effectively exploit the information from the third modality, especially during inference and in downstream tasks, thereby limiting the practical utility of these methods in real-world applications.
3.2 The TRIANGLE Solution
We aim at aligning three modalities directly their natural higher-dimensional space without relying on 2D projections and exploiting the contribution from all the modalities together. To this end,

---

tel
Flat Triangle
Cosine Regularizatio
(a) Large Area Misaligned Vectors
(b) Small Area Aligned Vectors
(c) Cosine Regularization
Figure 3: (a-b)TRIANGLE area as a measure of similarity. Misaligned vectors, large area (a); Aligned vectors, small area (b). (c) In the case of flat triangle, we apply the cosine regularization, bringing the alignment information between the two modalities on which the downstream task is about. In the example of video-text retrieval, regularize with the cosine between video and text.
introduce TRIANGLE: TRI-modAl Neural Geometric L.ariing, which builds the higher-dimensional space spanned by the modality vectors and computes novel similarity measure that can be readily applied in downstream tasks.
Idea. A generic embedding vector can be interpreted as point in a multidimensional space R", where is the embedding dimension. In the case of three modalities embedding vectors with unitary norm, the resulting points lie in the unit hypersphere and unambiguously determine a hyperplane (as long those points not aligned on the same line). The three modalities embedding vectors draw a triangle, which lies on the aforementioned hyperplane, as the plot in Fig. 2 shows. The area of the triangle spanned by those vectors determines measure of the similarity of such embeddings and its computation is easy in Rn as relies on three dot products.
Definition 3.1 (The TRIANGLE area is measure of similarity). The area A of a triangle in IR" given by:
1 (), (u, v)2,
A
where ( is the dot product and x-y. -. the three embeddings x, y, and z of the three modalities. Proof in Appendix. Intuitively, the smaller the area, the closer the three vectors are, meaning they are well-aligned. Conversely, the largest area occurs when two vectors point in opposite directions, with a corresponding cosine similarity equal to -1, while the third vector is orthogonal them, resulting in an isosceles triangle, as shown Fig. Therefore, the area of the triangle serves as direct measure of the similarity among all three vectors, eliminating the curse for pairwise computations.
z are two triangle sides computed among
3.3 TRIANGLE Constrastive Loss
In downstream tasks, common models such as ImageBind Girdhar et al. (023), Language Bind Zhu al. (2024), or VAST(Chen et al. (2023b) rely on the cosine similarity between the two modalities involved in the task. For example, in the widely used task of video-text retrieval, these models compute the cosine similarity between the embeddings of text and video frames, thus assigning the caption to the video with the highest similarity. In such common solution, including additional modality, that could significantly boost the model performance (e.g., the audio the aforementioned video-text retrieval task), is challenging, as there is no higher-dimensional shared space in which all modalities can aligned.
We propose to address this widespread limitation by leveraging the TRIANGLE formulation in 99) conventional contrastive losses. replacing the common pairwise cosine similarity. The proposed brand-new TRIANGLE contrastive losses follow:
CDr
1
Σ
log
v,, a:)/T)
(5)
B
Σ/1 (exppxp(ppp Vi, v;,):)/7)
5

---

B
log exp(-A(t, Vs a.)/7)
LTD =
(6)
Ej=1 exp(- vya,)/r)
By exploiting the triangle area A in the contrastive loss, from all modalities and assign the caption to the video according to the minimum area formed by the vector modalities, ensuring a more holistic and accurate alignment. We show how successfully leveraging the three modalities composing the data significantly brings improved performance.
effectively integrate information
In addition, to further guide the training process and following Chen et al. (2023b), we also employ data text matching (DTM) loss:
LDMM = F.L..a)-(T...A)) Patm + (1 - 1/ log(1 (1 Pdtm))
in which Patm are the output probabilities when feeding again caption tokens into the text encoder activating cross-attention layers to attend to the concatenated (along the sequential dimension) audio and video features as conditioning. Summing up everything, the final loss function is:
GTOT = (DDer + FT2D) + ALDTM-
(8)
Downstream Tasks Regularization. To further enhance the performance of TRIANGLE in mul- timodal alignment and effectively manage all possible vector positions in the higher-dimensional space, we propose to add a regularization to the area minimization process downstream tasks. This regularization leverages two-dimensional cosine similarity to complement the proposed alignment strategy with an existing ensures the contribution of all the modalities, while cosine regularization specifically controls only the alignment of the most relevant modalities to the downstream task. This dual approach guarantees true alignment in space, even in exceptional cases. A visual example is shown in Fig. (c).
similarity measure. In this way, area minimization
Formally, let x, y, z be the three modality embeddings, and define x y, V = x as the sides of the triangle. In a retrieval downstream task between the modalities x and y, whose angle is 8xy. the proposed alignment strategy can be expressed as:
A= 2 u1 (v (u, v)2 - a cos Oxy,
where the first term on the right side represents the of the triangle formed by the embeddings and a is the regularizing hyperparameter that balances the contribution of the area minimization and the cosine similarity in the alignment strategy.
This formulation ensures that the area minimization captures the alignment across all three modalities, while the cosine similarity regularization fine-tunes the alignment specifically for the retrieval task between x and y.
3.4 Advantages of the TRIANGLE
Firstly, TRIANGLE effectively models three modalities directly in the higher-dimensional space, without the need to employ some fusion strategies for any two of them. By avoiding the use of fusing layers for multiple modalities, TRIANGLE can work with different modalities: regardless of which modalities are involved, they will always define a triangle for which the area can be computed. This a crucial advantage that makes the proposed method both portable and versatile across different types of data. Moreover, the TRIANGLE objective offers an intuitive indicator of how closely the three embeddings cluster: low value of the area A indicates that the embeddings are well-aligned, while a high value suggests they are far apart. Indeed, in the Appendix, we show how the area value among matching embeddings decreases during training. These geometric constraints are pivotal when performing downstream tasks unseen datasets, thus ensuring the effective alignment of modalities. In practice, the true alignment of the three modalities directly improves the model performance, particularly with sample data in which the information contained in the third modality is essential to solve the task, as shown in Fig. [ Finally, in terms of computational load, TRIANGLE brings negligible increment of the computational time with only 0.0016 seconds compute the area of three vectors of dimension 2048 against the 0.0001 seconds of the cosine similarity computation with a batch of size 256 on RTX4080 in inference.

---

Table 1: Zero-shot multimodal text-to-video (T2V) and video-to-text (V2T) retrieval R@ results. Increment points computed wrt VAST with same modalities, number of parameters, and encoders. The difference between VAST and TRIANGLE results is statistically significant (p 0.001).
MSR-VTT
DiDeMo
NetivityNet
VATEX
Modality
T2V
V2T
T2V
V2T
T2V
V2T
T2V
V2T
UMT Liu ct al. 2022
T-V
33,3
34.0
31.9
Omni VL Wang al.
2022a
T-V
34.6
33.3
UMT-L Li
al. 2025
T-V
40.7
37.1
48.6
49.9
41.9
39.4
TVTSv2 Zeng al. 2023
T-V
38.2
34.6
ViCLIP Wang
202.5
T-V
42.4
41.3
18.4
27.9
15.1
24.0
VideoCoCa Yan ct [2022]
T-V
34.3
64.7
34.5
33.0
53.2
73.6
Norton |in ct 22024
T-V
10.7
ImageBind Girdhar al.
2023
T-V
36.8
InterViideo-1 Wang et
2022b
T-V
40.7
39.6
31.5
33.5
30.7
31.4
49.5
69.5
HiTcA Ye et
2022
T-V
34.4
43.2
mPLUG-2
Xu Xu
2023
T-V
45.7
VideoPrismn- Zhao el al.
2024
T-V
51.4
50.2
49.6
47.9
62.5
77.1
LanguageBind Zhu et al. 2024
T-V
44.8
40.9
39.9
39.8
41.0
39.1
GRAM Cicchell
2025
T-VA
54.2
50.5
$4.2
52.2
59.0
50.4
83,9
79.2
VAST Chen el al. 2023b
T-VA
49.3
43.7
49.5
48.2
51.4
46.8
80.0
77.3
TRIANGLEE (ours)
T-VA
55,2
52.5
54.9
53.1
59.7
54.1
83.9
80,9
TRIANGL.E Improvement wrt VAST
+5.9
+8.8
+5.44
+4..9
+8.3
47.3
+3.9
+3.6
4
Experiments
We run experiments on seven popular benchmarks and we conduct three different types of evaluation: first, a vanilla experiment in a controllable environment, then extensive experiments with pertaining and downstream tasks, and finally the training from scratch.
4.1 Vanilla Experiment
Let us consider experiment a controllable environment with three modalities and latent space of dimension We build a framework comprising three modalities: images with the MNIST dataset, audio with the AudioMNIST dataset Becker et al. (2023), and text, with the textual labels associated with the digit numbers from 0 to 9. The objective is correctly retrieving the label from the image and audio represen- tations. We develop three vanilla encoders, two-layer convolutional encoder for images, three-layer convolutional encoder for audio spec- trograms, and Word2Vec for text data. We run trainings with different losses: the conventional pairwise cosine-based loss, therefore comput- ing the cosine similarity between text and video, and then text and audio, selecting as anchor the text like Zhu al. (2024); the Symile (Saooral
100
4x Speedup
90
80
Worse convergence
70
Stuck local minima
50
40
Cosine
Symile
20
GRAM
10
TRIANGLE
50
100
200
Epochs
Figure 4: The proposed TRIANGLE shows faster and better convergence overall.
et al. (2024) loss function, the GRAM one Cicchetti et al. (025), and the proposed TRIANGLE in (5) and (6) Figure shows the results of the experiment. The proposed TRIANGLE obtains better. smoother, faster convergence, up to 4x speedup to reach 90% than the conventional cosine- based loss than GRAM, while achieving the best R@1 score overall. Symile, instead, stuck in local minimum that prevents it from surpassing an R@1 higher than 50. These results show the superior performance of TRIANGLE in better shaping unified latent space to favor the alignment of all the modalities, leading to improved retrieval performance.

---

4.2
Extensive Experimental Evaluation
4.2.1 Settings
The TRIANGLE model exploits the novel contrastive loss functions defined in (5) and (6) to fully align three modalities in a joint fashion. For the encoder models, we select as backbone the SOTA model designed for multiple modalities, Chen et al. (2023b), thus employing BERT-B, BEAT's Chen al. (202cc), and EVAClip-Viiie ViT-G [Sun al. (2023) as text, audio, and video encoder, respectively. We pretrain the TRIANGLE model on top of VhST et al. (2023b) (removing the fusing layers) on a subset of 150k samples randomly selected from the VAST27M dataset Chen et al. (0023)). We 150k samples to VAST, but this model quickly
also applied the additional pretraining strategy overfits, bringing no improvements. This is probably due to the convergence that the VAST model reached on the whole VAST27M dataset. On the contrary, TRIANGLE is able reshape and remodel the latent space learned by VAST encoders towards a more aligned and unified multimodal space. Following the experimental receipts of existing embedding models in the literature, we validate the TRIANGLE performance on three fundamental downstream zero-shot tasks: video retrieval (text- to-video and video-to-text), audio retrieval (text-to-audio), and audio-text classification. Superior performance on these tasks suggests a better alignment of the various modalities within the shared embedding space, highlighting the effectiveness of the proposed TRIANGLE method.
4.2.2 Zero-Shot Video Retrieval
The video retrieval task can be divided into two subtasks: 1) Text-to- text-to-Video Retrieval (T2V): Given natural language query, find the most relevant video from a set of candidate videos. 2) Video-to Te-e-tto-Text Retrieval (V2T): Given candidate video, find the most relevant natural language query.
We evaluate TRIANGLE zero-shot on a wide range of popular benchmarks, namely MSR-V VTT Xu et al. (2016). DiDeMo Hendricks et al. (2017). ActivityNet Caba Heilbron et al. (2015) and VATEX Wang et al. (2019). as shown in Tab. I for video-text tasks. These datasets contain the three modalities of interest (i.e., video frames, audio waves, and video captions). For each dataset, we use video frames randomly selected, as in CCen et al. (20236). More details about dataset sizes and hyperparameters in the Appendix. We report R @@1 scores for video-to-text and text-to-video results in Tab. [ The results in Tab. Jnniqully demonstrate the superiority of TRIANGLE against established SOTA models, both vision-only and multimodal models. Indeed, both in V2T and T2V tasks, TRIANGLE obtains the best performance overall. Spefically, with respect to VAST Chen et al. (2023b) which has the same encoders, number of parameters, and pretraining dataset, TRIANGLE improves the performance up to 9 points and overall by a minimum of 4 points of R @1 Such a breakthrough result is due to the ability of TRIANGLE to effectively leverage the third modality, thus leading to enhanced performance. Indeed, the audio modality introduces more information essential for solving the text-to-video task, especially in video with strong audio component that is reflected in the caption but that cannot be exploited solely from the frames. While VAST also uses three modalities, it fuses visual and audio embeddings using an MLP before comparing them to text via cosine similarity. This approach does not provide any geometric indicator of the aligmment, thus limiting its effectiveness in fully aligning the modalities. Instead, TRIANGLE improves this approach by using the straightforward formulation in (9) incorporated in the brand-new contrastive losses in (5). which shows clear geometric explanation and ensures a more robust alignment of the modalities. Furthermore, TRIANGLE outperforms generic m-modal methods like GRAM Cicchetti GRAMMCicchetti jt al. (2025) and Symile (Saporta et al. (2024), proving that the proposed similarity tailored for three modalities can better exploit three-modal cues than fully general n-modal losses.
4.2.3 Zero-Shot Audio Retrieval and Classification
We evaluate TRIANGLE in zero-shot audio retrieval on two popular benchmarks: AudioCaps Kim al. (2019) and VGGSound Chen et al. (2020). On Audiocaps we compute the zero-shot text-to- audio retrieval task. On VGGSound we compute zero-shot audio-visual classification. Due to the new YouTube policies, the complete VGGSound test set is not available anymore, thus we compute all the metrics (including comparisons) on subset comprising 5k samples. Results for both the datasets and tasks are reported in Tab. 2 in terms of Recall @ and Recall 10. TRIANGLE clearly outperforms all previous methods by up to 12 points, thus establishing new state-of-the-arr results across both examined datasets and tasks. Such a large improvement due to TRIANGLE F effective leverage of the third modality. Including the video modality introduces crucial information essential
8

---

Table 2: Zero-shot text-to-audio retrieval results on AudioCaps and audio-text classification results on VGGSound 5K.
AudioCaps
VGGSound 5K
Modality
R@1
R@1
R@
AVFIC
Nagrani et al.
2022
T-A
8.7
37.7
AVFIC Nagrani et al. 2022
T-AV
10.6
45.2
VIP-ANT Zhao cl 2022
TAA
27.7
37.7
ImageBind Girdhar
2023
T-A
9.3
42.3
LanguageBind Zhu
T-A
19.7
67.6
23.8
57.1
LanguageBind Zhu
2024
T-V
37.2
62.0
VAST Chen et al.
A125h
T-V
38.7
72.8
VAST Chen et 2023b
T-A
25.6
56.2
GRAM Crechelli el
2025
T-AV
33.2
75.3
40.6
78.1
VAST Chen et 20255
T-AV
32.1
65.4
39.6
74.5
TRIANGI (ours)
T-AV
32.2
77.1
44.8
80.0
Improvement wT VAST
+0.1
+11.7
45,22
+5.5
for solving the audio retrieval task, particularly for captions that are challenging infer using the audio modality alone. TRIANGLE obtains improved scores also in audio-visual classification, still boosting the performance by more than points in both R@ and R 10. 4.3 Learning the Space from Scratch
We perform a deeper study on the abil- better model
Table 3: Training from scratch and ablation study on MSR-
ity of TRIANGLE the latent space by letting TRIAN- GLE losses learn from scratch the MSR-VTT dataset for the mul- timodal -ot-to--audio/vieo (T2AV) and vndio/video-to-text (AV2T) tasks. In this scenario, we select the same three encoders and then compare VAST (pairwise cosine similarity- based with fusing MLP layers) Chen (2023b), Symile Saporta al. (2024) as it leverages the total correla- tion align multiple modalities, GRAM Cicchett et al. (2025) that can align ri modalities through volume, and the proposed TRIANGLE loss function. These models are the sole ones to effectively perform multimodal tasks, although TRIANGLE the only one tailored for the specific three-modal case. The encoders possess pretrained knowledge in this experiment, and the four methods are left free to shape the latent space according to the loss minimization. Table reports R and @ 10 scores that explicitly highlight the superior performance of TRIANGLE Despite the greater generality, Symile and GRAM underperform TRIANGLE on all three- modal retrieval benchmarks. Once again, this suggests that objective tailored triplets can exploit modality-specit cues more effectively than fully general n-modal losses.
VTT, with same encoders and different loss functions (cosine, Symile, GRAM), and the proposed TRIANGLE method w/ and w/o the LDTM- Statistically significant at 0.05.
Training from scratch
T2AV
AV2T
R @
R@ 10
R@1
R@ 10
VAST Chen et al. 2023b
36.5
79.3
35.5
77.3
Symile Saporta el al. (2024
0.3
3.1
0.4
3.6
GRAM Cicchetti et al. 2025
38.9
80.8
41.9
79.5
TRIANGLE w/o LDTM
33.3
74.4
40.4
81.7
TRIANGLE (ours)
39.4
81.8
41.9
80.0
Ablation Study. We perform an ablation study on the effect of the LDTM loss function in TRI- ANGLE. From Tab. is clear that the usage of such a loss function is crucial to obtaining better performance, and the proposed TRIANGLE configuration achieves the best scores overall. We perform more ablation studies in the Appendix.
5 Conclusion
We introduced TRIANGLE: TRI-modAl Neural Geometric LEarning, novel method for the align- ment of three modalities. Addressing the challenges of modality alignment that limit the performance of SOTA multimodal models, TRIANGLE introduces novel similarity measure computed directly in higher-dimensional space spanned by the modalities embeddings. This promotes joint alignment across the three studied modalities without additional anchors additional fusion layers. Our exten- sive evaluations demonstrate that TRIANGLE not only enhances the interpretability of the alignment process but also reaches zero-shot SOTA performance in multimodal downstream tasks.

---

Acknowledgments
This work was partially supported by the European Union under the NRRP of NextGenera- tionEU, partnership "Future Artificial Intelligence Research" (PE00000013 - SPOKE 5 CUP B53C22003980006 FAIR: High Quality AI) and partnership on "Telecommunications of the Future" (PE00000001 program "RESTART"), partially by the Progetti di Ateneo of Sapienza University of Rome under grant RM113188F75F8077 and RM12419 RM1241910FC4BEEA 10FC4BEEA, and partially by the Italian Min- istry of University and Research (MUR) within the PRIN 2022 Program for the project "EXEGETE: Explainable Generative Deep Learning Methods for Medical Signal and Image Processing", under grant number 2022ENK9LS, CUP B531D23013033006
References
Becker, S., Vielhaben, Ackermann, M., Müller, K.-R., Lapuschkin. S., and Samek, W. AudioM- NIST: Exploring explainable artificial intelligence for audio analysis on simple benchmark. Journal of the Franklin Institute, 2023.
Caba Heilbron, F., Escorcia, V., Ghanem, B., and Carlos Niebles, J. Activitynet: A large-scale video benchmark for human activity understanding. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Chen, H., Xie, W., Vedaldi, A., and Zisserman, A. Vggsound: A large-scale audio-visual dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725 2020.
Chen, S., He, X., Guo, L., Zhu, X., Wang, W., Tang, J., and Liu, J. VALOR: Vision-audio-languaa omni-perception pretraining model and dataset. ArXiv preprint: arXiv:2304.08345, 2023a.
Chen, S., Li, H., Wang, Q.. Zhao, Z., Sun, M.-T., Zhu, X., and Liu, J. VAST: A vision-audio-uubice text omni-modality foundation model and dataset. In Neural Information Processing Systems (NeurIPS), 2023b.
Chen, S., Wu, Y., Wang, C., Liu, S., Tompkins, D., Chen, Z., Che, W., Yu, X., and Wei, F. BEATs: Audio pre- pre-training with acoustic tokenizers. In International Conference on Machine Learning (ICML), pp. 5178-5193, 2023c.
Cicchetti, G., Grassucci, E., Sigillo, L., and Comminiello, D. Gramian multimodal representation learning and alignment. International Conference on Learming Representations (ICLR). 2025.
Elizalde, B., Deshmukh, S., Al Ismail, M., and Wang, H. CLAP: learning audio concepts from natural language supervision. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Fu, L., Datta, G., Huang, H., Panitch, W. C.-H., Drake, J., Ortiz, J., Mukadam, M., Lambeta, M., Calandra, R., and Goldberg, K. A touch, vision, and language dataset for multimodal alignment. In International Conference on Machine Learning (ICML), 2024.
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Misra, I. ImageBind one embedding space to bind them all. In IEEF/CVE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 15180-15190, 2023.
Grassucci, E., Cicchetti, G., and Comminiello, D. Closing the gap multimodal medical repre- sentation alignment. In IEEE Workshop on Machine Learning for Signal Processing (MISP), 2025.
Hendricks, I. A., Wang, O., Shechtman, E., Sivic, J., Darrell, T., and Russell, B. Localizing moments in video with natural language. IEEE International Conference on Computer Vision (ICCV), pp- 1804-5813.. 2017.
Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L. Openclip, July 2021.
10

---

Jia, C., Yang, Y., Xia, Y., Chen, Y.TT., Parekh, Z., Pham, H., Le, Q. V., Sung. Y.-H., Li, Z., and Duerig. T. Scaling up visual and vision- languaee representation learning with noisy text supervision. In International Conference
Machine Learning (ICMI.) 2021.
Kim, C. D., Kim, B., Lee, H., and Kim, G. AudioCaps: Generating Captions for Audios in The Wild. In NAACI-HLT, 2019.
Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, R., Xiong, C., and Hoi, S. C. Align before fuse: Vision and language representation learning with momentum distillation. In Neural Information Processing Systems (NeurIPS), 2021.
Li, K., Wang, Y., Li, Y., Wang. Y., He, Y., Wang, I.., and Qiao, Y. Unmasked teacher: Towards training-effiee efficient video foundation models. IEEE/CVF International Conference on Computer Vision (ICCV), pp. 19891-19903, 2023.
Lin, Y., Zhang, J., Huang, Z., Liu, J., Wen, Z., and Peng, X. Multi-gramularity correspondence learning from long-term noisy videos. In International Conference Learning Representations (ICLR), 2024.
Liu, S., Wu, Y.. Chen, C. W., Shan, Y., and Qie, X. UMT: Unified multi-mode transformers for joint video moment retrieval and highlight detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 3042-3051, 2022.
Luo, H., Ji, L.,, Zhong, M., Chen, Y., Lei, W., Duan, N., and Li, T. CLIP4Clip: An empirical study of clip for end end video clip retrieval. Neurocompurting. 508:293-3000 2021.
Nagrani, A., Seo, P. H., Seybold, B., Hauth, A., Manén, S., Sun, C., and Schmid, C. Learning audio-vider modalities from image captions. In European Conference on Computer Vision, 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A.,
Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
Ruan, L., Hu, A., Song, Y., Zhang, L.., Zheng, S., and Jin, Q. Accommodating audio modality in CLIP for multimodal processing. In AAAI Conference on Artificial Intelligence, 2023.
Saporta, A., Puli, A. M., Goldstein, M., and Ranganath, R. Contrasting with symile: Simple model-agnostic representation learning for unlimited modalities. In Neural Information Processing Systems (NeurIPS), 2024.
Sun, Q., Fang, Y., Wu, L. Y., Wang, X., and Cao, Y. EVA-CLIP: Improved training techniques for clip scale. ArXiv preprint: arXiv:2303. w:X.::2303.1388, 2023.
Uesaka, T., Suzuki, T., Takida, Y., Lai, C.-H., Murata, N., and Mitsufuji. Y. Understanding multimodal contrastive learning through pointwise mutual information. ArXiv preprint: arXiv:2404.1 19228, 2024.
Wang, J., Chen, D., Wu, Z., Luo, C., Zhou, L., Zhao, Y., Xie, Y., Liu, C., Jiang, Y.-G., and Yuan, OmniVL: One foundation model for Neural Information Processing, 2022a.
and video-language tasks. In Advances in
Wang, X., Wu, J., Chen, J., Li, L.. Wang, Y.-F., and Wang, W. Y. Vatex: A large-scale, high-quality multilingual dataset for video-ane and lad-lang research. In JEEE/CVF International Conference on Computer Vision, pp. 14991, 2019.
Wang, Y., Li, K., Li, He, Y., Huang, B., Zhao, Z., Zhang, H., Xu, J., Liu, Y., Wang, Z., Xing, S., Chen, G., Pan, J., Yu, J., Wang, Y., Wang, L., and Qiao, Y. Internvideo: General video foundation models via generative and discriminative learning. ArXiv prerint: arXiv:22/2.03
2022b.
Wang, Y., He, Y., Li, Y., Li, K., Yu, J., Ma, X. J., Chen, X., Wang, Y., Luo, P., Liu, Z., Wang. Y., Wang, I.,, and Qiao, Intern Vid: large- scale video-text dataset for multimodal understanding and generation. ArXiy preprint: 27X1v22307.06942, 2023.

---

Wang, Y., Ki, Li, Yu, J., He, Y., Chen, G., Pei, B., Zheng, R., Xu, J.. Wang. Z., Shi, Y., Jiang, T., Li. Li, S.. Zhang. H., Huang, Y., Qiao, Y. Wang, Y., and Wang. Intern Video2: Scaling video foundation models for multimodal video understanding. ArXiv preprint: arXiv:2403. 15377, 2024.
Xu, H., Ye, Q. Yan, M., Shi, Y., Ye, J., Xu, Li, C.,, Bi, B., Qian, Q. Wang, W., Xu, G., Zhang, J., Huang, S., Huang, F., and Zhou, J. mPLUG-2: A modularized multi-modal foundation model across text, image and video. In International Conference Machine Learning (ICML.), 2023.
Xu, J. Mei, T., Yao, T., and Rui, Y. Msr-vtt: A large video description dataset for bridging video and language. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Yan, S., Zhu, T., Wang, Z., Cao, Y., Zhang, M., Ghosh, S., Wu, Y., and Yu, J. VideoCoCa: Video-text modeling with zero-shot transfer from contrastive captioners. ArXiv preprint: arXiv:2212.04999, 2022.
Ye, Q. Xu, G., Yan, M., Xu, H., Qian, Q.. Q.. Zhang, J., and Huang, F. HiTeA: Hierarchical temporal- aware video-language pre-training. IEEE/CVE International Conference on Computer Vision (ICCV), pp. 13359-15370, 2022.
Yoon, S., Kim, D., Yoon, E., Yoon, H. S., Kim, J., and Yoo, C. D. HEAR: Hearing enhanced audio response for video-groundes dialogue. In Conference Empirical Methods in Natural Language Processing, 2023.
Zeng, Z., Ge, Y., Tong, Z., Liu, X., Xia, S., and Shan, Y. TVTSv2: Leamnin out-of-the-bee box spatiotemporal visual representations scale. ArXiv preprint: arXiv:2305. 14173, 2023.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sigmoid loss for language image pre-training.
IEEE/CVF International Conference Computer Vision (ICCV), pp. 11941-11955 2023.
Zhang, R., Guo, Z., Zhang, W., Li, K., Miao, X., Cui, B., Qiao, J., Gao, P., and Li, PoinCLIP:
Point cloud understanding by clip. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8542-8552, 2021.
Zhao, L., Gundavarapu, N. B.. Yuan, L., Zhou, H., Yan, S., Sun, J. J., Friedman, L., Qian, R., Weyand, T.. Zhao, Y., Hornung, R., Schroff, F., Yang, M., Ross, A., Wang, H., Adam, H., Sirotenko, M., Liu, T., and Gong, B. Videoprism: A foundational visual encoder for video understanding. In International Conference on Machine Learning (ICML), 2024.
Zhao, Y., Hessel, J., Yu, Y., Lu, X., Zellers, R., and Choi, Y. Connecting the dots between audio and text without parallel data through visual knowledge transfer. Association for Computational Linguistics (ACL), 2022.
Zhu, B., Lin, B., Ning, M., Yan, Y., Cui, J., Wang, H., Pang, Y., Jiang, W., Zhang, J., Li, Z., Zhang, W., Li, Z., Liu, W., and Yuan, L. LanguageBind: Extending video-language pretraining to n-modality by language-based semantic alignment. In International Conference on Learning Representations (ICLR), 2024.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction state three concrete contributions and quantita- tively claim up to 9 R@1 improvement. These claims are borne out by the SOTA tables that follow (e.g.. Table T.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
12

---

Answer: [Yes]
Justification: Limitations and possible solutions to them are discussed in the Limitations section of the Appendix.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Definition 5.1 in the Appendix provides all prerequisites and complete algebraic proof for the triangle-area formula, satisfying the theorem- panrem-pluuppee plus-proof requirement.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the elements reproduce the experiments contained in the anonymized repository link in the abstract.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: An anonymized repository is linked in the abstract, and all datasets used are publicly available.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experiment details and settings clearly reported in the Experiments Details section the Appendix. Furthermore, the same settings can be found in the configuration
files of the anonymous code link.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The statistical difference between the results of VAST and TRIANGLE in Table (main table of the paper) statistically significant as t-statistic: 8.73 and p-value: 5.20 x 10-5, Similarly, results in Table 3 are statistically significant at p < 0.05.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The Experiments Details section the Appendix reports the resources specifi- cations.
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttre////oorrppp..cpppblb////e
Answer: [Yes]
13

---

Justification:
The research conducted in this paper confirm the NeurIPS Code of Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There are no direct societal impact of this work.
Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The work releases only similarity loss and code; no scale generative model or high-risk dataset made available.
Justification:
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper cites the original papers that produced the datasets, no license is instead cited as is contained in the original repository.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The anonymized link contains all the explanations for the model proposed.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: There are no human subjects.
15. Institutional review board (TRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable because no human-subject research was conducted.
16. Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: the core method development this research does not involve LLMs
14

---

Appendix
5.1 Area of the TRIANGLE in any dimension
Definition 5.1 (The TRIANGLE area is measure of similarity). The area A of a triangle in IR" given by:
1 v u,,u/v, (u, v)?,
A = 2
(10)
Proof. Prerequisites:
1. (x,y) = y + InHini x, E R
2. ||x|| V(x,x)
3. sin² 0 cos? f =
sin 0 + -000
4. cos Oxy
(x.y) Txilla V(xx)
(x.y)
Let consider a generic triangle xyz with vertex x.y,z R Let us define x y and v = x z as two adiancent triangle side. Let Oue be the angle formed by u and v. Considering the prerequisites, the area of the triangle is defined as follows:
4 = xy
XZ sin ONv
2
Vu, u) v(v.) sin Oue
2
1
V /u, u) (v, v) v c A.o
2
1
(u, u) (v, v) ((u, u) (v, v)) cos Anv
2
1
{u, v)2 uu. (v, v)
(u, u) (v,v) ((u, u) (v. v))
2/
2
1
2 (u, u) (v. v).
5.2 Experiments Details
We perform both pretraining and training from seratch of the TRIANGLE model using 4xA000 GPUs. For pretraining, we sample 150,000 examples from the VAST27M dataset. Each example consists of video (comprising frames and audio track) paired with corresponding caption. The model backbone is adapted from VAST[Chen et al. (2023b), utilizing BERT-B, BEATs((hen et al. (2023c), and EVA-CLIP VIT-G [un et al. (2023) the text, audio, and video encoders, respectively, resulting in a total of 1.3 billion parameters.
Following the approach in Chen et al. (202b)), for the generic i th sample, extract sparsely sampled video frames from it and pass it through the video encoder to obtain the video representation t. The caption text is tokenized and processed by the text encoder to produce text embeddings f, For the audio modality, the audio track is segmented into 10-second clips, zero-padded as necessary, converted into 64-dimensional log Mel filterbank spectrogram using 25ms Hamming window, and encoded via the audio encoder to yield the audio representation a,,
During pretraining phase, use just one single frame and single 10-second audio clip extracted from each sample; In the graining-from-scete and single 10-second audio clip per video.
setting, we use eight randomly selected video frames
15

---

Table 4: Dataset statistics and hyperparameters. Modalities stand for T: text, V: video, A: audio. # F refers to the number of frames used for testing phase. AC refers the number of 10-seconds long audio clips used during testing phase
Benchmark
#Video
#Audio
#F # AC
Train
Test
MSR-VTT
9000
1000
8
DiDeMo
1003
ActivityNect
4917
8
VATEX
431
8
AudioCaps
700
8
VGGSound
5000
8
TRIANGLE is pretrained for 10k steps the 150k-sample subset of VAST27M. Retrieval perfor- mance is evaluated every 100 steps on the MSR-VTT test set, and the checkpoint with the best performance is selected. We employ initial learning rate of lc-4 with linear decay schedule and batch size of 256. In the maining-from-scratte
experiments we train from scratch the aforementioned encoders on the MSRVTT train dataset for 4 epochs with an initial learning rate of le-4 with linear decay schedule and a batch size of 64.
To evaluate the performance of TRIANGLE, follow standard evaluation protocol, randomly selecting eight video frames and one 10-second audio clip from each video to enable fair comparison with existing state-of-the-art models.
utilize several benchmark datasets for our downstream tasks:
MSR-VTT Xu et al. 2016) contains 10,000 video clips paired with 200,000 captions, covering broad range of topics such human activities, sports, and natural landscapes.
DiDeMo Hendricks et al. (2017) includes 10,000 long-form videos from Flickr, each annotated with four temporally ordered natural language descriptions. These captions correspond distinet moments within each video, offering fine-grained alignment between textual and visual content.
ActivityNet Caba Heilbron et al. (2015) comprises 20,000 long-form YouTube videos (averaging 180 seconds duration) and 100,000 captions. It spans 200 human activity classes, from daily routines to complex sports and interactions. Each video is annotated with both activity labels and temporal boundaries, enabling precise localization of actions.
VATEX Wang et al. 2019) consists of 41,250 video clips sourced from the Kinetics-600 dataset ?, accompanied by 825,000 sentence-level descriptions. However, due significant portion of videos becoming unavailable online (e.g., removed or made private), we use curated subset of 14,491 samples.
AudioCaps Kim et al. (2019) consists of 51, 31,000 audio clips, each 10 seconds long. The training set includes caption per clip, while the validation and test provide five captions per clip. We follow the dataset split protocol proposed by for the text-to-audio retrieval task.
VGGSound Chen et al. (2020) is large-scale audio-visual dataset with over 200,000 YouTube video clips, each 10 seconds long and labeled with one 309 audio classes. The dataset covers a wide array of sound events, including human actions. animal sounds, environmental noises, and mechanical events. Due to download limitations, we use a subset of 5,000 samples for testing.
5.3 Visualizing Learning Curves
Figure shows the loss functions for the vanilla experiment, in which it is clear that the proposed TRIANGLE obtains a better convergence with respect to the conventional cosine-based loss and to Symile and GRAM.
Furthermore, we also plot the area value among true matching pairs against the R@ the training from scratch experiment on MSR-VTT. Figure 6 shows the results. As it is evident, the area value minimized during training and, concurrently, the @@1 increases as the area among the true pairs decreases.
16

---

5
Cosine
Symile
GRAM
TRIANGLE
8000
10000
lterations
Figure 5: Vanilla experiment losses.
30
25
1.20
20
1.15
15
1.10
1.05
1.00
100
150
200
100
150
200
Epochs
Epochs
Figure 6: Area value versus R@ I during training from scratch on MSR-VTT dataset.
5.4
Examples Retrieval Outputs
We briefly describe the outcomes from three representative multimodal examples.
Sample VGGSOUND ID: -U7joUcTCo from T Os (i.c., from second 0 of the video)
Ground truth class: People coughing
Results for VAST
Rank of ground truth class: 22nd
TOP 5 Retrieved classes: [people belly laughing, people giggling, people whistling, male speech speaking, people cheering]
Results for TRIANGLE
Rank of ground truth class: 3rd
TOP 5 Retrieved classes: [people giggling, people belly laughing, people coughing, baby laughter, people sobbing]
Sample 2
VGGSOUND ID: -OvPFx wRRI from T = 30s
Ground truth class: People finger snapping
Results for VAST
Rank of ground truth class: 22nd
TOP 5 Retrieved classes: [male singing, child speech kid speaking, child singing, playing castanets, people giggling]
Results for TRIANGLE
Rank of ground truth class: 1st
TOP Retrieved classes: [people finger snapping, people eating apple, male singing, playing castanets, people cating crisps]
17

---

Sample 3
VGGSOUND ID: 2Yo0MGR2kmA from T = 31s
Ground truth class: Playing bagpipes
Results for VAST
Rank of ground truth class: 30th
TOP 5 Retrieved classes: [playing ukulele, playing sitar, playing mandolin, playing steel guitar slide guitar]
Results for TRIANGLE
Rank of ground truth class: 1st
TOP 5 Retrieved classes: [playing bagpipes, playing ukulele, playing banjo, playing sitar, playing steelpan]
From our qualitative analysis, we consistently observe that TRIANGLE effectively integrates infor- mation from all three modalities (visual, audio, and text). This results in superior performance tasks such as video classification and reasoning, where a holistic understanding across modalities is crucial. A particularly illustrative example is Sample 3 from our previous analysis. In the video, a man holding guitar or ukulele appears in the foreground, while another man with a bagpipe in the background. However, upon listening to the audio, it becomes clear that only the bagpipe is being played. TRIANGLE, successfully captures this multimodal context and correctly classifies the video "Playing bagpipes", whereas VAST misclassifies it as "Playing ukulele" by likely over-relying on visual cues.
5.5 Experiments with Different Modalities
We test TRIANGLE on different set of modalities in the Muuch-Vision-Lanuaae Dataset Fu al.) (2024), which comprises in-the-wild vision-touch pairs with language labels. The current state-of-the-ant model, Sethh-iisin----gguee
(TVL), leverages pairwise cosine similarity among all the pairwise, training the tactile encoder to align to the vision and text encoder from CLIP. We retrained TVL (first row in the table) and we train the same models with the proposed TRIANGLE loss (second row in the table). As clear from the scores Tab. 5 TRIANGLE outperforms TVL in the standard two-modal task, better aligning vision and tactile modalities. What is more, by involving our three- modal loss, we can also unlock novel task. Vision to Tactile&Tex.. that allows us to effectively align the three modalities altogether, building joint latent space.
Table 5: Comparison of TVL model Fu et 2024) on the TVL dataset against the proposed TRIANGLE, which outperforms the original TVL method.
Acc@I
Acc@5
Vis. Tact. Acc@1
Tact. Acc@5 Vis. Text Acc@
Vis.-Tact.TTxt Acc@5
TVL
36.7
53.3
79.9
93.1
TRIANGLE
83.1
94.6
82.8
94.6
83.5
94.7
5.6
Ablation Studies
5.6.1
Ablation Study
We conduct ablation studies validate the effectiveness of the TRIANGLE choices. In Fig. 7 report ablation studies made on the regularization weight a of (9). We propose to add regularization term to the area minimization using cosine similarity between the two modalities most relevant the downstream task (i.c., for vidco-text retrieval tasks, the audio may be crucial for the downstream task, but the contribution of video and text are surely more important). The weight contribution of area similarity and cosine similarity. If a O, then only area is considered; if a 1, equal weight is given to area and cosine similarity. As shown in Fig. [7] due to the particular case shown in Fig. (c) that cannot be properly interpreted by the area, it alone may produce suboptimal results. Injecting the cosine similarity regularization performance starts to improve. Interestingly, we discover that this improvement is far more meaningful in the Text to Visual-Audio retrieval task, whereas is not so relevant the visual/audio to text retrieval task. According to these ablations, we set = 1 for the first deck of experiments and 0 for the latter one.
used to balance the
18

---

MSR.VIT Tox Aosil-Aucio
DiDetl Text Vaual-Audio
Act- Not Text to Viuce Mudio
Activlyiit
Figure 7: Ablation studies for weight regularization on sample datasets MSR- VTT, DiDeMo, and ActivityNet.
Table 6: Performance of TRIANGLE trained from scratch on MSR-VTTI under different values of
A
T2AV R@I
T2AV R@IN 10
AV2T R@1
AV2T R@ 10
TRIANGLE
0.0
33.3
74.4
40.4
81.7
TRIANGLE
0.1
39.4
81.8
41.9
80.0
TRIANGLE
0.3
20.9
71.6
20.7
72.4
TRIANGLE
0.5
24.0
74.2
22.2
71.9
TRIANGLE
1.0
38.3
81.2
36.9
78.4
5.6.2 Ablation Study on X
We perform an additional ablation study on the value of À that regulates the influence LOTM in the total loss in (8). In all the experiments conducted in the paper, we set İ 0.1, inhereting this value from previous works (VAST and GRAM). In this ablation, we test different values ranging in (0.0, 0.1, 0.3, 0.5, 1.0). From the result table below, is clear that the setting with A 0.1 obtains the best results in 3 over scores, thus confirming the setting choice of the experiments in the main paper.
5.7 Limitations and Future Work
TRIANGLE is designed to solve three-modal tasks. However, its formulation can be also extended to more modalities. In the case of more than three modalities, such embeddings form a generic polygon. This polygon be divided into smaller triangles for which can straightforwardly compute the area with eq. (TOp in the main paper. The resulting area will be the sum of the subtriangles area. Several ways define the polygon can be potentially explored. Among others, we may leverage the Convex Hull that computes the smallest convex set containing all the points under consideration. We plan to investigate these solutions in future work.
19


Reviews:
[{"review_id": "nzw3bfCjhI", "reviewer": "Reviewer_u6BX", "summary": "This paper proposes TRIANGLE, a novel method for aligning three modalities (e.g., video, audio, text) in multimodal embedding spaces. The authors observe that existing contrastive approaches rely on pairwise cosine similarity with an anchor modality, lacking direct measures of alignment among all modalities. TRIANGLE introduces a **triangle area-based similarity measure** computed in the higher-dimensional embedding space formed by three modality vectors. This geometric area intuitively represents overall alignment (smaller area indicates better alignment) and is used to define a new contrastive loss. TRIANGLE avoids additional fusion layers and anchors, yielding more interpretable alignment rationales. Extensive evaluations on video-text retrieval, audio-text retrieval, and audio-video classification  demonstrate state-of-the-art performance and robust generalization across downstream tasks.", "questions": "You mention potential extensions via convex hull decomposition or sub-triangle summation. Could you provide preliminary results or formal insights on computational complexity and numerical stability in such extensions?\n\nCould you include or release examples of retrieval outputs (queries with top-5 retrieved videos, audios, and captions) to illustrate practical improvements and highlight specific cases where TRIANGLE succeeds over VAST or GRAM?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\nRigorous formulation with a novel, theoretically grounded geometric similarity measure.\n\nExtensive experimental validation across seven datasets and multiple tasks, demonstrating consistent improvements.\nStatistical significance testing is included, strengthening empirical claims.\nWell-structured explanation with intuitive figures illustrating the triangle area concept.\n\nWeaknesses\nResults on training from scratch are promising but still slightly below pretraining-based baselines (e.g. Table 3 MSR-VTT T2AV task).\n\nFigures could include qualitative retrieval examples (e.g., video-audio-text samples with retrieved captions) to improve interpretability for practical deployment.\n\nMinor typos and formatting inconsistencies in equations", "was_revised": true, "final_justification": "I found the majority of the rebuttal are well-justified. All my concerns have been addressed. Thus I raise my score to accept.", "rebuttal": "**On the training from scratch.** Table 3 presents the results of training all the models only on the MSRVTT dataset from scratch (that is, by randomly initializing all model weights), then evaluating on the test set. The purpose of this table is to provide a direct comparison with other methods, isolating the effect of loss functions and removing the influence of other factors such as pretraining data and specific training pipelines, not to achieve the best results overall. In this controlled setting, we demonstrate the superiority of TRIANGLE compared to common multimodal contrastive loss functions.\n\nIn Table 1, the models are pretrained using large datasets (e.g., VAST on 27M of videos, LanguageBind on 10M, and so on), while the training set of MSRVTT used in Table 3 contains only 9k videos. Therefore, the performance gap between Table 1 and 3 is given by the huge difference in dataset size, as it is well-known that pretraining data plays a crucial role in achieving state-of-the-art performance.\n\nOverall, TRIANGLE obtains consistent results even when the training dataset is smaller, proving the effectiveness of the proposed method once again.\n\n**On the extension via convex hull.** We conduct preliminary results on the AV-MNIST dataset adding a fourth modality, that is the Colored MNIST, so to have a total set of 4 modalities (greyscale MNIST images, colored MNIST images, audio MNIST spectrograms, textual labels), processed with four different encoders (details on the encoders in the main paper). We compute the convex-hull-based area loss using fan-triangulation from a fixed anchor (text), avoiding explicit convex hull construction. For $K$ modalities (4 in our case), this requires computing $(K−2)$ triangle areas (2 computations in our case), each with $O(D)$ complexity (where $D$ is the embedding dimension, 3 in our case). Therefore, the per-sample complexity is $O((K−2)·D)$, and the full contrastive matrix costs $O(B^2·(K−2)·D)$, with $B$ batch size, where the TRIANGLE complexity in the case single case of $K=3$ is equal to $O(B^2D)$. We observe no instabilities in this experiment. However, being a preliminary experiment, this solution is still an approximation and should be further developed and tested to handle all the special cases, but we achieved R@1 56.5 and R@5 97.5.\n\nWe include this discussion in the revised paper and we plan to further explore it in future works, thanks for raising this interesting question!\n\n\n**Examples of retrieval outputs.** Thanks to the Reviewer for the valuable suggestion. We agree that incorporating a concrete, real-world example would strengthen the contribution of our work. In response, we conducted an additional inspection on the VGGSound dataset. This dataset consists of multimodal videos, each comprising image frames and corresponding audio samples, with an associated textual class label. VGGSound includes a total of 309 distinct classes and we already report R@1 and R@10 in Tab.2.\n\nHere, we introduce an additional metrics, the Mean Retrieval Rank (MRR), which reflects the average rank position of the ground-truth class among all retrieved classes across the entire test set. The results are as follows:\n\n| Method    | Mean Retrieval Rank (↓) |\n|-----------|--------------------------|\n| VAST      | 27.3                    |\n| GRAM      | 16.7                    |\n| TRIANGLE | **14.8**                 |\n\nThese results demonstrate that TRIANGLE consistently ranks the relevant content more effectively compared to other models.\n\nIn addition, we also manually inspected several retrieved examples. Due to NeurIPS rebuttal policy restrictions, we are currently unable to include links to external multimodal content or images. However, we will include visual examples in the final version of the paper to further illustrate the model performance, and report here only the textual outcomes.\n\nWe briefly describe the outcomes from three representative multimodal examples:\n\n**SAMPLE 1** \nVGGSOUND ID:   $\\text{--U7joUcTCo}$ from T=0s (i.e., from second 0 of the video)  Ground truth class: **'People coughing'**.\n\n\tResults for **TRIANGLE**\n    Rank of ground truth class: **3rd**.\n    TOP 5 Retrieved classes: ['people giggling', 'people belly laughing', **'people coughing'**, 'baby laughter', 'people sobbing']\n\n\tResults for **VAST** \n    Rank of ground truth class: **22nd**\n    TOP 5 Retrieved classes: ['people belly laughing', 'people giggling', 'people whistling', 'male speech man speaking', 'people cheering']\n\n\n**SAMPLE 2** \nVGGSOUND ID:  $\\text{-0vPFx-wRRI}$ from T=30s Ground truth class: **'people finger snapping'**\n\n\tResults for **TRIANGLE**\n    Rank of ground truth class: **1st**\n    TOP 5 Retrieved classes: [**'people finger snapping'**, 'people eating apple', 'male singing', 'playing castanets', 'people eating crisps']\n\n\n\tResults for **VAST** \n    Rank of ground truth class: **22nd**\n    TOP 5 Retrieved classes: ['male singing', 'child speech kid speaking', 'child singing', 'playing castanets', 'people giggling']\n\n\n\n**SAMPLE 3** \nVGGSOUND ID:  $\\text{2Yo0MGR2kmA}$ from T=31s Ground truth class:  **'Playing bagpipes'**\n\n\tResults for **TRIANGLE**\n    Rank of ground truth class: **1st**\n    TOP 5 Retrieved classes: [**'playing bagpipes'**, 'playing ukulele', 'playing banjo', 'playing sitar', 'playing steelpan']\n\n\n\tResults for **VAST**\n    Rank of ground truth class: **30th**\n    TOP 5 Retrieved classes: ['playing ukulele', 'playing sitar', 'playing mandolin', 'playing steel guitar slide guitar']\n\n\nFrom our qualitative analysis, we consistently observe that TRIANGLE effectively integrates information from all three modalities (visual, audio, and text). This results in superior performance on tasks such as video classification and reasoning, where a holistic understanding across modalities is crucial.\n\nA particularly illustrative example is **Sample 3** from our previous analysis. In the video, a man holding a guitar or ukulele appears in the foreground, while another man with a bagpipe is seen in the background. However, upon listening to the audio, it becomes clear that only the bagpipe is being played. TRIANGLE successfully captures this multimodal context and correctly classifies the video as “Playing bagpipes”, whereas VAST misclassifies it as “Playing ukulele” by likely over-relying on visual cues.\n\n**Typos and formatting.** We double-checked the whole paper and revised typos and formatting errors, thank you!"}, {"review_id": "Ffv2aRzlWj", "reviewer": "Reviewer_mfjz", "summary": "The paper introduces a novel method for multimodal learning, named TRIANGLE, which is based on minimizing the area of the triangle formed by the embeddings of three modalities in an n-dimensional space. Unlike traditional approaches that use pairwise cosine similarity and rely on anchor-based alignment, TRIANGLE aligns all three modalities jointly. The triangle area provides a geometrically meaningful indicator of alignment: a smaller area corresponds to closer embeddings and hence better alignment, while a larger area suggests that at least one modality is misaligned. The authors evaluate their method extensively under three different evaluation settings: vanilla experiment, zero-shot retrieval and classification, and training from scratch. The empirical studies are conducted on the alignment problem between video, audio, and text modalities, demonstrating improvements over state-of-the-art methods. The experiments show superior performance across three different setups, with a negligible increase in computational time.", "questions": "1) Would it be possible to test the method on a different set of modalities to support the general claim?\n2) Can you improve the discussion on why TRIANGLE outperforms n-model losses methods?\n3) How do you handle the near-degeneracy triangle case that may be caused by collinearity? Does the method remain stable in such a scenario?", "limitations": "The method is limited by the number of modalities that can be used. However, a solution is proposed in the Limitation section in the appendix.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: The authors propose a novel method to better align three heterogeneous modalities. To overcome the limitations imposed by the cosine similarity, they improve the contrastive learning loss proposed in the state of the art by using the triangle area as a similarity metric, which allows the consideration of three modalities. The mathematical formulation, along with the proofs and possible regularizations, is discussed throughout the paper and in the appendix. The method has been extensively tested on many datasets and three different setups, namely: “vanilla experiment” on a controlled environment (MNIST dataset and variants), “zero-shot retrieval” exploiting pretraining, and “learning from scratch”. In all the proposed experiments, the method shows a significant improvement with respect to the state of the art or, in the worst case, a comparable result (only considering R@1 in the third experiment). Moreover, the method shows a faster convergence in the vanilla experiment with respect to the state of the art.\n\nWeaknesses: The limitations and future work should be discussed in the main paper. Even if the method has been extensively evaluated on many datasets, they always present the same modalities (text, video, and audio), would it be possible to test the method on a different set of modalites to show its performance is not only applicable on that limited set of modalities (In the introduction TRIANGLE is proposed as a generic three modalities method)?  In the paper, the author claims an improvement on the interpretability side. Knowing the importance of that word for the machine learning community, I expect it to be discussed a bit more throughout the paper. To conclude, while TRIANGLE outperforms GRAM and Symile in three-modal tasks, a more in-depth discussion or theoretical comparison on why a tailored triplet objective is inherently superior to general n-modal losses (beyond empirical results) would strengthen the paper.", "was_revised": true, "final_justification": "I raise my score to accept. The questions from the reviewers were properly addressed, and the additional experiments are good. I feel like the interpretability part can be improved, but it's not a big issue.", "rebuttal": "**Experiments with different modalities.** We test TRIANGLE on a different set of modalities in the Touch-Vision-Language Dataset [1], which comprises in-the-wild vision-touch pairs with language labels. The current state-of-the-art model, Touch-Vision-Language (TVL), leverages pairwise cosine similarity among all the pairwise, training the tactile encoder to align it to the vision and text encoder from CLIP. We retrained TVL (first row in the table) and we train the same models with the proposed TRIANGLE loss (second row in the table). As it is clear from the scores, **TRIANGLE outperforms TVL in the standard two-modal task**, better aligning vision and tactile modalities. What is more, by involving our three-modal loss, **we can also unlock a novel task, Vision to Tactile&Text, that allows us to effectively align the three modalities altogether**, building a joint latent space.\n\n|          | Acc@1    | Acc@5    | Vision-Tactile Acc@1 | Vision-Tactile Acc@5 | Vision-Tactile&Text Acc@1 | Vision-Tactile&Text Acc@5 |\n|----------|----------|----------|----------------------|----------------------|---------------------------|---------------------------|\n| TVL      | 36.7     | 53.3     | 79.9                 | 93.1                 | -                   | -                   |\n| TRIANGLE | **83.1** | **94.6** | **82.8**             | **94.6**             | **83.5**                  | **94.7**                  |\n\n[1] Letian Fu, et al. \"A Touch, Vision, and Language Dataset for Multimodal Alignment\", ICML 2024.\n\n**On the interpretability** We appreciate the suggestion for a deeper discussion on the interpretability. Conventional methods that fuses modalities through MLP layers (like VAST) or anchor-based pairwise cosines (like LanguageBind, ImageBind) produces scores whose internal meaning either depends on learned parameters or on different measures of alignment (i.e., for the three modal case, they need to compute the pairwise cosine between each of the modalities, resulting in three different measures, M1-M2, M1-M3, M2-M3), without having a direct and clear information on the **overall alignment of the space**. In our work, instead, we refer to \"interpretability” as **having a direct, human-readable geometric indicator of three-way alignment**, which is the area of the triangle formed by the modality embeddings. Indeed, intuitively, a small area implies that all three modalities point to similar semantics, while a large area signals disagreement among at least two of them. In this sense, the TRIANGLE measure of alignment is more interpretable than fusing strategies or multiple pairwise cosine similarities. We would like to thank the Reviewer once again, we include this discussion in the revised version of the paper.\n\n**On the triplet objective.** We would like to thank the Reviewer for giving us the opportunity to further discuss this important aspect of our work.\n\nLet us assume that we have three modalities, namely text, audio, and vision, although the same analysis can be applied to any other set of modalities.\n\nAll the three methods (i.e., Symile, GRAM, and TRIANGLE) involve a single unimodal encoder for each modality, and the normalized extracted embeddings are $\\textbf{t}$ text, $\\textbf{v}$ vision, and $\\textbf{a}$ audio. We report only $L_{D2T}$; the same holds for $L_{T2D}$. In what follows, $\\langle \\cdot, \\cdot \\rangle$ denotes the dot product between two vectors of the same dimensionality.\n\n**Symile** formulates the loss function to ensure that all its components contribute to maximizing a lower bound on the total correlation among all modalities involved in the training. It relies on the multilinear inner product (MIP), which is not a measure of the geometric representations similarity, as stated in their paper. We found that, in real-world scenarios, Symile performance is weak, likely due to the lack of geometric principles behind their method. Formally, Symile loss can be defined as:\n\n$$L_{D2T\\-Symile} =-\\frac{1}{B}\\sum_{i=1}^{B}\\log\\frac{\\exp(\\langle \\textbf{t}_i, \\textbf{v}_i, \\textbf{a}_i \\rangle /\\tau)}{\\sum^K_j\\exp(\\langle \\textbf{t}_j, \\textbf{v}_i, \\textbf{a}_i \\rangle/\\tau)}$$\n\nWhere $\\langle \\textbf{t}_i, \\textbf{v}_i, \\textbf{a}_i \\rangle$ is the MIP, which is an extension of the classic dot product and is defined as $\\langle \\textbf{t}_i, \\textbf{v}_i, \\textbf{a}_i \\rangle = \\sum^D_d{\\textbf{t}_i}_d \\cdot {\\textbf{v}_i}_d \\cdot {\\textbf{a}_i}_d$, with $D$ is the dimensionality of embeddings.\n\n**GRAM** introduces a geometrical approach for the multimodal pretraining of encoders based on the volume of the k-dimensional parallelotope spanned by the embeddings of the different modalities as\n\n$$L_{D2T\\-GRAM}=-\\frac{1}{B}\\sum_{i=1}^{B}\\log\\frac{\\exp(-Vol(\\textbf{t}_i, \\textbf{v}_i, \\textbf{a}_i)/\\tau)}{\\sum^K_j\\exp(-Vol(\\textbf{t}_j, \\textbf{v}_i, \\textbf{a}_i)/\\tau)}$$\n\nWhere $Vol(\\textbf{t}_i,\\textbf{v}_i,\\textbf{a}_i)= \\sqrt{\\det(\\mathbf{G})}$; $\\mathbf{G}=\\mathbf{Z}^{\\top}\\mathbf{Z}$; $\\mathbf{Z}=(\\mathbf{t}_i,\\mathbf{v}_i,\\mathbf{a}_i)$. We will discuss the details of this method few lines below.\n\n**TRIANGLE** also is rooted on geometrical concepts, however, its formulation is different from GRAM. The area of the triangle built by the embeddings in the latent space is taken into consideration as an alignment metric of the embeddings themself, resulting in:\n\n$$L_{D2T\\-TRIANGLE}=-\\frac{1}{B}\\sum_{i=1}^{B}\\log\\frac{\\exp(-A(\\textbf{t}_i, \\textbf{v}_i, \\textbf{a}_i)/\\tau)}{\\sum^K_j\\exp(-A(\\textbf{t}_j, \\textbf{v}_i, \\textbf{a}_i)/\\tau)}$$\n\nhere $A(\\textbf{t}_i,\\textbf{v}_i,\\textbf{a}_i) = \\frac{1}{2}\\sqrt{\\langle \\mathbf{u}, \\mathbf{u} \\rangle \\langle \\mathbf{v}, \\mathbf{v} \\rangle - \\langle \\mathbf{u}, \\mathbf{v} \\rangle^2}$, in which  $\\mathbf{u} = \\mathbf{t}_i - \\mathbf{a_i}$, $\\mathbf{v} = \\mathbf{t}_i - \\mathbf{v}_i$ are two triangle sides computed among the three embeddings $\\mathbf{t}_i, \\mathbf{v}_i,$ and $\\mathbf{a}_i$ of the three modalities.\nAlthough TRIANGLE can be mistakenly seen as a special case of GRAM when $n=3$, it can be shown that the two objective functions are fundamentally different mathematically and operate on distinct aspects of the geometrical alignment.\n\nGoing into details, by expanding the volume formula and computing all the multiplication needed to compute the determinant of the matrix $\\mathbf{G}$, it can be seen that\n\n$$Vol(\\textbf{t}_i,\\textbf{v}_i,\\textbf{a}_i)= \\sqrt{1 - \\textbf{v}_i\\textbf{a}_i^2 - \\textbf{t}_i\\textbf{a}_i^2 - \\textbf{t}_i\\textbf{v}_i^2 + 2 \\cdot \\textbf{t}_i\\textbf{a}_i \\cdot \\textbf{a}_i\\textbf{v}_i \\cdot \\textbf{v}_i\\textbf{t}_i}$$\n\nWhereas the area of the triangle is: \n\n$$A(\\textbf{t}_i,\\textbf{v}_i,\\textbf{a}_i) = \\frac{1}{2}\\sqrt{\\langle (\\mathbf{t}_i - \\mathbf{a_i}), (\\mathbf{t}_i - \\mathbf{a_i})\\rangle \\langle (\\mathbf{t}_i - \\mathbf{v}_i), (\\mathbf{t}_i - \\mathbf{v}_i) \\rangle - \\langle (\\mathbf{t}_i - \\mathbf{a_i}), (\\mathbf{t}_i - \\mathbf{v}_i) \\rangle^2}$$\n\nwhich is significantly different.\n\nGeometrically, the GRAM equation ($Vol$) is the volume of the parallelotope spanned by the three vectors with one vertex at the origin, measuring how mutually linearly independent the three origin-anchored vectors are. On the contrary, the TRIANGLE area ($A$) formula measures how far apart the  difference vectors are, i.e., the edges of the triangle with vertices at $\\mathbf{t}, \\mathbf{a}, \\mathbf{v}$, not comprising the origin. Therefore, the two losses optimize different geometrical entities and rely on diverse formulations.\nThese highlighted differences, when incorporated into contrastive learning loss functions, influence different aspects of the vector alignment, resulting in different downstream performance, favoring the three-modal-specialized TRIANGLE in all the comparisons.\nWe have updated the revised paper including this discussion, following the valuable suggestion of the Reviewer.\n\n**On the limitations in main paper** We can definitely move the limitations in the main body and organize the last page accordingly. Thanks for the suggestion."}, {"review_id": "MZIJjvV4pq", "reviewer": "Reviewer_tnky", "summary": "This paper introduces TRIANGLE, a novel multimodal alignment approach that simultaneously aligns three modalities by minimizing the triangle area formed by their latent representations with a cosine regularization. The extensive experiments across multiple benchmarks demonstrate TRIANGLE's superiority compared to existing baselines on the downstream tasks (retrieval and classification).", "questions": "Please ref to **Weaknesses**", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths:** \n\n- **Clear and Well-Structured:** The paper is well-organized, with detailed explanations of the preliminary, intuition, and methodology.\n\n- **Interesting and Novel:** This paper presents a geometrically intuitive solution for multimodal alignment that directly aligns embedding vectors from different modalities by minimizing the area of triangles formed by these vectors. The approach leverages the geometric property that smaller triangle areas correspond to better alignment between embedding vectors, providing an intuitive and interpretable framework for multimodal alignment.\n\n- **Extensive Experiments:** The authors conduct extensive experiments, and TRIANGLE achieves state-of-the-art performance compared to a wide range of baselines, showing significant improvements across multiple datasets. These results demonstrate the effectiveness of the TRIANGLE approach.\n\n### **Weaknesses:** \n\n- **Lack Theoretical Analysis:** While the proposed method is intuitive and demonstrates effectiveness, its theoretical foundation requires further clarification. Specifically, the connection between minimizing triangle area and achieving better alignment lacks sufficient justification—why does minimizing the triangle area lead to better alignment? I acknowledge that a formal proof may be beyond the scope of this paper; however, I suggest the authors provide a comparative analysis of how the TRIANGLE objective affects the cosine similarity of aligned latent representations relative to previous baselines to substantiate the geometric intuition. Additionally, including visualizations of the latent space transformations and empirical analysis of how triangle area reduction correlates with alignment performance would strengthen the theoretical grounding of the submission.\n\n- **Unclear Problem Definition and Annotation:** Some definitions and annotations in Section 3 are introduced without clear explanation or prior clarification. I recommend that the authors refine this section to improve clarity and readability.\n\n- **Insufficient Ablation Study:** While the paper provides an ablation study on key methodological components, investigating the effects of the $L_{DTM}$ loss function and the weight regularization parameter $\\alpha$, a more comprehensive analysis would strengthen the submission. Specifically, the authors only examine the significance of the $L_{DTM}$ loss function, but it remains unclear whether assigning different weights $\\lambda$ could lead to better performance.\n\n- **Formatting:** Please use` \\citep{}` for parenthetical citations instead of `\\citet{}` throughout the manuscript.", "was_revised": true, "final_justification": "The authors largely addressed my concerns. I decide to update my score to 5.", "rebuttal": "**Theoretical Analysis.** Thanks for giving us the opportunity to better clarify the link between area minimization and better alignment. The intuition is provided in Figure 3 (b) in the paper, where we show that as the area gets smaller, the three vectors collides. Formally, since the area is defined as:\n\n$$    A = \\frac{1}{2} \\sqrt{\\langle \\mathbf{u}, \\mathbf{u} \\rangle \\langle \\mathbf{v}, \\mathbf{v} \\rangle - \\langle \\mathbf{u}, \\mathbf{v} \\rangle^2},$$\n\nin which $\\mathbf{u} = \\mathbf{x} - \\mathbf{y}$, $\\mathbf{v} = \\mathbf{x} - \\mathbf{z}$, we obtain $A=0$ if $(\\mathbf{x} - \\mathbf{y})$ and  $(\\mathbf{x} - \\mathbf{z})$ are linearly dependent, that is when the three embedding vectors coincides having cross-cosine similarities equal to 1, thus representing perfect alignment.\n\nTo prove our theoretical claims, we conduct additional experiments to show how minimizing the triangle area brings improvements in the alignment of the latent space. Specifically, we test TRIANGLE in the Touch-Vision-Language Dataset [1], which comprises in-the-wild vision-touch pairs with language labels. The current state-of-the-art model, Touch-Vision-Language (TVL), leverages pairwise cosine similarity among all the pairwise, training the tactile encoder to align it to the vision and text encoder from CLIP. We retrained TVL (first row in the table) and we train the same models with the proposed TRIANGLE loss (second row in the table). As it is clear from the scores, **TRIANGLE improves the alignment of the pairwise comparison too**, better aligning vision and tactile modalities. What is more, by involving our three-modal loss, **we can also unlock a novel task, Vision to Tactile&Text, that allows us to effectively align the three modalities altogether**, building a joint latent space. This empirically demonstrates that minimizing the triangle area also improves the cosine similarity alignment between pairs of modalities.\n\n|          | Acc@1    | Acc@5    | Vision-Tactile Acc@1 | Vision-Tactile Acc@5 | Vision-Tactile&Text Acc@1 | Vision-Tactile&Text Acc@5 |\n|----------|----------|----------|----------------------|----------------------|---------------------------|---------------------------|\n| TVL      | 36.7     | 53.3     | 79.9                 | 93.1                 | -                   | -                   |\n| TRIANGLE | **83.1** | **94.6** | **82.8**             | **94.6**             | **83.5**                  | **94.7**                  |\n\n[1] Letian Fu, et al. \"A Touch, Vision, and Language Dataset for Multimodal Alignment\", ICML 2024.\n\n**Unclear definitions.** We have revised and updated the notations and the definitions in Section 3, thank you for pointing this out!\n\n**Ablation studies.** According to the Reviewer's suggestion, we perform an additional ablation study on the value of $\\lambda$ that regulates the influence of $L_{DTM}$ in the total loss in Eq 8. \nIn all the experiments conducted in the paper, we set $\\lambda=0.1$, inhereting this value from previous works (VAST and GRAM). In this ablation, we test different values ranging in (0.0, 0.1, 0.3, 0.5, 1.0).\nFrom the result table below, it is clear that the setting with $\\lambda=0.1$ obtains the best results in 3 over 4 scores, thus confirming the setting choice of the experiments in the main paper. \n\n| Training from scratch on MSRVTT | $\\lambda$         | T2AV R@1 | T2AV R@10 | AV2T R@1 | AV2T R@10 |\n|-----------------|---------------|----------|-----------|----------|-----------|\n| TRIANGLE |   0.0    |  33.3     | 74.4      | 40.4     | **81.7**  |\n| TRIANGLE |   0.1    | **39.4** | **81.8**  | **41.9** | 80.0 |\n| TRIANGLE |   0.3    |  20.9   | 71.6     | 20.7     |   72.4    |\n| TRIANGLE |   0.5    |  24.0   |  74.2    |   22.2   |  71.9     |\n| TRIANGLE |   1.0    |  38.3   |   81.2   |  36.9    |  78.4     |\n\n\nWe added this ablation in the revised manuscript, thanks again for the suggestion!\n\n**Formatting** Thanks for pointing this out! We have updated the citation formatting according to Reviewer's suggestion."}, {"review_id": "j7QusJKZEo", "reviewer": "Reviewer_bcfQ", "summary": "This work suggests a new loss function to align modalities in self-supervised multimodal models that take three modalities as input. The work begins by noting that standard cosine similarity, which only enables pairwise comparisons fails to directly optimize for the $n$-modality setting where $n > 2$. To address this, the paper proposes a new version of contrastive loss for the $n =3$ setting, TRIANGLE, that replaces calculation of cosine similarity with the area of the triangle generated by the three representations (one from each modality). The paper compares this function to existing approaches (both those that use an anchor modality and also broader generalizations that use notions of volume), noting that it adds limited computational overhead. Finally, the paper runs a range of experiments both in toy settings (multimodal MNIST) and in various more realistic retrieval tasks. Overall, TRIANGLE outperforms comparable approaches in many cases.", "questions": "- Line 338: The paper finds that the DTM score is essential. Can more be said about why this is?\n- Have the authors considered trying to extend TRIANGLE to more than 3 modalities? \n- Why isn’t (5) symmetric in its indices?", "limitations": "The paper does not address limitations. Limitations to address might include the fact that this only works for exactly three modalities. Another limitation is that this method is mostly evaluated in terms of retrieval.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- **The method is simple (in a good way) and makes intuitive sense:** TRIANGLE is intuitive, leveraging basic geometric concepts like area that hold even in high-dimensional spaces. Adding to its appeal, it would also be easy to integrate it into existing multimodal training or finetuning frameworks. \n- **The paper is clearly written and easy to read:** Even though this reviewer does not work in multimodal contrastive learning, they found this paper easy to read. The prose is clear and the main points and analysis are made concisely. \n- **At least for the tasks on which it is evaluated, TRIANGLE demonstrates strong performance:** In most cases, TRIANGLE achieves high performance relative to competing methods (but see some questions about this in the ‘Questions’ section below).\n\n\nWeaknesses\n- **The value of TRIANGLE is a bit more nuanced than what is conveyed in the paper and its relationships to similar methods are underemphasized:** One of the main points of the paper is that TRIANGLE overcomes a basic obstacle that many other approaches to multimodal contrastive learning have to grapple with: how to align more than two modalities via only pairwise comparisons. The paper does this by generalizing cosine similarity to the area of the triangle formed by the representation of the three modalities. The benefit of this approach is further emphasized in Sections 3.3 and 3.4 relative to anchor modality-based methods.  On the other hand, the paper also notes that there are at least two existing works that already address this through various generalizations of cosine similarity (e.g., parallelepiped volume, etc.). The reviewer would have liked to see a methodological comparison against these as TRIANGLE is a variation on at least GRAM. For instance, to this reviewer’s understanding, there should be nice comparisons of TRIANGLE to GRAM in the $n = 3$ setting. \n- **The value of the MNIST experiments is unclear:** Small-scale toy experiments can be useful for exploring aspects of a problem that are challenging to run or analyze at scale. However, unless the reviewer missed something, the kinds of questions asked in Section 4.1 were more or less the same ones asked for the larger-scale experiments. The reviewer suggests either utilizing this section to ask questions appropriate for small scale settings or removing it.\n- **Some of the claims about alignment appear wrong:** The paper claims that “this [anchor-based] alignment only ensures that all modalities are aligned with the anchor, without providing any indicators for the alignment between non-anchor modalities”. To this reviewer’s understanding, this is not true. Indeed, it is quite easy to see that when the pairwise comparison method is based on a true metric so that the triangle inequality holds, then when $d(a,b)$ and $d(a,c)$ are small, so is $d(b,c)$. The authors should correct me if I have misunderstood the claim here.\n\nNitpicks\n- There are multiple places in the paper where ‘receipt’ should be ‘recipe’ (e.g., Line 28).\n- The abstract is stilted and could use some polishing.\n- Line 23: The notation here is confusing. What is $\\mathbf{M}_i$? Given that $e_i: M_i \\rightarrow \\mathbf{M}_i$ one would assume that $\\mathbf{M}_i$ is a set but then it is stated that $\\mathbf{M}_i \\in \\mathbb{R}^n$, which doesn’t make sense.", "was_revised": true, "final_justification": "The authors addressed most of my concerns during the rebuttal phase including\n- Clearing up that the statement about anchors and alignment was primarily empirical in nature\n- More clearly contrasting TRIANGLE to other similar approaches\n- And adding another experiment.\nOverall, I think that this is an interesting paper that is much improved by the additions made during revision.", "rebuttal": "**On the comparison with GRAM.** We would like to thank the Reviewer for raising this interesting question. Let us focus on the three-modal case $n=3$ with text, audio, and vision, although the same analysis can be applied to any other set of modalities.\n\nEach method involves a unimodal encoder for each modality, and the normalized extracted embeddings are $\\textbf{t}$ text, $\\textbf{v}$ vision, and $\\textbf{a}$ audio. We report only L_D2T; the same holds for L_T2D. In what follows, $\\langle \\cdot, \\cdot \\rangle$ denotes the dot product between two vectors of the same dimensionality.\n\n**Symile** formulates the loss function to ensure that all its components contribute to maximizing a lower bound on the total correlation among all modalities involved in the training. It relies on the multilinear inner product (MIP), which is not a measure of the geometric representations similarity, as stated in their paper. We found that, in real-world scenarios, Symile performance is weak, likely due to the lack of geometric principles behind their method. Formally, Symile loss can be defined as:\n\n$$L_{D2T\\-Symile} =-\\frac{1}{B}\\sum_{i=1}^{B}\\log\\frac{\\exp(\\langle \\textbf{t}_i, \\textbf{v}_i, \\textbf{a}_i \\rangle /\\tau)}{\\sum^K_j\\exp(\\langle \\textbf{t}_j, \\textbf{v}_i, \\textbf{a}_i \\rangle/\\tau)}$$\n\nWhere $\\langle \\textbf{t}_i, \\textbf{v}_i, \\textbf{a}_i \\rangle$ is the MIP, which is an extension of the classic dot product and is defined as $\\langle \\textbf{t}_i, \\textbf{v}_i, \\textbf{a}_i \\rangle = \\sum^D_d{\\textbf{t}_i}_d \\cdot {\\textbf{v}_i}_d \\cdot {\\textbf{a}_i}_d$, with $D$ is the dimensionality of embeddings.\n\n**GRAM** introduces a geometrical approach for the multimodal pretraining of encoders based on the volume of the k-dimensional parallelotope spanned by the embeddings of the different modalities as\n\n$$L_{D2T\\-GRAM}=-\\frac{1}{B}\\sum_{i=1}^{B}\\log\\frac{\\exp(-Vol(\\textbf{t}_i, \\textbf{v}_i, \\textbf{a}_i)/\\tau)}{\\sum^K_j\\exp(-Vol(\\textbf{t}_j, \\textbf{v}_i, \\textbf{a}_i)/\\tau)}$$\n\nWhere $Vol(\\textbf{t}_i,\\textbf{v}_i,\\textbf{a}_i)= \\sqrt{\\det(\\mathbf{G})}$; $\\mathbf{G}=\\mathbf{Z}^{\\top}\\mathbf{Z}$; $\\mathbf{Z}=(\\mathbf{t}_i,\\mathbf{v}_i,\\mathbf{a}_i)$. We will discuss the details of this method few lines below.\n\n**TRIANGLE** also is rooted on geometrical concepts, however, its formulation is different from GRAM. The area of the triangle built by the embeddings in the latent space is taken into consideration as an alignment metric of the embeddings themself, resulting in:\n\n$$L_{D2T\\-TRIANGLE}=-\\frac{1}{B}\\sum_{i=1}^{B}\\log\\frac{\\exp(-A(\\textbf{t}_i, \\textbf{v}_i, \\textbf{a}_i)/\\tau)}{\\sum^K_j\\exp(-A(\\textbf{t}_j, \\textbf{v}_i, \\textbf{a}_i)/\\tau)}$$\n\nhere $A(\\textbf{t}_i,\\textbf{v}_i,\\textbf{a}_i) = \\frac{1}{2}\\sqrt{\\langle \\mathbf{u}, \\mathbf{u} \\rangle \\langle \\mathbf{v}, \\mathbf{v} \\rangle - \\langle \\mathbf{u}, \\mathbf{v} \\rangle^2}$, in which  $\\mathbf{u} = \\mathbf{t}_i - \\mathbf{a_i}$, $\\mathbf{v} = \\mathbf{t}_i - \\mathbf{v}_i$ are two triangle sides computed among the three embeddings $\\mathbf{t}_i, \\mathbf{v}_i,$ and $\\mathbf{a}_i$ of the three modalities.\nAlthough TRIANGLE can be mistakenly seen as a special case of GRAM when $n=3$, it can be shown that the two objective functions are fundamentally different mathematically and operate on distinct aspects of the geometrical alignment.\n\nGoing into details, by expanding the volume formula and computing all the multiplication needed to compute the determinant of the matrix $\\mathbf{G}$, it can be seen that\n\n$$Vol(\\textbf{t}_i,\\textbf{v}_i,\\textbf{a}_i)= \\sqrt{1 - \\textbf{v}_i\\textbf{a}_i^2 - \\textbf{t}_i\\textbf{a}_i^2 - \\textbf{t}_i\\textbf{v}_i^2 + 2 \\cdot \\textbf{t}_i\\textbf{a}_i \\cdot \\textbf{a}_i\\textbf{v}_i \\cdot \\textbf{v}_i\\textbf{t}_i}$$\n\nWhereas the area of the triangle is: \n\n$$A(\\textbf{t}_i,\\textbf{v}_i,\\textbf{a}_i) = \\frac{1}{2}\\sqrt{\\langle (\\mathbf{t}_i - \\mathbf{a_i}), (\\mathbf{t}_i - \\mathbf{a_i})\\rangle \\langle (\\mathbf{t}_i - \\mathbf{v}_i), (\\mathbf{t}_i - \\mathbf{v}_i) \\rangle - \\langle (\\mathbf{t}_i - \\mathbf{a_i}), (\\mathbf{t}_i - \\mathbf{v}_i) \\rangle^2}$$\n\nwhich is significantly different.\n\nGeometrically, the GRAM equation ($Vol$) is the volume of the parallelotope spanned by the three vectors with one vertex at the origin, measuring how mutually linearly independent the three origin-anchored vectors are. On the contrary, the TRIANGLE area ($A$) formula measures how far apart the  difference vectors are, i.e., the edges of the triangle with vertices at $\\mathbf{t}, \\mathbf{a}, \\mathbf{v}$, not comprising the origin. Therefore, the two losses optimize different geometrical entities and rely on diverse formulations.\nThese highlighted differences, when incorporated into contrastive learning loss functions, influence different aspects of the vector alignment. \n\nExperimental results demonstrate a clear improvement when using the TRIANGLE loss function compared to the GRAM loss function in the case of $n=3$ (results are reported in Tables 1, 2, and 3 of the paper). We highlight that all reported results for GRAM were obtained using the three-modal configuration ($n=3$) with T,V,A modalities like TRIANGLE to ensure a fair comparison.\n\nWe are pleased to include this methodological analysis in the supplementary material of our work.\n\n**On the vanilla experiment.** We would like to thank the Reviewer for the suggestion. We moved the vanilla experiment on MNIST to the appendix and we inserted the experiment with different modalities (see response to Reviewer mfjz) in the main body, to further prove the robustness of TRIANGLE in different settings. Thanks for the suggestion!\n\n**On the pairwise alignment claims.** Thanks for giving us the opportunity to clarify this point. While there exists cases in which aligning the embeddings to an anchor yield the alignment of also non-anchor embeddings, there also exist cases in which this may not be true.\nLet us suppose to have three vectors $a,b,c$, where $a$ is the anchor, each with coordinates $a=(-1, 1, 0)$, $b=(-1,0,0)$, $d=(0,1,0)$. Then, $\\theta=30°$ is the angle between $(a,b)$, which is quite tight, as $\\cos(\\theta)=0.87$ and reasonable to assume in a real-world scenarios. For simplicity, suppose also that the angle between $(a,c)$ is equal to $\\theta$ as well. Therefore, both $b$ and $c$ are quite aligned with the anchor $a$. However, the angle between $b$ and $c$ themselves is equal to $2\\theta$, and $\\cos(2\\theta)=0.5$, which instead is quite lower than the measure of alignment between embeddings and the anchor modality.\n\nIneed, our claim is not that pairwise-to-anchor alignment cannot imply anything about the remaining pairs at all, but that it does not provide a direct, explicit indicator of that alignment, as the alignment of non-anchor embeddings is not explicit or measured in such methods and there may exist cases in which non-anchor pairs are indeed less aligned.\n\n**Typos and abstract** Thanks for pointing this out! We have revised them!\n\n**Notation** Thanks for pointing this out and sorry for this misleading notation, we have revised it. \nThe role of encoders as a function is to transform data from their input domanin $\\textit{M}_i$ into the representation space $\\mathcal{Z}$, shared among all the modalities. Here, $\\mathcal{Z}$ is a subset of $\\mathbb{R}^n$ and so every vector $\\mathbf{z}$ of $\\mathcal{Z}$ is also a vector of $\\mathbb{R}^n$.\nIn conclusion $e_i: \\textit{M}_i \\rightarrow \\mathcal{Z}$, $\\mathcal{Z} \\subseteq \\mathbb{R}^n$. \n\n\n**Q1 On the role of DTM.** Following previous works such as VAST, to further fuse the multimodal embeddings, we use a multimodal encoder to rerank the pairs, outputing a score between 0 and 1 that is the probability of the three input embeddings have matching semantics or not. As explained in Sec 3.3, following a similar approach to VAST, we reuse the text encoder activating the cross attention layers to accomodate video and audio features to be a conditioning signal. This module is further trained using the loss L_DTM. \nWe have added this clarification in the revised paper.\n\n**Q2 On extension to more than 3 modalities.** In Sec 5.5 Limitations and Future Work, we discuss potential ideas for extending the triangle area concept to the concept of area of a generic polygon in the hyperdimensional space that allows the extension of TRIANGLE to more than 3 modalities. Furthermore, we conduct preliminary experiments on the AV-MNIST dataset adding a fourth modality, that is the Colored MNIST, so to have a total set of 4 modalities, but the method can be generalized to more modalities. For details, please see Response to Reviewer u6BX on the Convex hull.\n\n**Q3 On the Symmetry of L_D2T** Our contrastive loss function 1/2(L_D2T + L_T2D) is rooted on many previous works relying on the InfoNCE loss function introduced in [1]. The InfoNCE is not symmetric: it optimizes only one directional alignment at a time, ignoring the opposite modality direction [2]. As a consequence, contrastive losses explicitly imply a second term (i.e. the reverse direction) to enforce symmetry and ensure consistent bidirectional alignment [3, LanguageBind, ImageBind, GRAM, VAST].\nMuch more intuitivelly, with Eq 5 L_D2T many different text captions are contrasted with a single video-audio pair. For the video-audio pair that match with the correct textual description the corresponding area is reduced, for all the other triplets the area is enhanced.\nOn the other hand, with Eq 6 L_T2D many different video-audio pairs are contrasted with a single textual description.\n\n[1] A. van den Oord, et al. \"Representation learning with contrastive predictive coding\" arXiv:1807.03748, 2018.\n\n[2] Lin, Zudi, et al. \"Relaxing contrastiveness in multimodal representation learning\" IEEE/CVF WACV. 2023.\n\n[3] A. Radford,  et al. \"Learning transferable visual models from natural language supervision\" ICML 2021."}]

Author rebuttal:

Meta-review: This work addresses the problem of aligning multiple (>= 3) modalities in multimodal learning. It argues that prior work in multimodal learning have severe limitations as they rely on cosine similarity between modalities and select an anchor modality to which other modalities are aligned one-by-one. It claims that due to such an approach, such methods might not sufficiently well align non-anchor modalities. Along the lines of recent works (GRAM, Symile), the current work proposes an alternative similarity metric for multimodal learning that scales to arbitrary number of modalities. The key idea is to consider the area of the triangle formed by the embeddings of the three modalities as a measure of their dissimilarity and align the modalities jointly by minimizing the area. It is argued that the area of the triangle provides a geometrically meaningful indicator of alignment with smaller areas indicating better alignment. As area can be small also when the modalities are not closely aligned (e.g. flat triangles), the work employs cosine regularization of the two most important modalities for the downstream task. The loss function has two components - first is a contrastive loss based on triangle area and the second is a data text loss. The proposed method is evaluated in under different evaluation settings - vanilla, zero-shot retrieval and classification, and training from scratch. Experiments consider text, video and audio as the three modalities for alignment. In the rebuttal phase, a fourth modality, touch, was also included. Results of the experimental evaluation are very promising and demonstrates significant empirical gains over several baselines including GRAM and Symile. 


Strengths:

Novelty

mfjz: "authors propose a novel method to better align three heterogeneous modalities"

u6BX: a novel, theoretically grounded geometric similarity measure

Tnky: "presents a geometrically intuitive solution for multimodal alignment"

bcfQ : "TRIANGLE is intuitive, leveraging basic geometric concepts like area"

Encouraging results

u6BX: "Extensive experimental validation across seven datasets and multiple tasks, demonstrating consistent improvements."

mfjz: "The method has been extensively tested on many datasets and three different setups.. the method shows a significant improvement with respect to the state of the art or, in the worst case, a comparable result"

mfjz: "the method shows a faster convergence in the vanilla experiment with respect to the state of the art"

Tnky: "achieves state-of-the-art performance compared to a wide range of baselines, showing significant improvements across multiple datasets."

bcfQ:  "for the tasks on which it is evaluated, TRIANGLE demonstrates strong performance"


Weaknesses:

Concern about results for training from scratch

u6BX: "slightly below pretraining-based baselines"

Concern about complexity and stability of extension to K > 3 modalities.

Concern about effectiveness to modalities other than text, audio and video

mfjz: "test the method on a different set of modalites to show its performance is not only applicable on that limited set of modalities"

Concern about some claims made but unsubstantiated

mfjz: "author claims an improvement on the interpretability side. Knowing the importance of that word for the machine learning community, I expect it to be discussed a bit more throughout the paper"

bcfQ: "The paper claims that “this [anchor-based] alignment only ensures that all modalities are aligned with the anchor, without providing any indicators for the alignment between non-anchor modalities”. To this reviewer’s understanding, this is not true."


Concerns about ablation

Tnky: "investigating the effects of the  loss function and the weight regularization parameter , a more comprehensive analysis would strengthen the submission."

Lack of theoretical analysis of the proposed approach

 mfjz: "a more in-depth discussion or theoretical comparison on why a tailored triplet objective is inherently superior to general n-modal losses"

Tnky: "the connection between minimizing triangle area and achieving better alignment lacks sufficient justification"

Tnky: "provide a comparative analysis of how the TRIANGLE objective affects the cosine similarity of aligned latent representations relative to previous baselines"

Concerns about the soundness of the main idea

mfjz: "How do you handle the near-degeneracy triangle case that may be caused by collinearity? Does the method remain stable in such a scenario?"

Lack of qualitative analysis

u6BX: "extensions via convex hull decomposition or sub-triangle summation. Could you provide preliminary results or formal insights on computational complexity and numerical stability in such extensions?"


Final Comments:
This work addresses an important problem and provides a principled solution that is simple to implement and doesn't introduce significant computational overheads. The main strength of the work is its empirical gains over prior work on the datasets used in the experimental study. Claims on interpretability are not well-substantiated, theoretical understanding of how the proposed area-based loss is superior to prior art is lacking, and qualitative analysis of the results is missing. However, reviewers are positive about the novelty, methodology and experimental evaluation. During the rebuttal phase, authors and reviewers had a productive discussion resolving most of the concerns expressed in the reviews.
Decision: Accept (poster)