Gramian Multimodal Representation Learning and Alignment
Authors: Giordano Cicchetti
Eleonora Grassucci
Luigi Sigillo
Danilo Comminiello
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Representation Learning
Multimodal Alignment
Multimodal Contrastive Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: GRAM is a novel measure for multimodal representation learning supported by strong mathematical guarantees that, in a contrastive learning fashion, truly aligns multiple modalities in a shared higher-dimensional latent space.

Abstract: Human perception integrates multiple modalities—such as vision, hearing, and language—into a unified understanding of the surrounding reality. While recent multimodal models have achieved significant progress by aligning pairs of modalities via contrastive learning, their solutions are unsuitable when scaling to multiple modalities. These models typically align each modality to a designated anchor without ensuring the alignment of all modalities with each other, leading to suboptimal performance in tasks requiring a joint understanding of multiple modalities. In this paper, we structurally rethink the pairwise conventional approach to multimodal learning and we present the novel Gramian Representation Alignment Measure (GRAM), which overcomes the above-mentioned limitations. GRAM learns and then aligns $n$ modalities directly in the higher-dimensional space in which modality embeddings lie by minimizing the Gramian volume of the $k$-dimensional parallelotope spanned by the modality vectors, ensuring the geometric alignment of all modalities simultaneously. GRAM can replace cosine similarity in any downstream method, holding for 2 to $n$ modalities and providing more meaningful alignment with respect to previous similarity measures. The novel GRAM-based contrastive loss function enhances the alignment of multimodal models in the higher-dimensional embedding space, leading to new state-of-the-art performance in downstream tasks such as video-audio-text retrieval and audio-video classification.

Published as a conference paper at ICLR 2025
GRAMIAN MULTIMODAL REPRESENTATION LEARNING AND ALIGNMENT
Giordano Cicchetti', Eleonora Grassucci", Luigi Sigillo, Danilo Comminiello Dept. of Information Engineering, Electronics, and Telecomm., Sapienza University of Rome, Italy (name. surname) @unatee)uuntuu @nn @uniromal romal. it
ABSTRACT
Human perception integrates multiple modalities-such as vision, hearing, and language-into a unified understanding of the surrounding reality. While recent multimodal models have achieved sigmificant progress by aligning pairs of modal- ities via contrastive learning, their solutions are unsuitable when scaling to mul- tiple modalities. These models typically align each modality to designated an- chor without ensuring the alignment of all modalities with each other, leading to suboptimal performance in tasks requiring a joint understanding of multiple modalities. In this paper, we structurally rethink the pairwise conventional ap- proach to multimodal learning and we present the novel Gramian Representation Alignment Measure (GRAM), which overcomes the above- mentioned limitations. GRAM learns and then aligns n modalities directly in the higher- lighe--dimensional space in which modality embeddings lie by minimizing the Gramian volume of the k-dimensional parallelotope spanned by the modality vectors, ensuring the ge- ometric alignment of all modalities simultaneously. GRAM can replace cosine similarity in any downstream method, holding for 2 to n modalities and provid- ing more meaningful alignment with respect to previous similarity measures. The novel CRA.M-assed contrastive loss function enhances the alignment of multi- modal models the gigher-dimensionll embedding space, leading to new state- of-the-art performance in downstream tasks such as video-audio-text retrieval and audio-video classification. The project page, the code and the pretrained models are available at
github. 10/GRAM/
INTRODUCTION
Humans naturally process and integrate signals from multiple sensory modalities, such as sounds and visual inputs, to form coherent understanding of the world around them. Inspired by this, foundational models have attempted to replicate this capability by aligning pairs of modalities, such as vision and language, through contrastive learning techniques. One of the most significant con- tributions in this domain was CLIP Radford et al. (2021), which used a contrastive loss to align image and text representations based cosine similarity. CLIP has shaped the current approach to multimodal learning, and every subsequent model relies on the same contrastive-pairs fashion, even in the case of models involving more than modalities, such as Imageliin Girdhar Girdhar et al. (202)).
VAST (Chn et al. (2023c), and LanguageBind Zhu et al. (2024).
However, these models suffer from critical limitations, as they typically involve cosine similarity to align each modality to a chosen anchor modality (e.g., images, text, or audio) Sirnam et al. (2023):
Girdhar al. (2023), without ensuring consistent alignment between non-anee con-anchor modalities. This approach can be insufficient for tasks that require moosslmodal understanding beyond pairs. For instance, aligning video and audio separately to common textual anchor does not guarantee that the video and audio themselves are well-aligned. Consequently, models often fail in complex, real- world scenarios where interactions between multiple modalities are crucial. notable example video-text retrieval, where audio can be essential for correctly interpreting the scene. State-of-the- art (SOTA) models often neglect this, leading to suboptimal performance when audio is a critical factor Yoon et al. (2023). as show in Fig. of Appendix B Due to these inherent limitations,
'These authors contributed equally this work.

---

Published as a conference paper at ICLR 2025
eca t00
Mredolles Modality
Idlodaall ty:3
Modallty
Mulaiay
"AA black dog the forggrann barkiing in ressrberant emironsosos"
"A wontan playing the gattar ist large volamee in a dooedd spact"
My daltyy
Modallty
Semantically aligned data y> small volume
Semantically misaligned data large volume
Figure 1: Visualization of the GRAM intuition: on the left, embedding vectors from semantically aligned multimodal data build a parallelotope with small volume. On the right, where modalities are not aligned with each other, the formed parallelotope has a large volume.
the current trajectory of multimodal learning is reaching plateau, with the sole possible research direction of scaling up models and datasets bringing negligible improvements.
This paper presents a novel alignment measure that fundamentally rethinks the alignment process of multimodal models, by directly operating in the nigher-dimeenionall space spanned by the modality embeddings. The proposed measure can replace the cosine similarity in any downstream method providing more meaningful information about the embedding semantics, especially in the case of multiple modalities. Together with the origin of the axis, such & embeddings the edges of k-dimensional parallelotope in R", whose volume is strictly related to the alignment of the modalities as Fig. ssoows. Indeed, the lower the volume, the closer the modality vectors are, meaning they well-aligned. The Gramian Representation Alignment Measure (GRAM) exploits this intuition to provide meaningful information on the semantic alignment of more modalities in the higher-dimensional space in which they lie. Conversely the conventional cosine similarity, the proposed GRAM evaluates the aligmment of more modalities without requiring to compare pairs of them, therefore preserving the rich semantic relations of the multimodal space. Moreover, is well-defined for to n modalities, making it extremely flexible and adaptable to any real-world scenario and task. Further, we introduce novel volume-based contrastive learning loss function, which leverages the recently introduced GRAM to lead multimodal models in shaping unified and aligned embedding space.
dimension n define
We experimentally show how rethinking the multimodal alignment process with GRAM brings consistent improvement to the state the art in downstream tasks, and that our intuition that more modalities altogether provide richer semantic information is validated. Specifically, the multimodal model pretrained with the GRAM contrastive loss outperforms SOTA models by to 10 points in tasks such video-text and audio-text retrieval video classification in several datasets without any architectural modification and with the same number network parameters. These results prove once again that GRAM better models the world multimodality with respect the conventional pairwise alignment process.
Our main contributions be summarized follows:
We introduce GRAM, a novel similarity measure for multimodal representation learning and alignment based the volume computation of the parallelotope spanned by all the modality vectors. The proposed measure can replace cosine similarity in any downstream method, providing richer and more informative semantics on modality representations by guaranteeing geometric alignment between modalities simultaneously.
We introduce novel GRAM-based contrastive loss function that be involved to pretrain or finetune any previous existing multimodal model, building unified and aligned embedding space.
2

---

Published as a conference paper at ICLR 2025
We show that our measure also serves quantitative metries for the performance of past and future multimodal models, providing unique insights on latent feature alignment.
We provide the proof of the extension of GRAM from up to n modalities, mathematically proving, and experimentally showing, that GRAM findings hold for modalities, making it extremely generalizable.
We show how multimodal models trained with the proposed contrastive loss outperform SOTA models, bringing improvements in several downstream tasks such as video-audio- text retrieval and audio-video classification.
2 RELATED WORK
Two-modal Alignment. CLIP Radford et al. (2021) originally revolutionized multimodal learn- ing by providing a foundational model that effectively aligns two modalities, specifically images and text. This work has inspired subsequent advancements aimed at enhancing performance and alignment quality across modalities Uesaka et al. (2024); [Iharco et al. (2021): Zhai et al. (2023). Additionally, CLIP architecture has been adapted facilitate alignment between other modality pairs, such as audio and text in CLAP|Elizalde et al. al. (2023), video and text in CLIP4Clip Luo et al.) (202I), and point clouds and text in PointCLIP Zhang et al. (202I). These models commonly rely on contrastive learning strategies, which push dissimilar embeddings apart while drawing similar vectors closer together, providing a robust method for learning cross- moosl representations.
Multimodal Alignment. Recent research has extended beyond two-modal alignment to capture the more complex multimodal nature of real-world data [Ly al.) (2024). For instance, CLIP4VLA Ruan et al. (2023) integrates audio within the CLIP framework, aligning video, audio, and text in pairs, using audio embeddings as a central anchor. Building on this, ImageBind/Girdhinn et al. (2023) offers pre-trained multimodal framework encompassing modalities like depth and infrared, posi- tioning the image modality as the bridge across modalities. In contrast, LanguageBind Zhu et al. 2024 demonstrates that text, rather than images, is more effective anchor for multimodal integra- tion. Alongside these innovations, other models such as VALOR Chen et al. (2023b). VAST Chen et al. 2023c), mPLUG-2 Xu et al. (2023), VideoPrism Zhao et al. (2024), and Intern Video2 Wang et al. (2024b), among others, either propose large pretraining datasets or introduce architectural modifications that further optimize performance. Among these, VAST proposes to fuse multiple modalities by an MLP layer and then involve conventional contrastive losses between such fused omni-modality embedding and the anchor one. Despite the lack of interpretability of this solution, at least VAST builds the first attempt towards multimodal- space.
Despite their contributions, these approaches still operate within a constrained lower-dimensional space, often relying on cosine similarity on a 2D plane defined by two modalities or using fusion strategies to combine multiple modalities. As a result, these models fall short in fully exploiting the rich, high-dimensional information inherent in multimodal data, which is crucial for addressing more complex downstream tasks.
Gram Matrix in Deep Learning. The Gram matrix properties have been leveraged in deep learning models for better defining matrix theory behind these models Pennington & Worah (0117). or to improve performance in different downstream applications. Examples are learning rotation-invariant point cloud representations Xu et al. (2021), sound event detection Neto et al. (202I), neural style transfer Li et al. (017); Friedrich & Menzel (2019), and domain adaptation Nejjar al. (2023). In addition and interestingly. the Gram matrix has also been proved to bring some insights on GAN (eeaatiiooonSnnikkkk
representations Seddik et al. (2020).
GRAMIAN REPRESENTATION ALIGNMENT MEASURE
Our goal is to achieve comprehensive geometric learning and alignment of multiple modality rep- resentations within their inherent high-dimensional spaces. Unlike traditional pairwise alignment methods, GRAM advances beyond pairwise limitations to introduce a novel strategy for modeling and interpreting latent multimodal representations in an integrated manner. By jointly learning and then aligning these modalities, our method exploits the full potential of multimodal models, while maintaining high degree of geometric interpretabilityy The GRAM measure replace cosine
3

---

Published as a conference paper at ICLR 2025
similarity in any downstream method bringing several advantages such enhancing model perfor- mance and providing meaningful information on latent representation alignment.
3.1 PRELIMINARIES
Multimodal representation learning aims to learn latent representations from co-occurrent data modalities. The i-th modality is encoded from its own domain in an n-dimensional latent repre- sentation using an encoding function & : M, + M; with M, E IR". A representative example may be the video-audio-texx representation where we have tri-modal representation with three encoding functions: ev V + V as visual encoder, ε A + A as audio encoder, and er ' + 'T' as text encoder. Here, E R-NxxCxWxx x is the visual domain assuming videos with N frames each one with C' channels and resolution of W x H, A RRNN.XN, is the audio domain assuming audio with N. samples for each No channels, and T = [t.....tM] is the textual domain composed of set of tokens f2 E V Vocobulary. V. A, T R".
Usually, the similarity between two modalities M,, M, is obtained by computing the cosine of the angle 6,y between their two representations:
(M, M,)
cos,)),) = || |My]l
where (M, M,) is the dot product between modality M, and modality Mj. and |M;]] is the norm of M,
Since cosine similarity is not defined for three or more vectors altogether, the classical approach is to select one modality as bridge modality and align all the remaining N-1 modalities to the first one couple-wise. In this scenario whatever type of contrastive loss could be used, the most common is the one introduced by Radford et al. in CLIP|/adford al. (202T):
B Σ 1=1
cxp(m, a:/T)
1
B
cxp(a, m:/T)
CM2A
B log
E E cxp(m) ay/T) Ca2M
log
(2)
B
E exp(a, m₃/r)
1=1
where m; is the normalized embedding of the i-th modality data and a; is the normalized embedding of the bridge modality and B is the batch size parameter.
At inference time, the problem becomes even more apparent since conventional methods do not possess a direct mathematical formulation to compute similarity among three or more embedding vectors. Therefore, they rely on just two modalities develop suboptimal neural fusing strategies. For instance, LanguageBind Zhu et al. (2024) attempts to linearly combine two modalities and then compute similarity with the third one, while methods like UMT et al. (2022), m-PLUG2 Xu al. (203)) and VAST Chen et al. (2023c) introduce layers that fuse two or more modality embeddings before computing cosine similarity with the remaining one.
3.2 VOLUME OF THE PARALLELOTOPE SPANNED BY THE MODALITY VECTORS
A generic embedding vector with dimension n is a vector in the nighee-dimensional space IR", whose components indicate its extremity. In the case of various modalities encoded with the corre- spondent encoding functions, multiple embeddings vectors V1...., Vj lie the higher-dimenssinal space IR", i.e. V; R", Vi E [1. k). The aim of whatever multimodal representation framework is ideally have correlated embeddings near to each other and uncorrelated embeddings far away, so as to have an aligned and meaningful embedding space. In this paper, we argue that measure of the relationship among the vectors the hyperdimensional space can be given by the volume of the k k-dimensional parallelotope with these vectors sides. Recalling that the vectors are normalized to have a unitary norm, their extremities lie on the surface of the hypersphere with a radius equal to 1, as shown in Fig. 2 The way obtain the volume of whatever type of k-dimensional parallelotope is computing the determinant the Gram matrix G of the vectors Gantmacher (1959). as we show in this Section.
Definition 1: Gram Matrix. Let Viy..., VA be vectors in R", these points can arranged as columns in matrix A (Viy...y Vk). Then, the Gram matrix G(v Vk) Reck is defined:
4

---

Published as a conference paper at ICLR 2025
(i. V1
(V1, v2) (V2, v2)
(vi, Va (V2, Vk
(v2, V1.
G(v......, vv) = ATA=
(ve.Vi)
(Vk v2)
(vi. v&)
Theorem 1: Volume of the k-dimensional parallelotope. Given Vi...., Va be vectors in I" form- ing a k-dimensional parallelotope, and the Gram matrix G E * then its determinant, also called the Gramian, is the square of the volume of the k-dimensional parallelotope formed by the vectors Gantmacher [959]:
Vol(Vi...., Vo. = v
/ del GVV V&).
(4)
Note that det G = A = det A = |det 2 O, so the square root in Eq. 4 is well-posed.
Proof. See Appendix
Interestingly, the case of & n, i.e., when the number of modality vectors lower than the dimension of the ambient space n. the Gramian provides us reliable computation for the volume of k-parallelotope spanned by those vectors, yielding crucial information on the alignment of the modalities. Therefore, the Gramian computation of the volume is naturally and easily extended to the number of modalities that we want to use in our framework, from 2 up to n, where the latter is the dimensionality of the latent space. In the case when & >> the vectors are linearly dependent, the volume of the p-parallclotope is still positive but equal to zero and the Gramian is not informative. However, in current neural embedding models, k < the findings hold for the largest part of real-world scenarios known up to now.
3.3 GRAM: VOLUME AS A MEASURE OF MODALITY VECTORS ALIGNMENT
The notion of volume in hyperdimensional spaces offers an intuitive approach for understanding the geometric alignment of vectors, which are the representation of data from various modalities. Specifically, the volume spanned by a set of vectors provides a measure of their proximity in the hyperdimensional space. A smaller volume indicates that the vectors closely aligned, suggesting a close semantic relation between the underlying data. Conversely, a larger volume suggests that the vectors and, by extension, the input data they represent- are less correlated potentially opposite. Our Gramian Representation Alignment Measure relies on this crucial intuition to fully exploit the richness of the semantic data. This novel methodology facilitates the discovery of interrelationships between sub- modaliidlssiss by computing the volume of the k- d-dimensional parallclotope formed by any subset of vectors. Interestingly, the GRAM computation is also efficient as it relies on computing the determinant of a k x k matrix, where, in the largest part of real-world scenarios, h: < and usually or 4, meaning it requires negligible computation time. Importantly, the proposed measure is scalable and can be applied to any number of modalities, ranging from 2 to n. Interestingly, in the specific case where & 2, the volume computation reduces to the calculation of the area of parallelogram spanned by two vectors with the origin in the hyperdimensionnal space. Notably, this area mathematically linked to the angle between the two vectors, thus establishing a direct correspondence between volume and previous similarity methods in the k = 2 case. The extension of this concept to similarities higher-dimensional volumes involving l > vectors generalizes cosine and sine complex inter-vector relationships. A formal derivation of this generalization is provided in Appendix A
3.4 GRAM-BASED CONTRASTIVE LOSS FUNCTIONS
In this subsection, we completely rethink multimodal representation alignment, by introducing the proposed
contrastive loss functions for representation learning and alignment. We de- fine a novel multimodal contrastive loss function completely relying on the aforementioned volume computation. We then describe the additional loss function to enhance the proposed method further.
5

---

Published as a conference paper at ICLR 2025
Mocality Tokions
who
w
Modality 1 Encoder
Modaby
Modality 2 Encoder
Modality Encoder
Cpsa
"When Billo Roggins of Bag End innonneed that would shortly be colebrating fits elovengy first borthslyy with party sperial magnificence, much milk and escitement tn Hobbiton.
Modality it Encoder
Multimodal
Encodar
Epus
Plkm
Closs Tokens
Figure 2: GRAM-based model architecture. Class tokens from each modality are involved in shap- ing the k-dimensional parallelotope, whose volume indicates the semantic alignment of the modal- ities. All the tokens are then involved in the multimodal encoder to enhance the predictions. The model is pretrained with the proposed Gramian multimodal contrastive losses DDa and CDAM.
GRAM Multimodal Contrastive Loss. The core of our novel proposal is the volume computation measure for the semantic alignment of vectors. Therefore, exploit the volume defined in Eq. into the contrastive loss of Radford et al. (202T). Such novel loss encourages volume account in joined fashion, ensuring the
minimization computed by taking all the modalities semantic alignment of the modalities altogether. After extracting the embedding vectors using en- coder functions as defined Section 3.J we normalize them. In this way, each modality M; is represented in latent embedding vector and the norm of such vector is always 1. We select an anchor a choosen as one of the & modalities. Therefore, proposed GRAM multimodal contrastive loss is:
B L =
exp( Vol(a, m m mss)T)))))))))))
Σ log
(5)
B
j=1 exp(-Vol(ar m m....., m.) mT)
im1
1
exp(- (oo m2i............................. mk:)/7)
CADD
log
(6)
K Σj=1 exp(- exp(-Vol(a,, may. m/T)
The volume minimization ensures that the model aligns all the modalities toward a common goal, with the decisive geometric guarantee that the intermodalities converge. By normalizing the vectors unitary norm, we prevent degenerate cases where the k-dimesnional parallelotope collapses ward the origin of the Cartesian coordinate system, and we secure the minimization of the volume by moving vector directions towards each other.
On the anchor selection: metrics. An anchor can either be a single modality (e.g., text, which common tasks involving text-to-data problems) or multimodal anchor, consisting of two or more modalities combined (e.g.. both video and audio). Selecting a specific modality as the anchor is used to direct attention toward it, effectively modeling the entire latent space around that modality.
The choice of anchor modality plays crucial role in the final evaluation
In Eq and and Eq we consider the case of single modality anchor a choosen as one of the modal- ities. Therefore, a, refers to the embeddings of the anchor modality of æ-thh sample in the batch, while may refers to the embedding of x-th modality of the j-th sample in the batch. is the batch size, T learnable scaling parameter.
Data-Anchor Matching Loss. In addition to the proposed volume-based multimodal loss function, employ the supplementary data-anchor matching loss. This loss aims encouraging the model

---

Published as a conference paper at ICLR 2025
to infer whether pair of anchors and data is matched not. To compute such matching, any previous encoder model can be involved. Suppose to choose the single-modal text anchor, the text encoder can act as multimodal encoder, whose input is caption tokens and with the data features as conditioning through cross-attention layers. In this way, we obtain data features by concatenating unpooled features from all the encoders along the sequential dimension. At the bottom of the mul- timodal encoder, an MLP layer returns binary predictions Pdam- These predictions are compared with true labels y through binary cross entropy. y when anchor and all other modalities are matched, 0 otherwise. We follow Li et al. (2021) in the hard negative mining strategy, obtaining the following data-caption matching loss:
CDAM = (aaa.m..... (A MM) y log Pitam + (1 - y) log(1 (1 - Pdtam) )
GRAM-based Model. The proposed GRAM Multimodal Model is pretrained with the volume- based contrastive loss functions introduced in Eq. [] We additionally involve the data-anchor match- ing loss in Eq. to refine the model predictions. Figure shows the architecture of the model, which based on VAST [Chnn et al. (2023c) encoders. For each modality, we map the class tokens into the highhe-dimensional latent space and compute the volume as alignment measure. Meanwhile, all the modality tokens of the anchor flow into the multimodal encoder, and the tokens from the other modalities serve as conditioning for the final binary predictions. The final pretraining loss is combination of DDAA, LADD, and LAMM as:
Cror = (DDZa + CA2D) + ALDAM,
(8)
2
where A 0.1, following Chen et (023c We provide the details of the models in Appendix B
3.5
GRAM AS MODEL PERFORMANCE METRIC
As GRAM is a measure of the multimodal em- bedding space alignment, can also serve as metric for evaluating large multimodal mod- els. Indeed, the more aligned the multimodal latent space, the better the model will per- form in downstream tasks, as it possesses rich semantic information. We prove the validity of these claims by extracting the embeddings from different multimodal models, Language- Bind (LB), VAST, and GRAM-based model, and computing the zero-shot (zs) and fine-tuned (ft) Recall at I (R@) l). Then, we compute the proposed GRAM, which we rescale for visual- ization purposes and use 1 1 GRAM. Figure 3 shows the average, over 1000 test samples, of R@1 1 (the higher the better) and 1 GRAM (same) for the video, audio, and text embed- dings MSR-VTT. The linear regression line computed and added in grey dashed style.
65.0
62.5
60.0
Perforaace (R01
57.5
56.0
LB (zs)
52.5
A
VAST (29)
p= 0.923
GRAM (zs)
50.0
LB (ft)
47.5
VAST (ft)
VALOR (ft)
45.0
GRAM (ft)
0.04
0.06
0.08
0.10
0.12
0.14
0.16
Similarily (1 GRAM)
Figure 3: The proposed GRAM similarity is strongly correlated (p 0.923) with large multi- modal models performance in downstream tasks.
The plot shows strong correlation between the two metrics, confirmed from the Pearson corre- lation coefficient (p) equal to 0.923. This means that the volume of the k-dimensional parallelotope spanned by the modality embeddings of every model is strongly linked to the way such a model will perform in the task. Therefore, these results prove that the proposed GRAM is a good metric for large multimodal model performance and that it can be involved as such in future research.
EXPERIMENTAL EVIDENCES
In this Section, present the main results of the proposed GRAM contrastive loss and model in downstream tasks. In addition, we show how the multimodal latent space built with GRAM is more meaningful and disentangled with respect to others.

---

Published as a conference paper at ICLR 2025
Table I: Zero-shot multimodal text-to-video (T2V) and video-to-text (V2T) retrieval results in terms of Recall at I score (R 1). Increment points computed wrt VAST with same modalities.
MSR-VTT
DiDeMo
ActivitNee
Modality
I2V
V2T
T2V
V2T
T2V
V2T
12V
V2T
Liu.etl al. 2022
TV
33.3
34,0
31.9
Orna VD (Sene
A032
TV
31.6
33.3
UMT-L
TVTS+
VICLIP Wang in
21725
1-
42.4
41.3
18.4
27.9
15.1
24.0
VideoC
M12A
TVV
34,3
64.7
34.5
33.0
53.2
73.6
Norton Lin brain 2012
TV
Imagebind Cippllir
InternVille Aortii punelo
B-
40.7
39.6
31.5
33.5
30.7
31.4
49.5
69.5
HiTeA Yo piazed
T-V
34.4
43.2
4P00L A033
TV
45.7
ViskoPristi-
77.1
LanguazeBin.
40.9
39.8
41.0
39.1
VAST Chen dh 2376
T-VA
49.3
43.7
49.3
48.2
51.4
46.8
80.0
77.3
VAST erarer
TAVAS
50.7
49.0
82.1
78.7
GRAM Model (Ours)
49.5
$4.0
52.3
58.9
50.9
79.0
(+4.9)
(+6.7)
(+4.7)
52.2 (+4)
(+7.6)
(+3.6)
79.2 (+1.9)
GRAM Model (Ours)
T-VAS
$4.8 (+1.1)
$2.9 (+3.9)
83.5 (+1.4)
82.7 (+1.0)
Table 2: Finetuning multimodal text-to-video (T2V) and video-to-text (V2T) retrieval results in terms of Recall at score (R@) 1). Increment points computed wrt VAST with same modalities.
MSR-VTT
DiDeMo
Activity
WATEX
Modality
T2V
UMT-L gt 2023
T-V
58.8"
70.4
65.7*
64.4
72.0"
CLIP4 und mirer m 2
T-V
45.6
45.9
43.01
43.6
40.3
41.6
63.0
78.3
WCT mi S3res
T-V
52.5
518
49.4
50.2
49.8
48.1
Intem Vala-1
2022b
T.V
55.2*
$7.9"
59.1
62.2"
62.8"
71.1"
87.2"
T-V
46.8
2025
TV
53.1
56.4
VALOR-L et 7i
RUZAh
TVAS
54.4
57.6
63.4
76.9
TEFAL [hrantmn 31
TAA
52.0
61.0
Bimodar 1230
2024
T-V
36.8
TMASSWWnl Won-eveen NUMI
vid-T1
DEU A01
T-V
58.5"
704"
VAST Thoire di 207
TVA
558
576
65.6
62.0
68.8
66.7
86
84.1
VAST
Lilorr 20123
TV EVAS
56.6
57.6
87.5
84.0
GRAM Model (Ours)
55.7
5644
66.4
63.2
66.5
61.6
84.4
81.6
63.5 (+1.5)
(+1.1)
(40.2)
(+0.5)
GRAM Model (Ours)
TVAS
(4.0()744)
64.
87.7 (+0.7)
84.2 (40.2)
"Finctuning and evaluation with 12 frames.
4.1 EXPERIMENTS SETUP
We build the GRAM model with VAST models Chen et al. (2023c) as backbone. Therefore, the text, audio, and video encoders are BERT-B BEAT's Chen al. (2023a), and SWAClip-ViT--C Sun et al. (2023), respectively, with total number of parameters equal to IB. Obviously, we remove VAST fusing layers as GRAM does not require them due its ability to truly model the multimodal latent space. Starting from VAST pretraining models, we further pretrain those on small subset of VAST27M Chen et al. (2023c) comprising 150k samples using our defined loss functions. This operation is useful to reshape latent space already built by VAST using our GR GRAM-based contrastive loss function. We inherit from VAST the learned semantics space and we reshape it using our novel loss function. We set the batch size 256 and single epoch pretraining on 4 NVIDIA A100 cards.
As downstream datasets, we consider several well-known multimodal benchmarks that can be di- vided into three categories: (i) three- three-modal video-based, such as DiDeMo and ActivityNet, in which the crucial modality is video and the two other modalities (audio and text) are supportive; (ii) four modal video-based, such as MSR-VTT and VATEX, in which video is the main modality but also audio, text, and subtitles are supportive; and (iii) audio-based, like AudioCaps and VGGSound, in which the audio modality is the most relevant, while also video and text contain interesting informa- tion. Details about datasets, samples, and resolutions are in Appendix B
4.2 RESULTS AND DISCUSSION
Multimodal Text-to-Video Retrieval. Table T and Table 2 show the improvements that the pro- posed GRAM-based contrastive loss brings to large multimodal models in both zero-shot and fine- tuning scenarios, respectively. The proposed GRAM model surpasses by large margin the wide
8

---

Published as a conference paper at ICLR 2025
Table 3: shoo multimodal text-to-audio retrieval results AudioCaps and zero-shot audio-text classification results on VGGSound 5K.
AudioCaps
VGGSound SK
Modality
R@1
R@
Acc
Acc 10
AVFIC Nagrani et al.
2022
T-A
8.7
37.7
AVTIC (vaprant et Mll
T-AV
10.6
45.2
VIP-ANT
T-A
37.7
Imagellind Cardbarr et 3. 2023
T-A
9.3
42.3
Language Binn 7T0
2024
T-V
37.2
62.0
Languare
VAST VAST Chen el a 27750
T-V
38.7
72.8
VAST
TAA
25.6
56.2
20233
GRAM Model (Ours)
T.AV
(+
75.3 (+9.9) 40.6 (+1.0) 78,1 (+3.6)
L-SNE Visualization of VAST latent space
L-SNE Visualization GRAM latent space
-25
-5.0
-7.5
-10.0
Text Emberdinng
Audic Embeddings
Visual Embesidins
Peop crowd
Reversing beeps
Cat meowing
Figure 4: t-SNE visualization on VGGSound of VAST, casine-based, (left) and GRAM (right) latent spaces. GRAM better models the latent space, resulting in space more disentangled and highly interpretable. Class clusters are easily recognizable space, while video (squares) and audio (tri- angles) modalities that are closer to the classification text (star).
of comparison methods in every dataset we test and both cases. In the zero-shot scenario, the GRAM-based model improves the Recall scores with respect to its cosine-based counterpart VAST by an average of 4.5 points, representing true consistent advancement. These crucial results in zero-shot scenarios mean that the GRAM latent space is much more aligned and semantically sep- arated than the one built with cosine similarity. The meaningfulness of the unified semantic space is clear in finetuning results too, in which the proposed model improves the performance up to 7.4 points @ in the MSR-VTT dataset. The largest part of previous models can only consider pairs of modalities T-V or present fusing strategies to consider triplets T-VA, they solely rely the cosine similarity between the two. As a consequence, such models cannot fully exploit the semantic space that multiple modalities define. Instead, the GRAM-base -based model is pretrained to semantically align all the modalities, shaping a more informative, representative, and meaningful latent space. Finetuning results may be strongly affected by the quality of the original training data. We find that several datasets like DiDeMo and ActivityNet has audio modality semantically very different from the video one probably due to the random selection from YouTube. MSR-VTT, the other hand, shows higher correlation among multiple modalities that the GRAM model is able to exploit. This outcome demonstrates the importance modeling multimodal latent spaces unified way and the contribution of multiple modalities is pivotal to retrieve the correct data. Additional results and visualizations, including real-world examples in Appendix B particularly in Fig. 6
Multimodal Text-to-Audio Retrieval & Classification. Table 3 shows the improvements of the GRAM-based model in audio-to-text retrieval (recall at and 10 in AudioCaps) and audio-text classification (accuracy at and 10 in VGGSound). In these scenarios, the proposed GRAM-based model outperforms all the comparison methods. Specifically, GRAM surpasses VAST by up to 9.9 points of R @ 10, meaning the space that GRAM builds is much more aligned and disentangled than other methods. Table 3 reveals the large adaptability of the GRAM method, which can easily be employed in different datasets and tasks in zero-shot scenarios improving methods performance.

---

Published as a conference paper at ICLR 2025
4.3 VISUALIZING THE ALIGNED GRAM LATENT SPACE
To demonstrate the capabilities of the GRAM-based contrastive loss introduced in Section 3.4 we analyze the latent embedding space using t-SNE visualization. For this purpose, we utilize the VGGSound dataset, as it is the only dataset among those considered that includes class labels for its samples, favoring clearer representation. We visualize the top three classes with the most examples in our downloaded portion. For each class, we extract the embeddings using both VAST and the proposed GRAM--ased model for the class labels (text modality). as well for visual frames and audio samples.
Figure presents the resulting visualizations, where VAST embeddings are on the left, and GRAM space is on the right. The embedding space generated by the GRAM model is significantly more disentangled, with the three modalities of each class well-aligned around the text modality (star symbol). In contrast, the latent space produced by the VAST model appears more scattered, with the different modalities dispersed throughout the embedding space, exhibiting less alignment. Figure 4 provides the visualization of intuitions, i.c. that GRAM is able to effectively model the multi- modal latent space according to the semantics of different modalities. Therefore, in GRAM, all the modalities contribute to the correct classification, resulting in improved performance.
4.4 GRAM SCALING TO MORE MODALITIES
To experimentally confirm the proof that GRAM holds from 2 to n modalities, we include additional modali- ties in the MSR-VTT dataset. The largest part of mod- els employ solely the video to correctly retrieve the tex- tual caption. Therefore, first, we test GRAM with these two modalities only, proving that, this case, the volume computation degenerates to the area of the triangle and that this is still a reliable metric. Following, we add one- by-one the other modalities like audio and subtitles. Table 4 showws that GRAM effectively works for two modalities and that enriching the latent space with more modalities improves the performance from R@I I equal to 52.8 up to 54.8 in text-to-video retrieval. To further stress the proposed GRAM claims, we insert an additional modality to the four already present in MSR-VTT. We employ ChronoDepth Shao et al. (2024) to extract depth maps from video frames, and add a head to the GRAM vision encoder to process this additional modality. Once pretrained with five modalities on the subset of VAST27M, we perform zero-shot on MSR-VTT with all the modalities. Interestingly, not only the method still perfectly works in defining the 5-dimensional parallelotope spanned by the vectors and in computing the vol- ume, but also further increase the performance up to recall of 55.3. These results prove two key factors of this work: (i) the intuition that more modalities contribute to a richer semantic space and are often crucial in correctly performing donwstream tasks; (ii) the mathematical proof that the proposed volume computation naturally scales to more modalities beyond the classical ones.
Table 4: GRAM naturally scales to more modalities, shaping semanti- cally richer latent space that contribute in improving the model performance in downstream tasks. Text-to-Videe zero- shot results on MSRVTT Text
Video
Audio
Sub. Depth
R@I
52.8
54.1
54.8
55.3
5 CONCLUSION
In conclusion, we presented GRAM, a fundamentally new measure for multimodal representation learning and alignment that operates in the higher-dimensional space spanned by all modality em- beddings. By modeling the alignment through the volume of a parallelotope formed by k modality vectors, GRAM captures richer semantic relationships than traditional pairwise methods like cosine similarity. Furthermore, we introduced a novel GRAM- based contrastive loss, which leverages this geometric alignment to guide multimodal models in shaping unified embedding space. The model pretrained using our loss outperform state-of-the-art methods by significant margins across multi- ple tasks, confirming that GRAM generalizes well to a wide range of modalities and tasks. This work represents significant advancement in multimodal representation learning and alignment by addressing the limitations of pairwise alignment and providing a mathematically grounded, flexible solution for aligning any number of modalities. We believe that GRAM opens up new directions for the field, offering a more powerful and general framework for multimodal understanding and providing new insights into the structure of multimodal spaces.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
In this paper, we present a novel approach to multimodal representation learning. To ensure that our work can be easily reproduced and built upon by the research community, we have taken sev- eral key steps. First, the source code implementing our multimodal representation learning model available as part of the supplementary materials. The code includes all scripts necessary for training, evaluation, and data processing, while pretrained models will be released after reviewing process. Experimental settings and hyperparameters are available in the supplementary materials. We use publicly available datasets and details training, validation, and testing splits are reported in supplementary materials. In terms of theoretical contributions, we include clear explanations of assumptions and complete proofs for all claims made in the paper. Formal derivations and justifica- tions can be found in the appendix for further verification. Finally, we also provide details about the hardware and software environment used in our experiments.
By providing detailed descriptions, open-source code, and clear theoretical justifications, we aim make our work on multimodal representation learning fully reproducible and accessible to the broader research community.
ACKNOWLEDGEMENT
The work of G. Cicchetti, E. Grassucci, and D. Comminiello work was partially supported by the European Union under the Italian National Recovery and Resilience Plan (NRRP) of NextGenera- tionEU, Mission 4, Component Investment 1.3, partnership on "Telccommunications of the Fu- ture" (PE00000001 program "RESTART") and by the partnership on "Future Artificial Intelligence Research" (PE00000013 - SPOKE 5 CUP B53C22003980006 FAIR: High Quality AI). Addi- tionally, the work of I.. Sigillo was partly supported by "Ricerca innovazione nel Lazio incentivi per i dottorati di innovazione per le imprese e per la PA L.R. 13/2008" of Regione Lazio, Project "Deep Learning Generativo nel Dominio Ipercomplesso per Applicazioni di Intelligenza Artificiale ad Alta Efficienza Energetica", under grant number 21027NP000000136.
REFERENCES
Pranav Arora, Selen Pehlivan, and Jorma Laaksonen. Text-to-mulimodall retrieval with bimodal input fusion in shared cross- cross-modal transformer. International Conference on Language Re- sources and Evaluation. 2024.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2015.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: large-scale audio- visual dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725, 2020.
Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. In Interna- tional Conference on Machine Learning, pp. 5178 7793,, 2023a.
Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. VALOR: Vision-audio-languaa omni-perception pretraining model and dataset. ArXiv preprint: arXiv:2304.08345, 2023b.
Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Ming-Ting Sun, Xinxin Zhu, and Liu. VAST: A vision-audio-suutiill--eee omni-modalitg foundation model and dataset. In Neural Information Processing Systems (NeurIPS), 2023c.
Joonmyung Choi, Sanghyeok Lee, Jaewon Chu, Minhyuk Choi, and Hyunwoo J. Kim. vid-TLDR: Training free token merging for liight video transformer. IEEE/CVE Conference on Com pufer Vision and Pattern Recognition (CVPR), pp. 18771-18781, 2024.
11

---

Published as a conference paper at ICLR 2025
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. CLAP: learning audio concepts from natural language supervision. In IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Timo Friedrich and Stefan Menzel. Standardization of gram matrix for improved 3d neural style transfer. In IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1375-1382, 2019.
Felix R. Gantmacher. Matrix theory. Chelsea Publishing Company, 1959.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. ImageBind one embedding space to bind them all. In JEEE/CVF Con- ference Computer Vision and Pullern Recognition (CVPR), pp. 15180-15190, 2023.
Lisa Anne Hendricks, Oliver Wang. Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Rus- sell. Localizing moments in video with natural language. In IEEE International Conference on Computer Vision (ICCV), pp. 5804-5813, 2017.
Sarah Ibrahimi, Xiaohang Sun, Pichao Wang, Amanmeet Garg, Ashutosh Sanan, and Mo- hamed Omar. Audio-enhanced text-to-video retrieval using text-conditioned feature alignment. IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12020-12030, 2023.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi. Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 1655000000000500 2017.
Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. AudioCaps: Generating Captions for Audios in The Wild. In NAACL-HLT, 2019.
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Neural Information Processing Systems, 2021.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards vraining-efficien video foundation models. IEEE/CVF International Confer- ence on Computer Vision (ICCV), pp. 1999111903, 2023.
Yanghao Li, Naiyan Wang. Jiaying Liu, and Xiaodi Hou. Demystifying neural style transfer. In International Joint Conference on Artificial Intelligence, 2017.
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Un- derstanding the modality gap multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 2022.
Yijie Lin, Jie Zhang, Zhenyu Huang, Jia Liu, Zujie Wen, and Xi Peng. Multi-granularity cor- respondence learning from long-term noisy videos. In International Conference Learning Representations, 2024.
Ye Liu, Siyuan Li, Yang Wu, Chang Wen Chen, Ying Shan, and Xiaohu Qie. UMT: Unified multi- modal transformers for joint video moment retrieval and highlight detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3042-3051, 2022.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. CLIP4Clip: An empirical study of clip for end to end video clip retrieval. Neurocomputing, 508:293-304 2021.
Yuanhuiyi Lyu, Xueye Zheng, Xueye Kim, and Lin Wang. Omnibind: Teach to build unequal-seale modality interaction for omni-bind of all. ArXiv preprint: arXiv:2405. /6108, 2024.
12

---

Published as a conference paper at ICLR 2025
Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manén, Chen Sun, and Cordelia Schmid. Learning audio-video modalities from image captions. European Conference Computer Vision, 2022.
Ismail Nejjar, Qin Wang, and Olga Fink. GRAM: Unsupervised domain adaptation regres- sion by aligning inverse gram matrices. In /EEE/CVE Conference on Computer Vision and Pattern June 2023.
Recognition (CVPR), pp. 1 11744-11754,
Antonio Joia Neto, André G. C. Pacheco, and Diogo Carbonera Luvizon. Improving deep learning sound events classifiers using gram matrix feature-wist correlations. IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pp. 3780-3784, 2021.
Andreea-Maria Oncescu, A Koepke, Joao F Henriques, Zeynep Akata, and Samuel Albanie. Audio retrieval with natural language queries. arXiv preprint arXiv:2105.022: 02219292, 2021.
Jeffrey Pennington and Pratik Worah. Nonlinear random matrix theory for deep learning. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning (ICML), 2021.
Ludan Ruan, Anwen Hu, Yuqing Song, Liang Zhang, S. Zheng, and Qin Jin. Accommodating audio modality in CLIP for multimodal processing. In AAAI Conference on Artificial Intelligence, 2023.
Mohamed EI Amine Seddik, Cosme Louart, Mohamed Tamaazousti, and Romain Couillet. Ran- dom matrix theory proves that deep learning representations of GAN-data behave as Gaussian mixtures. In International Conference on Machine Learning, pp. 8573-8582, 2020.
Jiahao Shao, Yuanbo Yang, Hongyu Zhou, Youmin Zhang, Yujun Shen, Matteo Poggi, and Yiyi Liao. Learning temporally consistent video depth from video diffusion priors. ArXiv preprint: arXiv:2406.01499 2024.
Swetha Simna, Mamshad Nayeem Rizve, Nina Shvetsova, Hilde Kuehne, and Mubarak Shah. Pre- serving modality structure improves multi-modal learning. In /EEF/CVF International Confer- ence on Computer Vision (ICCV), pp. 21936-21946, 2023.
Quan Sun, Yuxin Fang, Ledell Yu Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: Improved training techniques for clip at scale. ArXiv preprint: arXiv:2303. 15389, 2023.
Toshimitsu Uesaka, Taiji Suzuki, Yuhta Takida, Chieh-Hsin Lai, Naoki Murata, and Yuki Mitsufuji. Understanding multimodal contrastive learning through pointwise mutual information. ArXiv preprint: arXiv:2404. 19228, 2024.
Jiamian Wang. Guohao Sun, Pichao Wang, Dongfang Liu, Sohail A. Dianat, Majid Rabbani, Raghu- veer M. Rao, and Zhiqiang Tao. Text Is MASS: Modeling as stochastic embedding for text- video retrieval. JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 16551-16560, 2024a.
Junke Wang. Dongdong Chen, Zuxuan Wu, Chong Luo. Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. OmniVL: One foundation model for language and video-language tasks. In Advances in Neural Information Processing. 2022a.
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for aideo-and-languraee research. In IEEE/CVF International Conference on Computer Vision, pp. 4581-4591, 2019.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu. Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning. ArXiv prerint: arXiv:22/2.03319) 2022b.
13

---

Published as a conference paper at ICLR 2025
Yi Wang, Yinan He, Yizhuo Li. Kunchang Li, Jiashuo Yu, Xin Jian Ma, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Y. Qiao. Intern A large-scale video- text dataset for multimodal understanding and generation. ArXiv preprint: arXiv:2307.06942, 2023.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Z.engg Jilan Xu, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, and Limin Wang. Intern Video2: Scaling video foundation models for mul- timodal video understanding. ArXiv preprint: arXiv:2403. 15377, 2024b.
Haiyang Xu, Qinghao Ye, Mingshi Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qiuchen Qian, Wei Wang, Guohai Xu, Ji Zhang, Songfang Huang, Feiran Huang, and Jingren Zhou. mPLUG-2: A modularized mudti foundation model across text, image and video. In International Conference on Machine Learning, 2023.
Jianyun Xu, Xin Tang, Yushi Zhu, Jie Sun, and Shiliang Pu. SGMNet: Learning rotation invariannn point cloud representations via sorted gram matrix. In /EEF/CVF International Conference on Computer Vision (ICCV), pp. 10468-10477, 2021.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. VideoCoCa: Video-text modeling with zero-shot transfer from contrastive captioners, 2022.
Qinghao Ye, Guohai Xu, Ming Yan, Haiyang Xu, Qi Qian, Ji Zhang, and Fei Huang, HiTeA: Hierarchical temporal-aware video-language pre-training. JEEE/CVF International Conference 2022.
Computer Vision (ICCV), pp.
Sunjae Yoon, Dahyun Kim, Eunseop Yoon, Hee Suk Yoon, Junyeong Kim, and Chang Dong Yoo. HEAR: Hearing enhanced audio response for dideo-grounde dialogue. In Conference on Empir- ical Methods in Natural Language Processing. 2023.
Ziyun Zeng, Yixiao Ge, Zhan Tong, Xihui Liu, Shutao Xia, and Ying Shan. TVTSv2: Learning out-of-the-box spatiotemporal visual representations at scale. ArXiv preprint: arXiv:2305.1417. 2023.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11941- 11952, 2023.
Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Jiao Qiao, Peng Gao, and Hongsheng Li. PoinCCLIP: Point cloud understanding by clip. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8542 8552, 2021.
Long Zhao, Nitesh Bharadwaj Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J. Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, Rachel Hornung, Florian Schroff, Ming Yang, David A. Ross, Huisheng Wang, Hartwig Adam, Mikhail Sirotenko. Ting Liu, and Boqing Gong. Videoprism: A foundational visual encoder for video understanding. In International Conference on Machine Learning, 2024.
Yanpeng Zhao, Jack Hessel, Youngjae Yu, Ximing Lu, Rowan Zellers, and Yejin Choi. Connecting the dots between audio and text without parallel data through visual knowledge transfer. ACL, 2022.
Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, Hongfa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li. Wancai Zhang, Zhifeng Li, Wei Liu, and Liejie Yuan. LanguageBiind: Extending lideollnguage pretraining to n-modality by language- based semantic alignment. In International Conference on Learning Representations (ICLR), 2024.
14

---

Published as a conference paper at ICLR 2025
A THEORETICAL APPENDIX
A.1 THE GRAMIAM COMPUTES THE VOLUME OF ANY k-DIMENSIONAL POLYTOPE
Theorem I: Volume of the k-dimensional parallelotope. Given vi,..., V be vectors in IR forming k-dimensional parallelotope. Vi....: V& be arranged as column vectors in matrix A = (Vi,.... V&) and the Gram matrix G is computed as i The determinant of the Gram matrix, also called the Gramian, is the square of the volume of the K-dimensional parallelotope formed by the vectors Gantmacher 1959):
Vol ..... V1a) det G V&)--
Note the similarity to the norm of vector; fact, v(v) |/vl the Gramian of single vector is the length, i.e., the I-dimensional volume, of that vector.
The Gram matrix A captures all the geometric information about the vectors Vi Vk the length of the vectors and the angles between them. We claim that this is enough information to easily determine the volume.
To mathematically prove the theorem we have to distinguish three different cases:
Consider the case where k = n; then
det G(VI..... vn. = det A T A = (det A) A) = |A
G( 20
Since Vol(V1,..., VK) del AJ, we obtain:
Vol(v1...... vk) det ATA G(VV....,vvk
Consider the case where & < then the above holds by simply restricting to k- dimensional subspace containing them; Given Vi.... Vk, where k 2 n, assume that they are linearly independent and pick an orthonormal basis Wi...., W& for their span, W. Ex- tend to an orthonormal basis Wi,...y W for R"; the matrix O sending w; + e, is an or- thogonal change of coordinates, so it does not change inner products: O(v)-O(w) v. Let V; O(v,) then
m
m2i
(v
mik 0
Since i; is in the span of Wi,..., Wi- The set vi...., Vg visibly lies in the k-dimensional subspace of vectors with last n l coordinates zero. Effectively we are restricting to the subspace W. Since O is orthogonal,
Vol(v v = Vol(vi...... vx)
and Vi vy Vi Vj. 80
v G(v vz) Vol(v' v = Vol(vi....., vR)
as desired.
When > n, the vectors are linearly dependent, so the k-volumee of the parallelotope is zero, since the determinant of A matrix is zero.
15

---

Published as a conference paper at ICLR 2025
A.2 THEORETICAL ADVANTAGES OF VOLUME COMPUTATION WRT COSINE SIMILARITY
Let us consider the simple case with three modalities: Text (T), Video (V), and Audio (A). The Gram Matrix is equal
TT
TA
TV
G
AT
AA
AV
VT
VA
VV
Now, let compute the determinant of the matrix G:
= -AW
(10)
Recall that T, V, embeddings are normalized to unit norm and so TT VV = =A=1:
det G) 1- (1 - TA- (AT - AV VT) TV (AT VA - VT)
(11)
1 VA -AA AV VT TV AT
(12)
V - -V² + 2 TA AV VT
(13)
Therefore, the volume computation through the Gram matrix includes in its computation all the cross-products, resulting in an alignment of all the modalities together. In contrast, current state-of- the-art methods based on cosine similarity only compute the similarities between the modalities and the anchor (T' A and 7V),, omitting the similarities between non-anchor modalities (AV), which in practice may not be aligned.
A.3 THE VOLUME IS RELATED TO THE ANGLE BETWEEN THE TWO VECTORS IN THE TWO-MODAL CASE
Consider two vectors V1, V2 E R with ||v
= |/2 = 1. The Gram matrix G is given by:
(vIvv) V1) (v2 vi)
V2) (V2 V2)
G
(14)
Then, compute the determinant of the Gram matrix:
det(G) = (v. vi v2) (v, vz)² v2
(15)
The volume of the k-dimensional parallelotope spanned by the modalities vI, is:
Vol v GG) = (vi vi) -vI vvIv2)2
(16)
Given that vi and v2 have norm equal to 1, equation 16 reduces to:
Vol = /de((G) V vv]
v2)2
(17)
The cosine similarity defined as cos = (v] /2) Ill ||v2/|. however, considering the unitary norm of the embeddings vectors, the cosine similarity reduced to cos(f) (v, vz). Therefore, equation 17 becomes:
Vol = del = /1 - coss = sin = sin
(18)
Therefore, in the case two modalities, the volume computation reduces to the sine of the angle between the two modality vectors.
16

---

Published as a conference paper at ICLR 2025
Table 5: Dataset statisties and hyperparameters. Modalities stand for T. text, V: video, A: audio, S: subtitles, D: depth. Frames refers both to training and inference.
Modality
Benchmark
#Video #Audio
Frames
Epochs
Train
Val
Test
AudioCaps
700
8
VGGSound
5000
8
Threemodal (T-V-A)
DiDeMo
8394
1065
1003
8
40
ActivityNet
10009
4917
8
20
MSR-VTT
9000
1000
Fourmodal (T-V-A-S)
VATEX
14060
431
8
3
Fivemodal (T-V-A-S-D)
MSR-VTT
9000
1000
8
B EXPERIMENTAL APPENDIX
B.1 EXPERIMENTAL DETAILS
We utilize several benchmark datasets for our downstream tasks:
MSR-VTT Xu et al.
2016) comprises 10000 video clips accompanied by 200000 captions, encom-
passing diverse range of subjects including human activities, sports, and natural landscapes.
VATEX Wang et al. (2019) consists of 41250 video clips derived from the Kinetics-60l 600 dataset Kay et al. (2017 7) and 825000 sentence-level descriptions. Since a large part of the dataset is now unavailable online due to removed or private videos thus we use only portion of the original dataset composed of 14491 samples.
DiDeMo Hendricks et al. (2017) includes 10000 long-form videos from Flickr, with each video annotated by four temporally ordered short sentences. The dataset is uniquely annotated with natural language descriptions corresponding distinct moments within each video. For each video, four short sentences are provided, arranged in temporal order to describe specific events or scenes.
ActivityNet Caba Heilbron al. (2015) comprises 20000 long-form videos (with average du- ration of 180 seconds) from YouTube and 100000 captions. has a diverse range of 200 human activity classes, spanning daily activities, sports, and various other complex human behaviors. Ac- tivityNet is annotated with both activity labels and temporal boundaries, providing fine-grained in- formation about when specific activities occur within each video.
AudioCaps Kim et al. (2019) is a dataset comprising 51000 audio clips, each with a duration of 10 seconds. The dataset annotation structure varies between its subsets: the training set contains one caption per clip, while the validation and test sets are annotated with five captions clip. We follow the split protocol established by Oncescu al. [202I) for the text-to-audio retrieval task.
VGGSound Chen et al. (2020) is a large-scale audio-visual dataset containing more than 200000 videos sourced from You Tube, encompassing diverse range of 309 audio classes. Each video clip in the dataset has a duration of 10 seconds and is annotated with single label corresponding to the predominant sound event occurring within the clip. The dataset covers wide spectrum of audio events, including human actions, animal vocalizations, natural phenomena, and mechanical sounds. Due to several downloading problems we use only portion of the original dataset composed of 5000 samples in testing.
For every dataset, we utilize the official split for retrieval tasks, the dataset splits, and the number of frames for fine-tuning and evaluations all the datasets in Tab. 5 We pretrain the GRAM-based model on a subset of the VAST27M Chen et al. (2023c) dataset comprising 150k random samples with learning rate of le 4 using the AdamW optimizer with weight decay and batch size of 256. For finetuning we reduce the batch size to 64 and change the number of epochs according to the specific dataset, the complete details are shown in
17

---

Published as a conference paper at ICLR 2025
Table 6: Ablation study loss functions. Recall 1 is shown for both Text-to- Video (T2V) and
Video-to-Text (V2T) tasks training from scratch on MSR-VTT and ActivityNet datasets.
MSR VTT
ActivityNet
TV
TA
D2A
A2D
DAM
T2V
V2T
T2V
V2T
X
X
36.4
36.7
23.6
21.3
AXX
X
X
V
/
20.9
29.0
16.3
18.9
X
X
/
37.1
38.7
23.6
24.5
X
V
X
38.0
41.2
30.0
30.0
X
V
V
V
38.9
41.9
30.2
30.1
Multimodal V2T Rgh
Mutimodal T2V Rg
TV-TA
25
15
12
500
18
1.kc
VZT Rig
T2V
23
15
Value
Training
GIAA
TV-TA
GRAM
34
322
20
15c
252
28
Figure 5: Multimodal V2T/T2V, V2T/T2V, Gramian Value and training loss of GRAM trained with our loss functions and GRAM trained with only TV-TA loss functions. ActivityNet Dataset, training from scratch.
B.2 ABLATION STUDIES
We perform ablation studies further validate our experimental results. To evaluate the effec- tiveness of the introduced volume-based loss functions and the contribute of each one we perform comparative test on MSR-VTT and ActivityNet datasets. Results shown Tab. [aae obtained train- ing from scratch MSR-VEL STR- and 20 epochs on ActivityNet. The use our proposed loss funtions allow the model to obtain superior performance with respect to the classical approach in which we cosine-based contrastive loss functions among pairs of modalities.
model with different training settings. We train the model for 4 epochs
To further investigate the superiority of our approach compared the classical method, which relies on measuring similarity among pairs of modalities, we present several key metrics that provide insights into the behavior of GRAM during training. The gramian value is computed averaging
18

---

Published as a conference paper at ICLR 2025
Table 7: Modality Gap in GRAM. Results computed extracted embeddings of MSR-VTT.
Method
VV
TT
AA
VT
TA
VA
VAST (zs)
0.36
0.16
0.57
0.65
0.84
0.96
GRAM (7s)
0.39
0.19
0.85
0.54
0.98
1.17
VAST (ft)
0.08
0.07
0.69
0.37
0.86
0.84
GRAM (ft)
0.21
0.23
0.84
0.40
0.96
1.09
the volume measure among all true predictions made by GRAM. In Fig. 5 we show the results of training from scratch on the ActivityNee dataset. First, we report the use of only the TV-TA contrastive loss functions (corresponding present the results when our proposed loss functions are applied (corresponding to row As shown, the introduction of GRAM-based loss functions significantly improves the alignment of the entire latent space. This approach not only enhances the alignment between text and video modalities but also outperforms the specialized TV loss. Furthermore, the training loss is much more stable, and convergence occurs earlier, all with the same set of hyperparameters.
row 1 of Tab. 6 represented by the red lines). Next, Tab. ().
B.3 DISCUSSION ON THE MODALITY GAP
We further explore the phenomenon of the modality gap. introduced by Liang et al. (2022). In Tab. 7 we report the mean cosine similarity between embeddings of the same modality (VV Vision, TT - Text, AA - Audio) and the distances between the centroids of different modalities (e.g., VT - distance between the Vision centroid and the Text centroid, and so on). The centroids are computed following the methodology outlined in Liang et al. (2022). Consistent with the con- clusions of Liang al. (002)). we observe that the modality gap exists, and its relationship with final performance metrics complex and not easily interpretable. An empirical hypothesis we pro- pose is that the loss functions we introduce favor two behaviors: i) it squeezes the clusters of each modality more than VAST, indeed the average cosine similarities inside the modalities higher. ii) It increases the gap among the modalities, probably producing a more sparse latent space. This clear from the last three columns of Tab. where the distances between modality cluster troids are shown. Again, as clearly stated in Liang et al. (2022). although GRAM obtains better performance in downstream tasks, there no mathematical proofs that link such performance to the larger modality gap.
B.4 ADDITIONAL RESULTS
Figure 6 shows a visualization that demonstrates the effectiveness of the proposed method. Specif- ically, using four triplets from YouTube consisting of video, audio, and text, first compute the cosine similarity between text-video and text-audio pairs across all text labels and videos/audios. These results are arranged into
matrix, and a temperature-scaled softmax applied the rows to obtain probability distribution for each text-video (audio) pair assigned by the model.
Next, we apply our proposed GRAM approach to the same dataset, once again performing temperature-scaled softmax on the matrix rows. In this case, the volume matrix is negated, as lower volume indicates a stronger correspondence among text, video, and audio.
As Fig highlights, conventional cosine-based methods fail in jointly exploiting both audio and video modalities, often misleading when single modality does not contain enough information for the correct classification. Conversely, the GRAM-basee based model exploits the semantically aligned multimodal space and jointly leverages all the modalities, leading to a better and correct retrieval in every case.
Additionally, we report Recall@1 and Recall 10 for text-to-video and video-to-text both zero- shot and finetuned experiments in Table 8 Table 9 Table I0 and Table IT Our GRAM model outperforms every method the very large set of comparison methods in T2V and V2T both in zero-shot and fine-tuning scenarios. More interestingly, the proposed method always achieves very
19

---

Published as a conference paper at ICLR 2025
Label = "A dog barking"
Label "A dag hawling"
Video
Video
Label 3 "A red is meowing"
Label 4 "A black cat meowing"
Video
Video 4
Text-to-Video Retrieval
Text-to-Audio Retrieval
Video 3
Video
Video
Audio
Audio
Audio
Nodio
Cosine Similarity Based
Cosine Similanty Based
Label
0.42
Label
-088
Label
-06
Label
-0.6
-0.4
Label 3
0.00
0.00
0.88
0.12
-.2
Label
0.00
0.00
0.75
0.24
Multimodal Text-lo-Video Retrieval
Video & Audio
Video & Audio
Video 3 3
Video & Audio
GRAM (Ours)
Label
0.6
Label
0.41
0.55
0.02
0.02
0.5
0.4
Label
0.01
0.01
0.73
0.25
-02
Label
-0.1
Figure 6: Confusion matrices of cosine-based approach and our proposed GRAM.
20

---

Published as a conference paper at ICLR 2025
high scores of @@@ 10, especially the 100% in Table 10ladd Table 11 for the VATEX dataset, meaning that the space the model shapes is extremely aligned.
Table 8: Zero-shot multimodal text-to-vides retrieval results. Recall at and Recall at 10.
Zero-Shot 12V Retrieval
VTT
DiDeMo
ActivityNet
VATEX
Modality
R@1
R@
10
R@1
R@ 10
R@1
R@10
R@1
R@ 10
UMT Liu 2022
T-V
33.3
66.7
34.0
68.7
31.9
72.0
Omni Wang 2022a
T-V
42.0
73.0
40.6
74.3
UMT-11
2223
T-V
40.7
71.8
48.6
79.0
41.9
TVTSv Zeng 2023
T-V
38.2
73.2
34.6
71.5
ViCLIP Wang
T-V
42.4
18.4
15.1
VideoCoCa Yan 2022
T-V
34.3
67.0
34.5
76.6
53.2
Norton Lin
T-V
10.7
31.6
Imagebind Girdhar
2023
T-V
36.8
70.0
Intern Video-L Wang
2022b
T-V
40.7
31.5
30.7
49.5
HiTcA Yo 2022
T-V
34.4
69.9
43.2
79.0
mPLUG-2 2023
T-V
47.1
79.0
45.7
71.1
VideoPrismi- D
2024
T-V
51.4
49.6
62.5
LanguageBimd Zhu al.
2024
T-V
44.8
78.7
39.9
74.6
41.0
VAST al. 20236
T-VA
49.3
80.0
49.5
76.9
51.4
83.6
80.0
95.7
VAST Chen 20235
T-VAS
50.7
74.4
82.1
96.8
GRAM Model (Ours)
52.8
82.9
54.0
80.7
91.2
81.1
99.5
GRAM Model (Ours)
54.2
83,9
54.2
79.3
83.9
98.6
GRAM Model (Ours)
T-VAS
54.8
82.9
83.5
98.8
Table 9: Zero-shot multimodal video-to-text retrieval results. Recall and Recall 10.
Zero-Shot V2T Retrieval
MSR-VTT
DiDeMo
ActivityNet
VATEX
Modality
R@1
R@ 10
R@
R@ 10
R@1
R@
R@1
R@ 10
UMT Liu 2022
T-
33.3
66.7
34.0
68.7
31.9
72.0
Wang el 2022a
34.6
66.6
33.3
68.5
UMT-L er 2025
T-V
40.7
24.9
TVTSv Zeng cl al.
2023
T-V
38.2
73.2
ViCLIP Wang
2025
41.3
27.9
VideoCoCa Yan
2122
34.3
67.0
ImageBind Girdhar
2223
36.8
70.0
Intern Video-L Wang
2022b
T-V
39.6
33.5
HiTeA Ye 2022
T-V
46.8
56.5
VideoPrism- Zhao
2024
50.2
76.2
LanguageBind Zhu al.
2024
40.9
75.7
76.2
VAST Chen al. 20236
43.7
77.1
78.6
77.1
95.2
VAST Chen
20236
T-VAS
49.0
76.2
78.7
97.7
GRAM Model (Ours)
T-V
49.5
81.7
52.3
80.3
50.9
85.4
79.0
98.3
GRAM Model (Ours)
T-VA
50.5
82.2
52.2
78.9
50.4
85.8
79.2
99.0
GRAM Model (Ours)
T-VAS
52.9
82.9
82.7
98.1
21

---

Published as a conference paper at ICLR 2025
Table 10: Finetuning multimodal text-to-video retrieval results.
Finetuning T2V Retrieval
MSR-
DiDeMo
ActivityNet
VATEX
Modality
R@1 1
R 10
R@
R@ 10
R@1
R@10
R
R@
OmniVL Wang 2022a
T-V
47.8
83.8
52.4
85.4
UMT-L
(0233
T-V
58.8
87.1"
70.4"
93.5"
66.8*
94.9*
72.0"
ViCLIP
Wang ef al. (223
T-V
52.5
49.4
49.8
CIIIP4Cip Luo et 2121
T-V
45.6
81.6
43.0
80.6
40.3
63.0
Norton Lin ct al. 2024
T-V
31.2
66.8
Intern V Ideo-L Wang al. 2022b
T-V
55.2*
57.9
62.2"
71.1"
HiTeA Ye et. (202
T.V
46.8
81.9
56.5
89.7
mPLUG-Z(Xu et al. 2023
T-V
53.1
84.7
56.4
85.2
TEFAI Ibrahmm et al.
20233
T-VA
52.0
86.1
61.0
95.3
Bimodal T2M Arora of al 2024
T-VA
36.8
T-MASS Wang ct al. 2024a
T-VA
52.7
85.6
53.3
87.7
65.6
97.2
vid-TLDR et al 2024
T-
58.5
86.9*
70.4*
94.0"
65.2*
94.5*
CheT
Chen ct a. 2023c
T-V
55.8
85.9
65.6
88.1
68.8
95.5
86.9
99.1
VAST Chen et al. 2023c
T-VAS
56.6
79.4
87.5
99.5
GRAM Model (Ours)
TVV 55.7
86.4
66.4
89.9
66.5
96.0
84.4
99.8
GRAM Model (Ours)
T-VA
58.4
87.0
67.3
90.1
69.9
96.1
87.0
99.5
GR ARAM Model (Ours)
T-VAS
64.0
89.3
87.7
100.0
Finetuning and evaluation with 12 frames.
Table 11: Finetuning multimodal video-to text retrieval results.
Finetuning V2T Retrieval
MSR-VTT
DiDeMo
ActivityNet
VATEX
Modality
R@1
RØ
R@1
R@ 10
R@1
R@
R@I
10
UMT-L Li al.
2023
T-V
58.6
65.7"
64.6"
86.0"
CLIP4Clpp Luo 1202
T-V
45.9
43.6
41.6
78.3
ViCLIP Wang
T-V
51.8
50.2
48.1
VAST Chen el al. 20236
T-VA
87.4
62.0
87.7
66.7
95.3
84.1
VAST et al.
2023c
TVVA
57.6
80.2
84.0
99.7
GRAM Model (Ours)
T-V
56.4
87.6
63.2
91.6
64.6
95.1
81.6
99.8
GRAM Model (Ours)
T-VA
59.0
89.1
63.5
91.4
66.9
95.4
84.6
100.0
GRAM Model (Ours)
TAVAS
91.5
84.2
Finetuning and evaluation with 12 frames.
22


Reviews:
[{"review_id": "7QMqLUaBjz", "reviewer": "Reviewer_X14r", "summary": "This manuscript proposed a novel strategy for the multimodal representation alignment, providing more informative semantics for multiple modalities through geometric alignment. The proposed method leverages Gramian volume to replace the traditional cosine similarity to preserve rich semantic Relations between multiple modalities. Additionally, the authors proposed the GRAM-based contrastive loss to improve the alignment between each modality. Finally,  they also proved the Gramian volume can serve as quantitative metrics to measure the performance of multimodal alignment for downstream tasks.  The paper is well-organized and provide a novel insight of multimodal learning and alignemnt.", "questions": "1. The comparison on confusion matrices of cosine-based approach and proposed method should be exhibited in the former paper instead of in the supplementary materials. Also, in order to demonstrate the effectiveness of the proposed GRAM Multimodal Contrastive loss, it is important to conduct more ablation studies, comparing the vanilla contrastive loss. It is reasonable to conduct ablation studies including: (1) without proposed L_D2A and L_A2D, (2) without L_D2A, with L_A2D; (3) without L_A2D and with L_D2A; (4) without L_DAM .etc. on ActivityNet and VATEX  datasets. Also, it is more appropriate to exhibit the confusion matrix  in the supplimentary in the section 3.5 GRAM as Model Performance Metric to show the effectiveness of the proposed metrics. \n\n2. In the introduction, the author should more explicity explain the concept of anchor modality. If the author quote this concept from another articles, please cite the articles.  Additionally, it is more plausible to conduct more explicit mathematic conduction on  superiority of the Gramm Volum comparing to trainditonal cosine similarity in the supplimentary. In the Figure 4, it is also neccessary to implement T-SNE visualization using cosine similarity to mask the comparisons.\n\n3. Since the article proposed a new metrics,  it is neccessary to evalute other the-state-of-the-arts Methods (VAST, LanguageBind and VideoPrism-b and so on)  using the proposed metrcis.  \n\n4. In the table, the method's performances surpass the other the-state-of-the-art Methods significantly. In such case, the author should release the code link for the readers to validate after the publication or should provide anonymous link of the partial datasets in the table.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper demonstrates a novel method to better align heterogeneous modalities in higher-dimensional geometric spaces, which take the sematic relation between multiple modalities into consideration. \n\n2. It demonstrated that the proposed Gramian volume can be utlized as a novel metrics to evaluate the alignment performance of  the mulitmodal model. \n\n3. The performance of the model is outstanding than other state-of-the-art methods. \n\n4. The use of figures and tables in this paper to  illustrate the clear concepts and findings is effective.", "weaknesses": "1. The comparison on confusion matrices of cosine-based approach and proposed method should be exhibited in the former paper instead of in the supplementary materials. Also, in order to demonstrate the effectiveness of the proposed GRAM Multimodal Contrastive loss, it is important to conduct more ablation studies, comparing the vanilla contrastive loss. It is reasonable to conduct ablation studies including: (1) without proposed L_D2A and L_A2D, (2) without L_D2A, with L_A2D; (3) without L_A2D and with L_D2A; (4) without L_DAM .etc. on ActivityNet and VATEX  datasets. Also, it is more appropriate to exhibit the confusion matrix  in the supplimentary in the section 3.5 GRAM as Model Performance Metric to show the effectiveness of the proposed metrics. \n\n2. In the introduction, the author should more explicity explain the concept of anchor modality. If the author quote this concept from another articles, please cite the articles.  Additionally, it is more plausible to conduct more explicit mathematic conduction on  superiority of the Gramm Volum comparing to trainditonal cosine similarity in the supplimentary. In the Figure 4, it is also neccessary to implement T-SNE visualization using cosine similarity to mask the comparisons.\n\n3. Since the article proposed a new metrics,  it is neccessary to evalute other the-state-of-the-arts Methods (VAST, LanguageBind and VideoPrism-b and so on)  using the proposed metrcis.  \n\n4. In the table, the method's performances surpass the other the-state-of-the-art Methods significantly. In such case, the author should release the code link for the readers to validate after the publication or should provide anonymous link of the partial datasets in the table.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ixIe9V10Ts", "reviewer": "Reviewer_vQSX", "summary": "The paper addresses limitations in conventional pairwise contrastive learning methods used for multimodal data integration, where alignment is typically achieved by mapping multiple modalities to a single anchor (e.g., image or text). These methods often fail to ensure semantic consistency among non-anchor modalities. The authors introduce a novel measure, the Gramian Representation Alignment Measure (GRAM), which utilizes the volume of a parallelotope formed by modality vectors in a higher-dimensional space to achieve a holistic alignment across modalities. The GRAM-based approach not only improves semantic alignment but also enhances performance on various multimodal tasks.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1、Innovative Methodology: The introduction of GRAM as a higher-dimensional geometric measure provides a new approach to aligning multiple modalities simultaneously, addressing the limitations of traditional pairwise methods.\n\n\n2、Strong Empirical Results: The paper demonstrates substantial performance gains across multiple multimodal benchmarks, including video-audio-text retrieval and classification tasks, showcasing the effectiveness of the GRAM-based approach.", "weaknesses": "1. Does the volume complexity of GRAM computation increase significantly as the number of modalities increases? Especially in high dimensional space, does this calculation affect the real-time performance and scalability of the model? Has any consideration been given to approximate or optimize computations to reduce the computational burden?\n\n2.GRAM is based on the volume of k-dimensional parallel polyhedra built from multimodal vectors as the alignment metric. For sparse or dense distributions in high-dimensional Spaces, does this volume still accurately reflect the semantic relations between modalities? Is there an experimental analysis to verify the robustness of this approach under different data distributions?\n\n3.The paper mentions GRAM as an alternative to the traditional cosine similarity, but does this new alignment measure introduce instability or optimization difficulties during the actual training process? Have ablation experiments been performed to investigate the effect of GRAM on the convergence speed and stability of the model? In addition, are the effects of different learning rates and regularization strategies on model performance taken into account?\n\n4.Does GRAM still provide a significant advantage in simple modal alignment tasks, or does its performance gain mostly manifest in complex multimodal interactions? Are there specific experimental comparisons that demonstrate the trade-off between GRAM and cosine similarity? In other words, under what circumstances is it advisable to choose GRAM over cosine similarity", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sOJfUuafsM", "reviewer": "Reviewer_D5ua", "summary": "This paper proposes a novel Gramian Representation Alignment Measure (GRAM) for multimodal alignment. Compared with the methods based on pairwise cosine similarity, GRAM scales well with the number of modalities.", "questions": "In table 2, all models except for UMT-L and InternVideo-L use 8 frames, and they two use 12 frames? Why adopt a unified setting for a fair comparison?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The idea of taking advantage of determinant of the Gram matrix of the representation vectors of all modalities to measure their similarity is novel. Besides, the computational cost of the similarity calculation is low. \n\n2.The proposed method provides a new insight to understand the representation geometric of multimodal learning.", "weaknesses": "1.The Gram model in the experiment section is obtained from VAST pretaining models, which make it difficult to verify that Gramian measure is superior to cosine similarity in aligning modalities. The reviewer suggests the authors train the model from scratch for a fair comparison.\n\n2.Considering this Gramian measure is a new for modality alignment, the experimental analysis for it is not enough. The reviewer expects to observe how the Gramian similarity varies during training or fintuning, and compare it with cosine similarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pQJaghYotK", "reviewer": "Reviewer_iJhF", "summary": "This paper focuses on the problem where cosine similarity effectively represents the correlation between paired modalities in multimodal learning scenarios, but performs poorly with more than two modalities. It proposes a Gramian Representation Alignment Measure (GRAM) to quantify modality correlation by calculating the volume of a parallelotope. And then the traditional contrastive loss is improved with GRAM, which can hold for 2 to n modality and providing more meaningful alignment with respect to previous similarity measures. Extensive experiments validate the effectiveness of the method.", "questions": "see in Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper proposes a new volume-based modality alignment method, which sounds reasonable and overcomes the limitation of traditional contrastive learning that only computes between paired modalities. \n2. Comprehensive evaluation demonstrates the effectiveness of the proposed methods", "weaknesses": "1. Lack of the details about how to decide anchor modality for Eq. 5-7.\n2. When one modality is selected as anchor, minimizing the cosine similarity between the anchor and all other modalities on hypersphere can also minimize the volume of parallelotope. What is the difference between the two in terms of physical meaning? \n3. [A] demonstrated the modality gap from contrastive learning and pointed that varying the modality gap distance has a significant impact in improving the model's downstream zero-shot classification performance and fairness. How to explain this phenomenon by your measure? and is there modality gap from GRAM-based contrastive loss?\n4. The setup of the comparison experiment is not entirely fair. Since the proposed model is pretrained with the new loss and data set, the same should be done for the comparison methods. Meanwhile, ablation experiments for different losses ($L_{D2A}$ and $L_{DAM}$) are missing.\n\n\n\n[A] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. NeuIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)