QUEST: Quadruple Multimodal Contrastive Learning with Constraints and Self-Penalization
Authors: Qi Song
Tianxiang Gong
Shiqi Gao
Haoyi Zhou
Jianxin Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Contrastive Learning
Multi-View Learning
Multimodal Learning
Vision-Language Representation Degeneration
Primary area: deep_learning_architectures
TL;DR: We propose QUEST, a novel framework with quaternion objectives and constraints to both capture shared and unique information.

Abstract: Multimodal contrastive learning (MCL) has recently demonstrated significant success across various tasks. However, the existing MCL treats all negative samples equally and ignores the potential semantic association with positive samples, which limits the model's ability to achieve fine-grained alignment. In multi-view scenarios, MCL tends to prioritize shared information while neglecting modality-specific unique information across different views, leading to feature suppression and suboptimal performance in downstream tasks. To address these limitations, we propose a novel contrastive framework name *QUEST: Quadruple Multimodal Contrastive Learning with Constraints and Self-Penalization*. In the QUEST framework, we propose quaternion contrastive objectives and orthogonal constraints to extract sufficient unique information. Meanwhile, a shared information-guided penalization is introduced to ensure that shared information does not excessively influence the optimization of unique information. Our method leverages quaternion vector spaces to simultaneously optimize shared and unique information. Experiments on multiple datasets show that our method achieves superior performance in multimodal contrastive learning benchmarks. On public benchmark, our approach achieves state-of-the-art performance, and on synthetic shortcut datasets, we outperform existing baseline methods by an average of 97.95\% on the CLIP model.

QUEST: Quadruple Multimodal Contrastive Learning with Constraints and Self-Penalization
Qi Song'; Tianxiang Gong2,, Shiqi Gao?, Haoyi Zhou! Jianxin Li
School of Software, Beihang University School of Computer Science and Engineering, Beihang University ZZhongguanun Laboratory, Beijing
{songqi23, gongtx, gaoshiqi, hacyi, 1ijx))buaa.edii
Abstract
Multimodal contrastive learning (MCL) has recently demonstrated significant suc- cess across various tasks. However, the existing MCL treats all negative samples equally and ignores the potential semantic association with positive samples, which limits the model's ability to achieve fine-grained alignment. In multi-view scenar- ios, MCL tends to prioritize shared information while neglecting modality-specifff unique information across different views, leading feature suppression and sub- optimal performance in downstream tasks. To address these limitations, propose a novel contrastive framework named QUEST: Quadruple Multimodal Contrastive Learning with Constraints and Self- Seff-Penalization. In the QUEST framework, we propose quaternion contrastive objectives and orthogonal constraints to extract suf- ficient unique information. Meanwhile, shared information serimaatio-guudeed guided penalization is introduced to ensure that shared information does not excessively influence the optimization of unique information. Our method leverages quaternion vector spaces to simultaneously optimize shared and unique information. Experiments on multi- ple datasets show that method achieves superior performance in multimodal contrastive learning benchmarks. On public benchmark, our approach achieves fate-offthe-art performance, and on synthetic shortcut datasets, we outperform existing baseline methods by an average of 97.95% on the CLIP model.
1 Introduction
Multimodal Contrastive Learning (MCL) has demonstrated robust representation capabilities and generalizability and effectively transferring to various downstream tasks (e.g. cross-modal re- trieval 40
image captioning (37 72 73).. However, simply applying contrastive learning in multimodal scenarios presents significant challenges. In particular, contrastive learning treats all negative samples equally, ignoring the potential semantic relationships between negative samples and the anchor. 2 Besides, current contrastive learning methods focus on maximizing mutual information
between two views 688, 70 while ignoring unique information [41] In multi-view scenarios, the assumption that modalities share substantial task-related information often does not hold, especially in complex datasets with minimal inter-modal overlap. ① Meanwhile, recent studies [56] 75) in-
dicate that contrastive learning often neglects significant portions of input information, leading to feature suppression [ 8 and shortcut learning [25 (57), where models minimize loss through the simplest path (e.g, shared information [I), sacrificing deeper learning. These issues prevalent
multimodal 45 54
and multi-view tasks 76 82
43). Recent approaches focus on pre-
Equual contribution Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

125
Pull closer
High Vilues
Cac
Con
Orthogomelia
W:LAO 01
W-dodd 01 eg eocc
Metrics
(a) QUEST perfomance
(b) Quatemion Embedding Space
Figure I: (a) Our QUEST outperforms baselines 97.95% on average when trained with task-related unique information and evaluated downstream tasks on the CLIP model.
(b) We build the
quaternion emhedding space, which aligns shared and unique representations from different modalities through the application of constraints and self- seff-penalization. The Lsic narrows the gap between shared representations, while LP-UIC pulls the plane spanned by intra-modality shared and unique representations closer. Furthermore, Orthogonalization loss Coos employed constrain the area.
serving more unique information, including reconstruction regularization 777 4), implicit feature modification [58], and factorized representation [41], among others. However, these methods either overly introduce noise which may harm downstream tasks, or rely on certain assumptions (e.g.. find that these methods do not explicitly distinguish unique information and still optimize using contrastive learning. making it difficult to avoid the model explicitly extract both task-related unique and shared information without introducing too much noise?
augmentation 41 65 45 Additionally,
learning shortcuts (68] 22)] This raises the question:
To this end, proposes novel contrastive framework called QUEST: Quadruple Multimodal Contrastive Learning with Constraints and Self-Penalization, designed to enhance the extraction and integration of both shared and unique information across multimodal data. Our primary motivation to develop mechanism that effectively captures unique information through a novel quaternion multimodal embedding space, as illustrated in Figure [bb). This embedding space aims to pull shared representations closer while aligning the unique representations with the shared representation common plane. We achieve this by leveraging the properties of the normal vector from the cross-product to diversified unique representation. Consequently, our approach aligns commonalities across modalities while preserving the distinctive unique features of each modality.
Specifically, we first split network into three components: an encoder, a shared decoder, and unique decoder. The encoder learns general features with little bias toward specific tasks, while the shared decoder and unique decoder learn agreement and discriminative information, respectively. We build contrastive loss to constrain learning of shared information. To avoid the unique decoder degenerating into the shared decoder, we propose novel contrastive objectives and orthogonal constraints to optimize the quaternion vector space. Finally, self- penalization is used to prevent shared information from overly affecting quaternion vector space optimization. Our framework seeks to mitigate shortcut learning, offering more nuanced, task-related learning paradigm. Our main contributions be summarized as follows:
We develop novel framework efficiently extract shared and unique information across multi- modal data. To avoid the degeneration of the unique decoder, we propose an algorithm that utilizes quadruple embedding to constrain unique information from different views in plane space.
We consider that traditional CL overly relies on shared information due to data bias, causing failures with negative samples containing shared information related to the positive sample. Meanwhile, to prevent shared information from dominating the extraction of unique information, we introduce a gelf-penalization mechanism dynamically reweight the distribution of negative samples, which penalizes hard negative samples. We provide theoretical analysis to show how this penalization effectively improves the extraction of unique information.
We achieve state-of-the-arr on popular datasets (e.g. MS-COCO 9 and Flickr30k 80 compared the baseline, demonstrating the general effectiveness of QUEST. Additionally, experiment results on synthetic shortcut datasets outperform baselines 97.95% on average for CLIP, verifying the efficacy of QUEST.

---

2 Methods
2.1 Problem Formulation
For different modalities {M, given modality, denoted M₁, along with its corresponding of views { N; 2 for one modality Mi encoder parameterized by O represented as FM, (:; 11). and another modality M2 encoder parameterized by 22. denoted FM, (: θ2). These encoders process the sample from modal M and M2, for those modalities with multiple views, like modal one M₁ and each of its views through their respective encoder, resulting corresponding general representations H M, FM, (zf;; [=f; 0i) and H Ma FM.(al; θ2). However, as illustrated in Figure 2 the InfoNCE loss maximizes task-related features shared across all modalities during training (i.e. T(X'; 3 Y)). while simultaneously suppressing the unique task-related features of each individual modality(ie. T(Xa: Y|Xx) and T(XB; Y(Xx)). This process ultimately results in the loss unique information. Therefore, the general representations are then separated into task-related shared and unique features through different decoders, i.e., the representations of different modalities H HM, are inputted separately into different decoders GM, (-) parameterized by Φ; For the complete notations, refer to Appendix C Table
Task T Y
Task T Y
((X:YY|A))
I(XA:
I( x
Modality MA
Modality MB
Modality M
Modality Mr X B
XA
XB
X'A
T(XA; X;;
T(X XB::) Y)
Figure 2: Feature suppression in multi-view contrastive learning. We define T(XA; Xn; Y) as task- related shared information, T(Xx: Y|XB)) and T(X;; Y\X as task-related unique information related to task Y modalities X, and X respectciv Contrastive losses, such InfoNCE, tend to maximize the task-related shared information while suppressing the task-related unique information in each modality. Left: before training with InfoNCE. Right: after training with InfoNCE.
2.2 Overview
In multi-view scenarios, the relationships between different modalities are many-to- -onnnyy (c.g., for im- age retrieval, multiple captions can refer to the same image). Within single modality, different views contain task-related unique and shared information. Additionally, task-related shared information may also exist among negative samples (as indicated by the red shading FigureB). Therefore, optimizing solely for shared information while ignoring unique information suboptimal.
2c Lpayi
x
Orthogmaatoe
Orthogmmolin
z
Cew
Zy
Unique Decoder
Shard Desoder
Shored Decoder
Uniiqe Decoder
Encoder
Encoder
sample
To address the challenge of overlooking unique information inherent different perspectives in multimodal scenarios, we introduce the effective framework called QUEST: Quadruple Multimodal Contrastive Learning with Constraints and Self- Penalization. This framework extends existing con- trastive learning methods by incorporating four- partite architecture specifically designed to enhance the capture and integration of distinctive modal- specific features. The overall architecture is shown in Figure According to [84] 62 34 from the
Modal M.
Mi
somple
semples Shore Informaton
Rogubresanzio
Figure 3: Framework of QUEST. The unique decoder is utilized to extra view-specific unique information and this process is guided by the proposed constraints and penalization.

---

perspective of neural network architecture, shallow layers learn low-level and general features while deeper layers learn task-biased high-level semantic features. Therefore, we define the encoder output as H, which contains shared S and unique U information related to the task, H S U U. Con- sequently, we shared shallow layers for general representation reduce computational cost and optimize the shared decoder and unique decoder for S and U, respectively.
2.3 Quadruple InfoNCE
For each modality, the input data X; (sampled from views) undergoes transformation by a modality- specific encoder FM; (-), producing an intermediary general representation denoted as HM,, Conse- quently, we introduce two decoders: shared information decoder GM. (-) and unique information decoder GM, (-). These decoders are tasked with disentangling the shared and unique components from the representation H,,, respectively. Let θ represent the parameters of shared encoder, while 6; and V; symbolize the decoders for shared and unique information, respectively. The representation can be formulated as follows:
Z,u M. (H (HM, ②.)
Z = M., H (.)) = ()),;
(1)
O;); V.).
Shared Information Constraint. In multimodal and multi-view scenarios, data from different modalities and views often encapsulate shared information vital for model training. We conduct shared Information Constraint (SIC) to maximize the lower bound of MI between representations from different views to encourage the shared decoder to learn agreement related to the task. This constraint is optimized by computing the InfoNCE loss between (Zą and Z2), defined as,
exp(s(Z), 1/7)
Lsic EB Ezz
log exp(s(Z) 23/))// + Σx 19 (Z; 7/-))
(2)
Unique Information Constraint, In contrast to shared information, unique information modality- specific and task-related, providing essential insights for downstream tasks. To preserve this, introduce a Unique Information Constraint (UIC), extracting the unique information that exists within different views, which are unrelated to each other yet relevant to the task. Relying solely on Shared Information Constraint (SIC) is insufficient to preserve this information 41 3. Striet constraints, such directly enforcing consistency of distributions across diverse views, may lead to the Unique Information Constraint converging to the SIC, particularly in scenarios with identical input representations, loss functions, and network structures. To address this issue, we implement less stringent constraint, aiming to maximize the similarity between the spaces formed by unique and shared information across different modalities. Firstly, we derive the representation space of normal vectors for shared and unique embedding spaces through cross-product calculations,
Z Z x Z.
In the newly projected space, our objectives aim to maximize the alignment of unique representation from different modalities within the plane spanned by the shared representation, the magnitude can be formulated as:
Z x - (Z) bリ)
2 || sin | ||
sin cos
where sin a and sin 3 represent the sine similarity between the shared and unique representation across different modalities, and 7 represents the cosine similarity of the normal vector. We maximize sin a and sin 3 via orthogonalized cosine loss Lees and contrastive loss to maximize cos y, the unique information constraint can be formulated as:
exp(s Znd 3/T)
Luce
Σ
1
Ezz
log exp(>(Z)) 25+ 1/T) Σm 1y (xp(s(Z 7 )/T)
Zi
Z
(5)
Z

---

By pulling positive samples closer and pushing negative samples away, we constrain the shared and unique representations of different modalities coexist within the same spatial plane as much as practicable. Concurrently, we employ Less to maintain the diversity of unique information and ensure that Z # z It worth noting that we do not impose any constraints on the unique representations of different modalities, as these may be inherently unrelated.
Applying standard InfoNCE to extract unique information leads to suboptimal general- ization, as demonstrated in Section 3.3 To address this, we propose UIC with indirect vectors Z", where the cross- prosupr operation fundamentally alters gradient propagation patterns compared to
Comparison.
pair-wised InfoNCE. Specifically, given LInfONCE log E exp
exp(2,,27 /T)
70 83
the gradient
with respect to anchor embedding Zu as follows,
ILIINNCC 87.
1
3,Zn),,
(6)
where B. exp(2x-2./1
Ei- exp(Z.-2./3 and Z is random sampled from positive set ( ...Z... Under the assumption that different views hold both shared and unique information, we have I [[22; Y) [2,;; Y): + +(Z,; Ylz,), where I [22;; Za; Y) represents shared information between two views and
represents unique information for jth view. With sufficient training iterations, the unique information tends towards noise as shared information dominates the accumulated gradient
(first term in Eq. ( This is consistent with the conclusion of MI [41] Assume that the features obtained from the encoder consist of shared features S which correspond to the anchor and unique features U, represented as Z = (S U U$). Traditional contrastive learning defines an additive model z = (S+ whereas Zi, (S x U$) in our model. Intuitively, there exists Ç satisfes < < Z, (S x )²) =... Zu (S x U?). Therefore, UIC is weaker constraint that ensures quaternion vectors between different views lie on the same plane as much as possible. we use both SIC and UIC simultancously, SIC will pull the shared representations of different views closer, while UIC will ensure that the unique representations of different views lie on the same plane as much as possible, rather than measuring their cosine similarity, as unique information is uncorrelated. We conducted extensive experiments to verify this (Section 3.3 for more details).
2.4 Shared Information Guided Constraint
Contrastive learning fundamentally operates by optimizing vector representations to minimize dis- tances between positive pairs while maximizing distances between negative pairs in the embedding space. However, critical limitation inherent in conventional approaches stems from their undif- ferentiated treatment of all samples within a batch B as negative examples. This indiscriminate categorization may inadvertently cause the model to overlook potential semantic relationships (as illustrated by the red shading in Figure 3. despite their shared semantic content (e.g.. different image captions containing identical substrings in image-text retrieval tasks). Such misclassification significantly impairs the model's 's representation capacity. We refer to these cases as "hard negative samples. While existing methods attempt address this issue through clustering- -based approaches, their two-stage nature limits practical adoption. Leveraging our dual-branch architecture, we propose more effective solution that directly utilizes the output of the shared decoder as a supervision signal for the penalty term.
Considering our objectives is to optimize the shared information decoder through Lsic and unique information decoder through LuIc, and the shared information also affects the optimization of unique information shown in Eq. (4). attempt to the intra-model shared information similar to
penalization to guide the optimization of unique information. Unlike soft label [49] 17 16
47
60 24]
which aim to mitigate the strict constraints of one-hot labels, preventing overconfidence by retaining more potential positive samples, our method aims to impose stricter constraints to suppress shared information in the process of learning unique information. Specifically, the more shared information between the anchor and all the negative samples, the greater the encouragement in learning unique information between the anchor and the positive sample. Formally, for shared representation Zł and Zj. the weighted similarity matrix can be formulated as:
P exp(A|S - diag(S) + I)),
(7)
5

---

where S - " is the similarity matrix, is identity matrix, À E IR is learnable weight param- eter. Next, the weighted similarity matrix P E RN*N is utilized penalization to supervise the optimization of unique information satisfied Pisaat oc s(2f, 2J), the penalized UIC can defined
as:
exp(s(Z,). 23 )/T)
CrAic Σ
M,NM
Bzy
log
exp(s(Z,, 1/7) Σi= 11 exp(Pk s(Z 23x 1/r)
(8)
Σ Lex.
Gradient Analysis. For simplicity, set À Σm exp(Px ( Zzz)/T) and ignore the second term, we reformulate Eq.
exp(P* s(Z )/T)
Ezy
log
exp(Pt s(Z.. 25 1/T) + Σ y exp(Px (Z Zia 1/T)
(9)
pt s(Z?,
Ezz
log A
The gradient can be calculated as (Appendix D.2 for details):
aCpuIc
p
0st
Pas*
0s
Ps exp
(10)
87.
k=
where s+ "Z, and * = sZ, Z
Intuitively, P represents the belief mass based on shared information, and the larger Ps indicates more shared information between hard positive samples, which also influences the optimization of the unique decoder as in Eq. { When Ps = 0 (i.e, sZZ, 2 0), will degenerate to original InfoNCE loss. For hard negatives samples hold shared information where s (Z, Z > O, we increase the gradient using penalty term which ensures that even if Z relatively large in Eq. 4 the lower loss reinforce other terms to constrain the overall value. From mutual information perspectives (Appendix D.3 details), we have
T(z 2,) 2 H (Z (2, log N -EPANC
(11)
When all negative samples h satisfy 8 (Z Zzk) R O, we obtain H (2,lZz) H(Z))Z)). Subse- quently, as the shared information between hard positive samples increases, H (Z, correspond- ingly increases, thereby elevating both the lower bound of mutual information and the confidence level, which consequently facilitates the learning unique information.
2.5 Training Objectives
In summary, we apply (1) SIC (Eq. 2 to keep shared information relevance between different modalities, (2) UIC (Eq. 5 build quadruple embedding space by maximizing the alignment of normal vector spanned by shared representation and unique representation, and the shared information jointly optimized by SIC and UIC, (3) penlleenalizttion (Eq. 7 amplify the effect of false positive samples in unique information optimization. The overall objective can be formulated as:
LQUEST Lsic + LPAIC.
(12)
Experiment
3.1 Experiment Setup
Baselines and Setup. Shorteut learning refers to the process in deep learning model training where the model completes tasks (such classification, retrieval, etc.) by learning simple and discriminatory features while ignoring the semantic and more complex features of the data. This result in poor model performance downstream tasks. Latent target decoding (LTD) [].

---

and implicit feature modification (IFM) 58 are two methods that mitigate shorteut learning by reducing feature suppression. LTD reconstructs the caption representations the latent space of Sentence-BEie model, allowing the encoder to mitigate feature suppression via correct mapping. IFM perturbs diseriminatory features through encoders and removes part of these features to avoid learning shortcuts, which is implemented as dual loss combined with the InfoNCE loss. We provide source code of our paper.
Image Caption RII)) retrieves the most relevant sample in another modality by using a sample of one modality as a query. In this task, there are two retrieval modes: text-to-image (t2i)
and image-to-text (i2t). We evaluated our method in the ICR task using CLIP [52 and VSE++ 23
models on Flickr30k 80) and MS-COCO [42] datasets.
Bleeker et al. 3 proposed synthetic shortcuts for the vision-language framework. This allows us
to evaluate whether vision language (VL.) models capture easy-to-learn discriminatory features or task-related information. We add MNIST Images to the top of pictures and appending corresponding numbers at the end of their respective captions. This controlled approach preserves the original information from both modalities and increases explicit mutual information between them.
Evaluation Metrics. To evaluate the model's performance on the Flickr30k and MS-COCO-Caption datasets, the Recall (i.e. R@I, 1, R, R@ 10, which refers to the proportion of instances where the correct answer appears among the top K returned results out of all instances) and recall sum (RSUM) were selected as evaluation metrics for both i2t and 12i retrieval.
Implementation Details. We select ViT-B/32 as the visual backbone for CLIP and resnet152 for the VSE++ model when we evaluate them on the caption retrieval task using Flickr30k and MS-COCO dataset. We fine-tuned the pre-trained CLIP on downstream tasks and trained VSE++ from scratch with our method.
The key of multi-view assumption lies in the extraction of unique information, current methods achieve this by employing a single-layer MLP, ensuring orthogonality with shared information or through data augmentation and factorized loss. In our framework, single-layer MLP is used for the ResNet backbone, while two-layer Transformer implemented for the Transformer backbone. Detailed methodologies are provided in the Appendix B.2
A Alternatives of Unique Decoder.
3.2 Performance Evaluation
QUEST vs. Vanilla InfoNCE. QUEST outperforms the vanilla InfoNCE, as shown in Table [ On the Flickr30k test set, QUEST yields R@l improvements of (2.4, 1.5) for CLIP and (1.1, 3.4) for VSE++ in i2t and t2i tasks.. The corresponding RSUM metrics increase by 3.2 and 12.1. On MS-COCO, QUEST achieves R@1 gains of (1.6, 0.9) for CLIP and (3.1, 3.2) for VSE++ in i2t and t2i tasks, with corresponding RSUM improvements of 8.1 and 17.3. Additionaly, QUEST exhibits faster convergence the optimal solution.
Experiment on Synthetic Shortcuts. To assess the effectiveness of our proposed QUEST mitigating feature suppression in Contrastive Learning, we use synthetic shortcuts 33 by injecting easy-to-learn and discriminatory shared information into the image-text training dataset. We then evaluate the model's 'ssei performance on downstream tasks with and without these synthetic shortcuts to determine if the presence of shortcuts causes the suppression of other task-related information and an over-reliance on shorteut features. Our baselines' results consistent with
As shown in Table ! adding shortcuts leads performance degradation across all models some degree, indicating that the models have not learned sufficient shared and unique information. However, our method outperforms LTD and IFM, indicating captures rask-related information more effectively in downstream tasks (evaluation without shortcuts).
CLIP Performance Enhancement. Our method significantly enhances the CLIP model's perfor- mance on the Flickr30k and MS-COCO datasets. Compared to InfoNCE, we observe substantial @ improvements in both i2t and t2i tasks, with RSUM increases of 91.6 and 240 on Flickr30k and MS-COCO, respectively. Our approach also outperforms previous SOTA methods, surpassing CISAANCE+IM on Flickr30k and showing sinificant performance improvements against CIaENNC on MS-COCO.
7

---

Table 1: Result Flickr30k and MS- IS-COOO with varied method. se denotes shortcut, we evaluate CLIP and VSE++ w/wo shortcut on i2t and 12i task. QUEST outperforms InfoNCE and achieve superior performance compare with other baselines in most cases. ?denote use of Itd.
MS-COCO
RSUM
RSUM
R81
RG5
RG910
R&1
R895
Gusars
Baca
96.0.30
72.
95.-01
058-03
96.1
023-03
16.5as
74..a1
81.1en
5-65
97.
1:13
58.5230
724kma
92.3gz0
959
542.8
63.5mm
90.130
923gu
46.3.3
74.720
84.1asn
447 447.4
Gusecce-1M4
57.4a51
M7 Aund
90.1asn
73.2ns
32.2100
60-6un
544.9un3
630
96.0a11
926
47.2a50
7566na
84.5as
449 5.1.5
789..s
D1.54ea
950-03
546.7.11
65.4.6
87.7.38
93
48.5.38
757..0
455.6.011
67.2
84.0.ss
84:2.a5
416.2
136.09
422--7
22.410
32.7,17
149.4.077
61.0g13
Sau
83.2a38
7aus
70.8
88 1u1
463.6 2377
188
41. 41.8
56 24
16.5gmm
3944g0
52.0aaa
73 20.8
01 S:13
05.6a00
58.9mm
S1.4
01 01.1
658. 2:
25.4a1a
06.
06.5227
58.2
17.1
38.
51.3210
205 Amas
EQUESTIOLM)
84.2
97.7a02
88.9
934
$27.8.14
50.8.m
7544244
24.
37.9
389.4.21
VSE++
52, B2:6
87.8as1
30.5uu
00.8esm
794
000.3.as
42.
72.7a5.
652
30.0sam
612
70.5aa1
363 8.2.
M4.1ata
81.
42.5mn
T1.Ma5a
81.3un
41M.8u1
43.0m
837am
32-4433
625
70.7as0
370
GuSeCE-UM
52 53-4ato
76.9.:
85.3a50
39.ann
88.8461
78.2.s:
800.7.02
4022m0
70.8
81 6..
50.8a50
61.8a00
74.3aco
350 3.1.
Eounstionn
54.7ata
81.3.01
8888.s
42.9mm
72.3
816-1.
421.6-25
483-m1
75.5es
85,4-04
34.1-31
645.az
76,34s
381.1115
Gusecc
C.lato
0.4.33
0.8as0
0.41c0
10.ns
2.uua
6.1a50
02.03
0.0as0
0.100
0.2a00
06.00
51.8-07
65.6.14
20.7-1
49.2c0
626-13
274.6-11
39.0a
13.7450
216-03
3.449
11.Gra
18.1430
714-a4
Susecz-a1
0.
0.9432
0.5gem
1.0au3
3.2
0.1g00
02a0
0.0gam
012m0
0.2200
0.7au
EQUTSTIOUn
24.
48.
GL.115
1755m3
43.41e6
86.5ns
251
10.5
27.9
40.6.03
9.4.55
29.0.14
42.6111
160.0.s
Comparative Analysis of VSE++ Improvements. VSE++ also shows improvements on both datasets, with increases in R&1 for i2t and 12i tasks compared to LIMONNE. On MS- VSE++ outperforms LINGNNCEIIMM enhancing R&1 and rsum metries. However, LILINCEE-LTD achieves better performance on Flickr30k, excelling in text modality reconstruction and enhancement, This discrepancy may be attributed to VSE++'s GRU text encoder, which appears to be less effective in capturing modality-indeeeedent information on smaller datasets like Flickr30k.
3.3 Ablation Study
We also evaluate the effects of two constraints (shared information constraint Lsic and unique information constraint Luic) and self-penalization on the performance of our proposed QUEST method. These variants, denoted as Lsic. LvIc, LP.UIC, are evaluated through modality-specific decoder objectives in our ablation studies. The evaluation is conducted on the ICR task, comparing the performance differences between CLIP and VSE++ models on the Flickr30k and MS-COCO datasets.
Decoder Configuration and Objective Function Assignment. As shown in Table 2 both Lsic and Luic individually exhibit performance drops compared to the full QUEST. CLIP and VSE++ perform poorly when only using unique information constraints, achieving 55.1 and 40.7 on COCO and 80.8 and 47.8 on Flickr30k, respectively, suggesting that capturing unique information alone is insufficient for improvement. Furthermore, the results demonstrate that self- seff-penalization enhances the performance of QUEST, preventing overconfidence and retaining more potential positive samples. CLIP and VSE++ have better performance when using LrAnC instead of Lune. In addition, our QUEST method outperforms the approach utilizing LSICHUIc. The ablation study aligns with our theoretical framework, confirming that simultaneously capturing shared and unique information in multi-view contrastive learning, along with the application of self-penalization, leads to significant improvements in downstream tasks.
Unique Decoder Degeneration. Our architecture includes a unique decoder module. To evaluate its impact the task, replaced this unique decoder with shared decoder to determine if performance improvements were due to parameter changes. As shown in Table 2 using dual shared decoders (Lsic+sic) did not enhance performance and even resulted in decline compared to single shared decoder (Lsic). We refer to this decline as the degradation of the unique decoder.
3.4 Evaluation on More Modalities
We conducted extensive experiments on three modalities: image, text and audio. For image-audio evaluations, we leveraged the FMA [19 and GTZAN [69] datasets, while text-audio experiments were evaluated on CLOTHO
21 and AUDIOCAPS
[36] datasets. As shown in Figure 4 pro-
8

---

Table 2: Ablation study image caption retrieval task with different training objectives. DI and D2 denote decoders the architecture. Decoder with all Xbeneath are omitted, while those with indicatee optimization with corresponding objective functions. Bold and underlined numbers indicate the best and second-best results, respectively.
Methods
Flickr30k
MS-COCO
RSUM
RSUM
Cace
Csce
Cupe
EpunC
Rolt
R1
Ri11
R8010
RX01
RX0G
RX01
Ri81
RERG
8810
CLIP
CLIIP
92.3
X
X
/
X
90.8
34.4
96.5
66.9
88.4
02.0
519.9
55.1
81.6
50.4
-3.33
72.5
82.9
424.8
/
98.1
57.33
90.7
628.2
85.9
X
$1.9
96.2
98.0
00.3
9000
94.2
1.29
01.9
S8.6
92.8
47.4
25.1
$3.9
X
76.6
95.9
59.0
S1.4
90.9
54.4
79.2
86.7
39.9
08.
78.9
407.5
X
X
89.3
97.8
90.2
91.5
05.0
546.7
65.4
87.7
93.6
48.5
75.7
84.7
455.8
VSE++
VSE++
X
X
X
12.6
01.8
19.4
618.9
72.7
85.2
30.9
73.5
363.7
47.8
72.8
80.8
36.7
63.2
13.3
371.6
40.7
71.2
82.1
30.3
60.5
73.0
357.8
X
69.0
74.1
81.3
36.4
64.1
75.1
578.0
71.4
82.4
30.8
60.6
73.2
359.3
53.3
70.8
87.6
68.1
78.0
74.1
94.4
32.3
62.8
74.7
373.2
54.7
80.3
68.2
42.0
70.3
79.6
415.1
45.5
75.5
85.4
34.1
64.5
76.3
381.1
PSA-IR -Be
GTZAN-BA
OTZAN 42
01.5
190
66-
220
80
20
345
148
Figure 4: Performance comparison of InfoNCE and QUEST methods with additional audio modality on image-to-audio (i2a) and audio-to-image (a2i) retrieval tasks across FMA, GTZAN, CLOTHO, and AUDIOCAPS datasets.
posed methods outperformed the baseline. For simplicity, employed the almost simplest decoder structure (linear layers) and did not implement modality fusion or any cross-modal interactions. We believe that enhancing the architecture, strengthening cross-modal interactions, employing larger batch size and incorporating pre-training would yield even higher performance.
3.5
Case Study
On Flickr30k, CIsANNCE over-privileges shared information, particularly for synthetic shortcut data, while LOUEST emphasizes task-relevant semantic information. In the text-to-image retrieval case, LIsGaNCE leverages modality-shan shared information like "gold" and "bicycle" for retrieval. When shared information explicitly injected across modalities through shortcuts, LIstONCE pays less attention to the semantic information in queries.
4 Related Work
4.1 Multimodal Contrastive Learning
In the field of multimodal contrastive learning, substantial progress has been achieved harnessing the synergistic effects of multimodal data [38 46 I1, [4). Multimodal contrastive learning mainly concentrates on developing methods to semantically align data from diverse modalities through contrastive learning techniques, including optimizing contrastive objectives to enhance agreement
between paired data [52 33 79 15), improving the selection of positive and negative sample [85
35 64), optimize training mechanism
28 12 and designing innovative network architectures

---

Query
Quary
Top
white shires and proson colling
(Cjnn the middle a shopping corte
there
puinted
with gold bicycle.
white shit kneeing
C man drevsed in gold with his look the city. CAA mun wearing gold
black the Blxt
old bike
w hile standlnn with his pald bike.
aoon in whrte hor looking
the cancraa of
CAA painted in gold with cycle
stands sucet.
and stands arzinee the backsron
jeckst wenen spran
A man in gold oost loes gold
a colortally painted wall.
icycle the
shetr hour the 3344
C.A. dark haired woman in green
and white apron sis amid her out
door dispbry flowers sale
man on miceclee nerbomss
aa
4
crowd the 1934
A womus sitting by dhied
lover dipllyy ostsice mark
and jear jumping oatshors with leaos on the ground 02 022622222 26244 o
Thoee kide wearinng beosae shirts
A womnn sclls Dowers and insm 9 194518
a. 49455 18
C.A lady sirng down conding
The dressed white one with Back punts, are doins kans 022624
her steal
A womas in apron shopping a
market. 9 994518
(b)
Figure 5: Case Study: (a) Image-to-text retrieval, where the results of LOUEST and LIntONCE are denoted by italies and underlines, respectively. (b) Text-to-image retrieval, where red and green borders indicate the top-5 retrievals using LQUEST: while blue borders represent those using LIMONNCE- The upper and lower sections in both (a) and (b) demonstrate scenarios with and without shortcuts, respectively.
6 26 10 Nevertheless, our work is orthogonal to most of the aforementioned studies. Our shared branch approach can be integrated with existing training methods such as MoCo (28) and SimCLR
7
4.2
Shortcuts Learning
Shortcut Leanning refers the tendency of deep neural networks to exploit simple but potentially unreliable features (i.c.. "shortcuts") in data for prcision-making rather than learning more complex reliable features (25). This phenomenon can lead to poor model performance on -fist--diittiuutio
data and is particularly common in multimodal retrieval [3] and VQA tasks [55] [8]. Robinson et
al. [38] proposed strategically adjusting the feature distribution of positive and negative sample pairs to achieve implicit feature modification in contrastive learning, guiding models to learn more robust feature representations. Sanchez al. (59) propose maximizing mutual information to capture data attributes in shared and exclusive representations, while minimizing it between them to enforce disentanglement. LTD (4) introduces an additional decoder to reconstruct input text descriptions in the latent space of a universal sentence encoder, preventing image and text encoders from suppressing predictive features. More recently, some works strive to enhance the estimation of mutual information through the utilization of stricter bounds [30] 41 33, 51 or the introduction of regularization constraints 45 consequently preserving unique information more effectively.
Conclusion
We introduce QUEST, a framework utilizing specialized decoders to extract both unique and shared information via shared information-guidde constraints and self- pelf-penalization. This study addresses the challenges of imbalanced negative samples and task-related unique feature suppression Multimodal Contrastive Learning. Our method optimizes shared and unigue representations simultaneously. outperforming ofate-of-the-arr methods in preserving unique information and enhancing contrastive learning. Unlike traditional approaches that employ direct dot products to minimize distances between positive samples, QUEST leverages quaternions for the indirect optimization of unique and shared information. However, the application of cross products in high-dimensional spaces is limited, complicating the control of high-dimensional representations and reducing theoretical interpretability. For further discussion on these limitations, see the appendix (Appendix E)
10

---

Acknowledgements
This work was supported by the grants from the Natural Science Foundation of China (62202029), and Young Elite Scientists Sponsorship Program by CAST (No. 2023QNRC001), Thanks for the computing infrastructure provided by Beijing Advanced Innovation Center for Big Data and Brain Computing. This work was also sponsored by CAAI-Huawei MindSpore Open Fund. Haoyi Zhou is the corresponding author.

---

References
[1] Mido Assran, Randall Balestriero, Quentin Duval, Florian Bordes, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael G. Rabbat, and Nicolas Ballas. The hidden uniform cluster prior in self-supervised learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[2] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing
mutual information across views. Advances in Neural Information Processing Systems, 32, 2019.
[3] Maurits Bleeker, Mariya Hendriksen, Andrew Yates, and Maarten de Rijke. Demonstrating and reducing shortcuts in vision-language representation learning. Transactions on Machine Learning Research, 2024. [4] Maurits J. R. Bleeker, Andrew Yates, and Maarten de Rijke. Reducing predictive feature suppression in resource-constraingg constrained contrastive image-caption retrieval. Trans, Mach. Learn. Res., 2023, 2023.
[5] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency, pages 77-91. PMLR, 2018.
[6] Krishna Chaitanya, Frtune Erdil, Neerav Karani, and Ender Konukoglu. Contrastive learning of global and local fcatures for medical image segmentation with limited annotations. Advances in Neural Information Processing Systems, 33:12546-1253
2020.
[7] Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages
1597-1607. PMLR, 2020.
[8] Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Syxtems, 34:11834-11 1834-11845, 2021.
[9] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and Data collection and evaluation server. CoRR,
C. Lawrence Zitnick. Microsoft COCO
captions:
abs/1504.00325, 2015.
[10] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750-155 15758, 2021.
[11] Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel Salz, Schastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigeerver, Nan Ding, Keran Rong, Hassan Akhari, Gauray Mishra, Linting Xue, Ashish V. Thapliyal, James Bradbury, and Weicheng Kuo. Pali: A jointly-saaldd multilingual language-image model. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali. Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[12] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pages 9640-9649, 2021.
[13] Pengyu Cheng. Weituo Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In International Conference on Machine Learning, pages
1779-1788. PMLR, 2020.
[14] Alexandra Chouldechova and Aaron Roth. The frontiers of fairness in machine learning. CoRR,
abs/1810.08810, 2018
[15] Ching- Yao Chuang. R Devon Hjelm, Xin Wang. Vibhav Vineet, Neel Joshi, Antonio Torralba, Stefanie Jegelka, and Yale Song. Robust contrastive learning against noisy views. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16670-16681, 2022.
[16] Sanghyuk Chun, Wonjae Kim, Song Park, Minsuk Chang, and Seong Joon Oh. Eeev caption: Correcting
false negatives by collecting European Conference on Computer Vision, pages 19 Springer, 2022.
image-caption associations for ms-coco. In
[17] Sanghyuk Chun, Scong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8415-8424, 2021.
[18] Corentin Dancette, Remi Cadene, Damien Teney, and Matthieu Cord. Beyond question-based biases: Assessing multimodal shortcut learning in visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1574-1583, 2021.
[19] Michaël Defferrard, Kirell Benzi, Pierre Vandergheynst, and Xavier Bresson. FMA: A dataset for music analysis. In Sally Jo Cunningham, Zhiyao Duan, Xiao Hu, and Douglas Turnbull, editors, Proceedings of the 18th International Society for Music Information Retrieval Conference, ISMIR 2017, Suzhou, China, October 23-27, 2017, pages
2017.
[20] Michaël Defferaard Sharada P. Mohanty, Sean F. Carroll, and Marcel Salathé. Learning to recognize musical genre from audio: Challenge overview. In Pierre-Antoine Champin. Fabien Gandon. Mounia Lalmas, and Panagiotis G. Ipeirotis, editors, Companion of the The Web Conference 2018 on The Web Conference 2018, WWW 2018, Lyon. France, April 23-27, 2018, pages 1921-1922. ACM, 2018 [21] Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: an audio captioning datasct. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pages 736-740. IEEE, 2020.
[22] Andre Esteva, Alexandre Robicquet, Bharath Ramsundar, Volodymyr Kuleshov, Mark DePristo, Katherine Chou, Claire Cui, Greg Corradoo Sebastian Thrun, and Jeff Dean. A guide to deep learning in healthcare.
12

---

Nature medicine. 25(1):24-29, 2019.
[23] Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler. VSE++: improving visual-semantic embeddings with hard negatives. In British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK. September 3-6, 2018, page 12. BMVA Press, 201.
[24] Chen Feng and loannis Patras. Adaptive soft contrastive learning. In 2022 26th International Conference on Pattern Recognition (ICPR), pages 2721-2727. IEEE, 2022.
[25] Robert Geirhos, JömnHenrik Jacobsen, Claudio Michaclis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):655-673 2020.
[26] lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems. 33:21271-21284, 2020.
[27] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in Neural Information Processing Systems, 29. 2016.
[28] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 72999738, 2020.
[29] Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems. 33:9995-10000 2020. [30] R. Devon Hjelm, Alex Fedorov, Samuel Lavoie- aavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler. and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA. USA, May 6-9, 2019. OpenReview.ne., 2019.
[31] Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. Improving fairness in machine learning systems: What do industry practitioners need? In Proceedings of the 2019
CHI conference on human factors in computing systems, pages 1-16, 2019.
[32] Keli Huang. Botian Shi. Xiang Li, Xin Li. Siyuan Huang, and Yikang Li. Multi-modal sensor fusion for auto driving perception: A survey. CoRR, abs/2202.00 02000,022333222 2022.
[33] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference
Machine Learning, pages 4904-4916. PMLR, 2021.
[34] Longlong Jing and Yingli Tian. Self- superviuee visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1 1):403744058, 2020.
[35] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative
mixing for contrastive learning. Advances in Neural Information Processing Systems, 33:21798-21888 2020.
[36] Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In NAACL-HIT, 2019.
[37]
Chenliang Li. Haiyang Xu, Junfeng Tian, Wei Wang. Ming Yan, Bin Bi, Jiabo Ye, He Chen, Guohai Xu, Zheng Cao. Ji Zhang, Songfang Huang, Fei Huang. Jingren Zhou, and Luo Si. mplug: Effective and efficient vision-
lcarning by cross-modal skip-connections. In Yoav Goldberg, Zomitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 7241-7259. Association for Computational Linguistics, 2022.
[38] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language -aaagge--aaeegpae pre- wre-training with frozen image encoders and large language models. In International Conference on Machine Learning, pages 19730- 19742. PMLR, 2023.
[39] Junnan Li, Ramprasaath Selvaraju. Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances
in Neural Information Processing Systems, 34:9694-970s 2021.
[40] Xiujun Li, Xi Yin, Chunyuan Li, Pengehuan Zhang, Xiaowei Hu, Lei Zhang. Lijuan Wang. Houdong Hu, Li Dong. Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16. pages 121-137. Springer, 2020.
[41] Paul Pu Liang. Zihao Deng, Martin Q Ma, James Y Zou, Louis Phil-Philippe Morency, and Ruslan Salakhut- dinov. Factorized contrastive learning: Going beyond multi-view redundancy. Advances in Neural
Information Processing Systems, 36, 2024.
[42] Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects context. David J. Fleet, Tomás Pajdla, Bernt Schicle, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pages 740-755 Springer, 2014.
[43] Yijie Lin, Yuanbiao Gou, Xiaotian Liu, Jinfeng Bai, Jiancheng L.,, and Xi Peng. Dual contrastive prediction for incomplete multi-view representation learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4447 2022.
13

---

[44] Chang Liu, Henghui Ding. and Xudong Jiang. Gres: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 23592-23601,
2023.
[45] Shengzhong Liu, Tomoyoshi Kimura, Dongxin Liu, Ruijie Wang. Jinyang Li, Suhas Diggavi, Mani Srivastava, and Tarek Abdelzaher. Focal: Contrastive learning for multimodal time-series sensing signals in factorized orthogonal latent space. Advances in Neural Information Procesing Systems, 36, 2024.
[46] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task aask-agnotii visiolinguistic representations for vision-and-language tasks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buee Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 13-23, 2019.
[47] Jie Ma, Chuan Wang. Yang Liu, Liang Lin, and Guanbin Li. Enhanced soft label for supi-super semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision,
pages 11855-1995 2023.
[48] Ninarch Mchrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairesss in machine learning. ACM Computing Surveys (CSUR). 54(6):1-35, 2021.
[49] Zarana Parckh, Jason Baldridge, Daniel Cer, Austin Waters, and Yinfci Yang. Crisscrossed captions: Extended intramodal and intermodal semantic similarity judgments for MS-COCO. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL. 2021, Online, April 19 23, 2021, pages 2855-2870. Association for Computational Linguistics, 2021.
[50] Petra Poklukar. Miguel Vasco, Hang Yin, Francisco S Melo, Ana Paiva, and Danica Kragic. Geometric multimodal contrastive representation learning. In International Conference on Machine Learning, pages 17782-17800. PMLR, 2022.
[S1] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Leurning, pages 5171-5180. PMLR, 2019. [52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMI.R 2021.
[53] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International Conference Machine Learning, pages 8748-888 144888763. PMLR,
2021.
[54] Valentin Radu, Nicholas D Lane, Sourav Bhattacharya, Cecilia Mascolo, Mahesh K. Marina, and Fahim Kawsar. Towards multimodal deep learning for activity recognition on mobile devices. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. pages 185-188, 2016.
[55]
Ishaan Singh Rawal, Alexander Matyasko, Shantanu Jaiswal, Basura Fernando, and Cheston Tan, Dissect- ing multimodality in videoqa transformer models by impairing modality fusion. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
[56] Joshua Robinson, Li Sun, Ke Yu. Kayhan Batmanghelich, Stefanic Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances in Neural Information Processing Systems, 34:4974-4986, 2021.
[57] Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances in Neural Information Processing Systems, 34:4974-4988
2021.
[58] Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? In Marc' Aurelio Ranzato, Alina Beygelzimer. Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 1974-4986. 2021.
[59] Eduardo Hugo Sanchez, Mathieu Serrurier, and Mathias Ortner. Learning disentangled representations via mutual information estimation. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part XXII 16, pages 205-221. Springer, 2020.
[60] Yuanjun Shi, Linzhi Wu, and Minglai Shao. Adaptive end-to-end metric learning for zero-shot cross- domain slot filling. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6291-6301, 2023.
[61] Benjamin Shickel, Patrick James Tighe, Azra Bihorac, and Parisa Rashidi. Deep ehr: survey of recent advances in deep learning techniques for electronic health record (ehr) analysis. IEEE Journal of Biomedical and Health Informatics, 22( 1604, 2017.
[62] Bharat Singh, Soham De. Yangmuzi Zhang, Thomas Goldstein, and Gavin Taylor. Layer-specific adaptive learning rates for deep networks. In 2015 IEEE 14th International Conference on Muchine Learning and Applicutions (ICMLA), pages 364-368. IEEE, 2015.
[63] Karthik Sridharan and Sham M. Kakade. An information theoretic framework for multi-view learning, In Rocco A. Servedio and Tong Zhang, editors, 2/ Annual Conference on Learning Theory- COLT 2008,
14

---

Helsinki, Finland, July 9.12, 2008, pages 403-414, Omnipress, 2008
[64]
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XI 16, pages 776-794. Springer, 2020.
[65]
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827-6833 2020.
[66]
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. In Algorithmic Learning Theory, pages 1179-1206. PMLR, 2021.
[67]
Hao-HungTue Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis- Louis- Philippe Morency. Self-superrised
learning from multi-view perspective. In 9th International Conference on Learning Representations, ICLR 2021. Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In 8th International Conference on Learning Representations. ICLR 2020, Addis Ababa, Ethiopia, April 26-30. 2020. OpenReview.nef 2020. [69] George Tzanetakis. Automatic musical genre classification of audio signals. In ISMIR 2001, 2nd International Symposium on Music Information Retrieval, Indiana University, Bloomington, Indiana, USA, October 15-17, 2001, Proceedings, 2001.
[68]
[70] Aätonn van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, CoRR,
007.03748. 2018.
[71] Michacl Vcale, Max Van Kleck, and Reuben Binns. Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making. In Proceedings of the 2018 chi conference human factors in computing systems, pages 1-14, 2018.
[72] Jianfeng Wang. Zhengyuan Yang. Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A generative image-to-text transformer for vision and language. Trans. Mach. Learn.
Res., 2022, 2022.
[73] Peng Wang. An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to- sequence learning framework. In International Conference on Machine Learning, pages 23318-253 2-3440..
PMLR, 2022.
[74] Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as foreign language:
Beit pretraining for all vision and vision- languae tasks. CoRR, abs/2208. 10442, 2022.
[75] Tete Xiao, Xiaolong Wang. Alexei A. Efros, and Trevor Darrell. What should be contrastive in contrastive learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. Open 2021.
[76] Chang Xu, Dacheng Tao, and Chao Xu. A survey on multi-view learning. CoRR, 304113344444444 2013, [77] Jie Xu, Shuo Chen, Yazhou Ren, Xiaoshuang Shi, Hengtao Shen, Gang Niu, and Xiaofeng Zhu. Self- weighted contrastive learning among multiple views for mitigating representation degeneration. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023. NeurlPS 2023, New Orleans. LA. USA, December 10- 16, 2023, 2023.
(78) Zhao Yang, Jiaqi Wang, Yansong Tang. Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language- aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 18 18155-18166 165, 2022.
[79] Chun-Har Hsiao Ych, Cheng- Yao Hong. Yen- Ven-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. In European Conference on Computer Vision, pages 668 684. Springer, 2022. [80] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metries for semantic inference over event descriptions. Trans. Assoc. Comput. Linguistics, 2:67-78. 2014.
[81] Olaf Zawacki- Zawacki-Richter, Victoria Marin, Mclissa Bond, and Franziska Gouverneur. Systematic review of research on artificial intelligence applications in higher education wheree are the educators? International
Journal of Educational Technology in Higher Education, 16(1):1-27, 2019.
[82] Changqing Zhang, Zongbo Han, Huazhu Fu. Joey Tianyi Zhou, Qinghua Hu, et al. Cpm-nets: Cross partial mulli-view, networks. Advances in Neural Information Processing Systems, 32, 2019.
[83] Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung X. Pham, Chang D. Yoo, and In So Kweon. How does simsiam avoid collapse without negative samples? A unified understanding with self-supervised contrastive learning. In The Tenth International Conference Virtual Event. April 25-29, 2022. OpenReview.ned. 2022.
Learning Representations, ICLR 2022,
[84] Linfeng Zhang, Xin Chen, Junbo Zhang. Runpei Dong. and Kaisheng Ma. Contrastive deep supervision. In European Conference on Computer Vision, pages 1-19. Springer, 2022, [85] Rui Zhu, Bingchen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen. Improving contrastive learning by visualizing feature transformation. In Proceedings of the IEEE/CVF International Conference
on Computer Vision, pages 10306-10315, 2021.
15

---

Appendix
A
Broder Impact
Multimodal contrastive learning, emerging learning paradigm, enhances the model's perfor- mance in understanding and processing multimodal data by establishing connections and comparisons between multiple modalities. With the increasing demand for processing multimodal data, widespread application of large models, and rapid development of embodied intelligence, the application of mul- timodal contrastive learning in the real world requires more consideration of safery and application.
Safety Privacy and data security [5] have exacerbated concerns the context of sensitive data processing, such facial recognition and personal identification information in multimodal learning. datasets to avoid incorporating private information into the training process much as possible. Concurrently, the training of multimodal algorithms often exhibits biases [4] that can contribute to discrimination or unfair treatment of certain groups, reducing the fairness and inclusivity. Our approach may require bias detection the application of bias reduction methods
We adopted
[8 12
71 in real-world applications to ensure that multimodal algorithms maintain unbiased ethics.
Application Multimodal learning, particularly when integrated into embodied intelligence, has far- reaching impacts across various domains. In healthcare [22] 61. it enhances advanced diagnostic tools and personalized treatments, which also requires specific measures to assure patient privacy and data security. In education (81B), multimodal AI can personalize learning experiences, potentially reducing educational disparities. which requires an unbiased AL. Embodied intelligence [53) in social interaction facilitates accessibility and provides companionship for vulnerable populations, especially as pre- me-training then fine-tuning becomes the mainstream training paradigm, our method helps models learn more coddal-nndeenddent unique information; unlike models such CLIP, which focus too much on shared information.
Future work Our research mainly focuses extracting unique information in modalities from different views. In terms of model architecture, we have implemented unique decoder to extract unique information and train the model with the quadruple loss with constraints. Our approach and Bleeker [3 ct.al both adopt the supervised learning method. Future work could explore extracting unique information through supervidd learning (SSL) approaches. One potential direction for this exploration involves augmenting the input data. Liang et al. [I] proposed FactorCL to factorize information into shared information and unique information by augmentation implement SSL. Furthermore, in the era of large language models (LLM), leveraging the powerful capabilities of LLMs to explicitly extract existing data unique and shared information between different views, as shown in Fig By combining this with the original data, training can be conducted on this explicitly separated information for enhanced model performance.
Shared information
Caption balding man wearing a red
A man, red life jacket, sitting, boat(canoe)
life jacket sitting a small boat.
Unique information
Caption2: green shirt and red life jacket sitting canoe drifting around the lake.
Caption
Caption2
a green shirt, drifing. the lake
balding (man),
small (boat)
Figure 6: Shared and unique Information in Multimodal Multi-view Scenario. A single image can be described from multiple viewpoints, each containing shared information and distinctive details unique to the specific perspective.
16

---

B Experiment Details
B.1 Datasets
Flickr30k is a benchmark commonly used in computer vision (CV) and natural language processing (NLP), The tasks applicable to this dataset involving image-caption and multimodal learning, like im- age understanding. visual question answering and generating text descriptions for images. Flickr30k contains 31783 images collected from the Flickr platform, each with five corresponding descriptive captions, totalling 158,915 captions.
Microsoft Common Objects in Context (MS-COCO) is large-scale dataset containing over 328000 images, and each image is paired with at least five detailed captions annotated by humans for other CV tasks like segmentation; MS-COCO also provides segmentation masks, key points and relationships between objects. The images within the MS-COCO cover multitude of object categories, activities, and scenes, representing a broad spectrum of settings and contexts.
Free Music Archive (FMA) is an extensive, open-access dataset designed for Music Information Retrieval (MIR) research, encompassing 917 GiB of audio data (equivalent to 343 days of playback) from 106,574 Creative commons-licensee tracks. This diverse collection, spanning 16,341 artists, 14,854 albums, and 161 hierarchically organized genres, provides researchers with high-quality, full-length audio files, pre-computed features, and rich metadata including track and user information, tags, and textual descriptions. FMA's comprehensive nature makes it ideal for various MIR tasks such as genre classification, artist identification, and music recommendation, while its predefined train/val- idation/test splits and subsets of varying sizes facilitate reproducible research and benchmarking in the field. Code, data, and usage examples are available at httttt:://githiii com wwwefdffffffmmmmmme/
GTZAN is benchmark dataset widely used in Music Information Retrieval (MIR) and audio signal processing research, particularly for tasks involving musical genre classification and audio feature extraction. The dataset comprises 1,000 audio excerpts, each 30 seconds in duration, equally distributed across 10 distinct musical genres: blues, classical, country, disco, hip-hop, jazz, metal, pop, reggae, and rock. All audio samples in GTZAN are standardized to 22,050 Hz sampling rate, mono channel, and 16-bit resolution in WAV format, facilitating consistent analysis and algorithm development. Despite some noted limitations, GTZAAN remains valuable resource for evaluating and comparing various approaches in automatic music genre recognition and related MIR tasks.
Clotho is diverse audio captioning dataset comprising 4981 audio samples (15-30 seconds each) from Freesound, paired with 24,905 crowdsourced captions (8-20 words each). Designed to facilitate general audio content description using free text, Clotho emphasizes perceptual diversity by providing multiple captions per audio and excluding visual or contextual cues during annotation. The dataset's s post-processing, including removal of unique words and speech transcription, enhances its suitability for developing and evaluating audio captioning systems.
AudioCaps is a seminal dataset for audio captioning, comprising 46,000 audio clips from AudioSet with human- authored descriptions. This large-scale corpus has become the benchmark for evaluating audio captioning models, catalyzing advancements in audio representation and multimodal learning. AudioCaps has facilitated the development of innovative architectures such as the Audio Captioning Transformer and retrieval-augmented models, significantly contributing to audio audio-language research. Its impact extends beyond captioning, influencing broader studies in audio-visual integration and inspiring more comprehensive datasets in the field.
B.2 Experimental Settings
Table 3: Multimodal Model Training Details.
Model
vs-com
Vieal Ewodky
Tou Encoler
Parans
Rosch
BS optiniter
verraay Mgg
scholaer
Epuck
B5
optinion
Ir
wameppssp sage
ahaa
McpLiR aoaac_asnalig
sdam
>kydR
OKU
6TM
CLIP
236
Detailed training settings In our experiments, we conducted a comprehensive comparison of multimodal model training strategies using the Flickr30k and MS-COCO datasets. We used VSE++ and CLIP, each employing distinct training configurations tailored to their respective architectures and optimization requirements.
17

---

For the VSE+ model, we utilized training regime consisting of 30 epochs with a batch size of 128. The optimization process was driven by the Adam optimizer with a learning rate set Notably, no warmup steps were employed, and the learning rate was adjusted using stepl. .R scheduler. This configuration was consistently applied across both the Flickr30k and MS-COCO datasets. The visual encoder for VSE++ was based on ResNet152, while the text encoding was handled by a GRU, resulting in a total parameter count of 67 million.
10 10-4
Moreover, the CLIP model was trained over five epochs with a larger batch size of 256. Optimization was performed using the AdamW optimizer with a significantly lower learning rate of 10-3, In contrast to VSE++, CLIP incorporated 100 warmup steps and utilized a cosine annealing scheduler for learning rate adjustment. This setup was also uniformly applied both datasets. The visual encoder for CLIP was a ViT-B/32, and text encoding was managed by a Transformer, leading to substantially larger parameter count of 152 million.
Unique Decoder Implementation Details We used CLIP(VIT/B-33 backbone) and VSEs+(res backbone); we chose a single-layer MLP for vsc++ and two-layer Transformers for CLIP. Both of them use nine layers as a unique start layer and 0 for no unique start layer. We choose the hyperparameters alpha_t as 0.08 most experiments and set positive_sample false.
B.3 More Implementation Details
The Luie algorithm enhances contrastive loss in representation learning through novel embedding space involving quadruplets. This approach primarily comprises two stages: the computation of similarity map from quadruple embeddings and the subsequent caleulation of cross- cross-entropy loss based on this map. The method leverages shared and unique embeddings to construct expressive feature representations, further augmented by three- viceedimensional vectors generated via cross- poos-ppoduuts, thereby intensifying the discriminative power of embeddings.
As shown in I the algorithm firstly processes four embedding vectors::rsared Tuniques Y/hared and Vinique, representing shared and unique features across two data sets. Padding may be applied to ensure dimensional consistency (Divisible by 3) across embeddings. The GETSIMMAP function, shown in 2 then calculates the similarity map between Fshared and 3/bared reflecting the similarity between shared feature embeddings. This computation involves normalizing the embeddings. obtaining preliminary similarity matrix via dot products, and adjusting the similarity values through exponential weighting and diagonal normalization.
Upon obtaining the similarity map, the algorithm transforms Ishared and Tunique (along with their y counterparts) into dice-dimensional vectors, triplet embeddings, via cross products. This geometric transformation aims to further enhance the distinctiveness of embeddings. The triplet embeddings are then normalized to ensure numerical stability during dot product calculations. Ultimately, the algorithm calculates the dot product of Tiriplet and Ytriplet, scales the result, and applies elsment-wise multiplication with the similarity map to produce the final logits. These logits, after scaling and absolute value adjustments, are used to compute the cross-entropy loss.
We also use orthogonal cosine embedding loss 3 to quantify the similarity between pairs of input vectors, facilitating the training of models on tasks that distinguish between similar and dissimilar data points. The objective of this loss computation is to minimize the discrepancy between predicted logits and actual labels, thereby optimizing the representational capacity of the embeddings.
The synthetic shortcuts experiment enables a quantitative analysis of the model's 's reliance on shortcuts when they are present and its ability to capture shared and task-relevant unique information. The re- sults demonstrate that our proposed method effectively mitigates the feature suppression phenomenon, contributing to improved performance on downstream tasks compared to previous approaches.
Generalization Capability Across Multiple Modalities To evaluate the generalization capability of our proposed approach on more modalities, we conducted comprehensive experiments across various modalities, including visual, textual, and acoustic domains. Specifically, we performed image- retrieval experiments utilizing the FMA (Free Music Archive) and GTZAN datasets, with quantitative results presented in Table 4 For text-audio retrieval evaluations, we leveraged the CLOTHO and AUDIOCAPS datasets, with comparative performance metrics detailed in Table
The empirical results demonstrate that our proposed models consistently outperformed the baseline approaches across all experimental configurations. For simplicity, employed the almost simplest
18

---

Algorithm 1 Luic loss calculation
Require:
Latent representation I-share;; Funique: Mshared;; Munique-
Ensure: Luic
1: function CALCULATEE (Fic(Fsshred Funiques Ushareds Munique
2: B, C' + shape of sharee
3: if' padding is enabled config then
4: pad_size + (3 - mod 3) mod 3 Wshared & pad Ishared with pad_size size Runique t- pad Punique with pad_stt size
5:
6:
7:
Yshored pad Yshared with pad size
8:
Vunique & pad Vunique with pad_size
9:
end if
10:
sim_may GET Vshared) mini heads mini_heads <- integer division of C by 3
12:
participated_ dims mini heads 3
13:
Ishared + reshape Zshared participated_dee dims to B, -1,3)
14:
Tunique f- reshape Tunique participated dima to (B, -1,3)
15:
Yshared f reshape i, participated dims to (B -1.3)
16:
Dunique reshape Yuniguel:: 1.: participatedddeeec dims] to (B, -1,3)
17:
Tuic f- cross product of Øshared and Zunique along dimension 2, then reshape to (B, -1)
18:
Yuic
product of Yshhardd and Vunique along dimension then reshape (B, -1)
19:
Tuic -- normalize Tuic along the last dimension
20:
Yuic normalize Yuucc along the last dimension
21:
logits - Juic x Vuic
22:
logits absolute value of logits
23:
logits + scale x logits x sim_map nwm_logits. B
24:
25:
labels + range from 0 to num logits - -1
LQUAD t CE loss of logits with labels + CE loss of logits" with labels
27:
Coos + GESCCoLLSS
Vanigue) + (ESCOSLOSS(Za, (ZSSSSS(Sshreeee @shored, Tanigue)
LUIC <- LOUAD + Leos
29:
return Luic
end function
Algorithm 2 Calculate similarity map
Require: latent representations æ,
Ensure: similarity map sim_map
1: function GETSIMMAP(r. y)
2: I < normalize x along the last dimension
3: y +- normalize y along the last dimension
4:
+ I X y"
5: him_map clamp himppapp between 0 and
6: sim_map + exp(sim_mas
7: fill the diagonal of sim_map with
8: return sim_map
9:
end function
19

---

Algorithm 3 Loss loss calculation
Require: inique_informationn Tunique shared information I'shared
Ensure: Less
1: function GETCOSLoSS Fshared
2: c-le-12
3:
product sum +- (Zunique Fshared) . sum dim 1)
4:
malnitude_square) f- (uniqques T Funique )-sum( (am = 1) +
5:
maadnilud_squarryy t- (3-hhared 3ishared) J-sum (dim 1) +
denominator <- /mag_aquars squarel x mag wag__quare2 6:
7:
cos t product pooduct_sum___ sum sum//enne /dnominator
8:
zeros < temsor of zeros with same shape as Tunique along dimension 0 cos
9:
pos_loss
10:
neg_loss + clamp(|cos 0.0 min = 0)
target < tensor of with same shape as Tunique along dimension
12:
Initialize loss loss_pos pos and loss_nee neg with the same shape as target, pos loss, and zeros for each index in target do if largel [i] is then
14:
loss loss_pos{? +- pos goo_loss(i)
16:
else
17:
loss_posi - zeros[i]
18:
end if
19:
target is then
20:
jass_negii] + neg_ [oss|i]
21:
else
22:
loss [ig_n [i] +- zeros [
23:
end
24:
end for
25:
loss f- loss_pos + loss_nse
26:
return mean of loss
27:
end function
Table 4: Image audio retrieval results FMA and GTZAN datasets.
i2a
a2i
Method
Datasets
@@D
R@5
R@ 10
RSUM
R@]
R@5
R@
RSUM
InfoNCE QUEST
15.87
28.62
35.87
80.36
12.50
25.50
29.12
67.12
FMA
17.83
30.87
38.50
87.20
13.50
26.52
30.62
70.64
InfoNCE QUEST
34.01
84.73
94.41
213.15
32.48
78.68
90.86
202.02
GTZAN
41.62
88.83
97.65
228.1
35.53
82.23
93.40
211.16
decoder structure (linear layers) and did not implement modality fusion or any cross-modal interac- tions. We believe that enhancing the architecture, strengthening cross-modal interactions, employing a larger batch size and incorporating pre-training would yield even higher performance.
Compute Resources. All experiments in this paper are run on a single NVIDIA A100 GPU. The implementation is based PyTorch 2.0.1. It takes about hours train the CLIP-based model Flickr30K for 5 epochs, and the maximum training time for other experiments does not exceed 12 GPU hours.
20

---

Table 5: Text audio retrieval results CLOTHO and AUDIOCAPS datasets.
t2a
a2t
Method
Datasets
@
R@5
R
RSUM
R@1
R@5
R@I
RSUM
InfoNCE QUEST
20.
51.30
66.56
138.02
20.06
52.66
68.23
140.95
CLOTHO
21.10
52.45
68.86
142.41
22.36
54.23
70.42
147.01
InfoNCE QUEST
4.59
17.79
26.22
48.6
5.45
15.78
22.48
43.71
AUDIOCAPS
5.16
18.08
27.17
50.41
6.02
15.98
24.78
46.78
21

---

C
Notation
Table 6: Notation used in the paper.
Symbol
Description
M.
Modality
of
The th sample(view) of modality
H MM
Representations as the encoder's 's output
Z and z denote the decoder's output representations for shared and unique
information, respectively.
z
The cross product of Z and Z
LinfONCE
InfoNCE loss
LInfONCE+LTD
Loss that combines InfoNCE and LTD
DINSONCE-IMM
Loss that combines InfoNCE and IFM
LsiC
Shared information constraint loss
LuIc
Unique information constraint loss
LP-UIC
Penalized Luic
EPUIC
Reformulated CP-UIc. ignore second item
Los
Orthogonalized cosine loss
LQUEST
Overall Quardruple InfoNCF: Loss
s
The similarity matrix between shared embedding
P
The weighted similarity matrix between shared embedding Z and Z
B
Batch image-caption pairs
7
temperature coefficient
XA
random variables from modility MA
P
the weighted similarity matrix, utilized as penalization
Modality M encoder parametrised by θ; takes sample X as input and returns its latent representation Hy, [m
FM(:0)
Modality M decoder parametrised by di takes representations H as input and returns its latent representation Zi,, = (H
Gu(:2)
s(3-)
Scoring function
Z(Z))Zi)
The conditional entropy of Z, given Z,
(Z,IZ))
Conditional entropy under the penalized term P
T(Z,, 7z)
Mutual information between Z, and Zj
22

---

D Analysis.
D.I Mutual Information
In many machine learning tasks, it is often observed that different views simultaneously harbour both task-relevant shared information and unique information (e.g., image captioning [38] 40 72 and
referring expression segmentation 778 4 To this end, we work with a dual-encoder architecture.
Contrastive learning with multiple views obtains mutual information between different modalities by learning the similarity among different views. [70] et al. introduced a lower bound on mutual information, known as InfoNCE, which is based on the concept of Noise Contrastive Estimation (NCE); it compares the compatibility of different views by maximizing the mutual information of positive pairs and minimizing the mutual information of negative pairs, learning to extract the consistent representation across different modal. It can be defined as follows:
N
exp [1s MR)
InfoNCE (X;oNCE((Xy)) = (((ww))p(a.
log
(13)
N
1=1 E exp (14429))
In this equation, I; and represent paired samples from the joint distribution the two random variables under consideration, while yj represents samples from the marginal distribution of one of the variables. The function fx, y) is learnable function, often parameterized by a neural network, that aims distinguish between the paired samples and the independently sampled ones.
In the realm of aason-laggaage contrastive learning, the InfoNCE loss function (70 enhances the similarity between positive sample pairs relative to negative ones. Mathematically, the InfoNCE loss articulated as:
exp(sim(x, E exp(sim(z,, W//TT
LIstaNCE
log
(14)
where sim(x, y) calculates the similarity between samples x and y. typically computed through dot product or cosine similarity. y+ denotes the positive sample corresponding to æ, and {v, ) is set including positive and K negative samples. The parameter serves a temperature coefficient, modulating the scale of similarity scores.
InfoNCE loss prevails in many contrastive learning algorithms and performs well in applications. assumes that there exists duplicated information across varied views or representations. Contrastive losses like InfoNCE loss tend to maximize the mutual features between different views while suppressing task-relevant features that may be used in downstream tasks in multi-view representation learning.
Multi-view redundancy 66 30
63 65
Mutual information (MJ), often denoted T(X; Y). is fundamental concept in information theory that quantifies the statistical dependence between two random variables, X and Y. It measures the reduction uncertainty about one variable when the value of the other known and is defined as:
p(a, v)
[(;;y) = E(z,)))p(z,,) flo p(x )p())
(15)
where p(x,y) y) is the joint distribution, and p(x:) and p(y) are the marginal distributions of X and Y. In machine learning, particularly in deep learning, MI is often used as an objective function or regularization component to promote or constrain the interdependence among variables. However, the precise quantification of MI is only attainable in limited instances, it requires the closed form of the density function and the tractable logarithm density ratio between the joint and marginal distributions. In most machine learning applications, practitioners only have access to samples from the joint distribution, making the direct computation of MI infeasible.
The InfoNCE bound has several desirable properties that make it an attractive choice for estimating mutual information. Firstly, it is a lower bound on the true mutual information, T(X; Y) Z INCE. which means that maximizing the InfoNCE bound leads to an increase in the true mutual information.
23

---

Secondly, the bound is computationally tractable and can be efficiently optimized using standard gradient-based methods, such as stochastic gradient descent.
As shown in Fig 2 multimodal mutual information can be divided into shared information T(XA; XB; Y) and task-relevant unique information (T Y))B) and T(XB; YJXa)). Information useful for the task can be represented as
1 = -
(16)
=I(XA: + (XXX
+((::
D.2 Gradient Analysis
For the simplified loss function Eq.
9
we first calculate the partial derivative of Z
ac
0x
P
Ost
(17)
X
AZ
OZ;
where s+ = sZZ Zj ). Define s = =
then we have:
p +
p+
0s
Pus
Pk
0s
p+ Ost
exp
exp
X
OZ
OZ.
k=1
ptst
p+
exp
Os
1
m
PAs
0s
Pk exp
(18)
X
OZ.
Ar
7
37.
k=1
P+ Ost
1
PEs
Os
Pk
exp
dzp
X7
T
k==
where X
For energy function sZ, 2j) Z. / [[,]]), the derived gradient the vector Z; shown as:
En exp(P Z /TT)
2, 2
(s( 2,)
Z; Z, Zi
Z5
OZ.
| |
(1
1
Z,
,(Z,Z, ||Z,1||Z,||
(19)
[2
1
(Z, s (Z. 2,
D.3 Mutual Information Estimation
Let Z4 and Z; represent the compact vector representation from different modalities input; the classical mutual information between Z; and Z, can be defined as:
I(Zi, Zz) Σ Zz) p(Zi, Zzy) 20.7,
(20)
p(Z:)p(Z,)
According to 70,, the density ratio plZ..2,) the similarity score and 7 is the temperature parameter. The penalty term P is defined as follows:
p(2.)p(2,) is expressed as exp(= Zz)))T))) where s sZ,, 2,) is
P P= cxp
diag Z;Z,)) + I]).
(21)
The penalty term does not affect the numerator but amplifies the denominator in Eq. 8 thus the energy function in Eq. (8) can be reformulated as:
exp(s(74, 2/)/T) 1
p(Z, Z2,) Z.)
exp(s(Z,, 2,)/T log P) x p(Z,)
where PXP (22)
24

---

Next, we derive the lower bound of MI,
B1 12.0.
ÉPAIIC -E log
piZ 2akk)
p(2
M(2ye)
((Zzk)
(Z 7es)
log
p 21y Zik)
p(Zj()
((zkk)
(N 1) B(ZzzlZze)
Elog
5( ((ee)
p((3z)
(23)
pl (Zikk
log
+ 51 (2x)
(N 11
p(Z(N)
2 E log
[1 ZAck)
11 (2,1Z)) (2,) + log N
(Z,1Zz) I(Z,, 2,) (Zz)ZZ)) + log N
E Limitation
Dimension restrictions. The limitations of the cross-product in the context of QupleInfoNCE'' unique information extraction are theoretically constrained. When contrastive learning is applied to both unique and shared representations, it can lead model degradation, and trade-off choice the cross prosucp which is weaker constraint. In oowdiieesioona space, it is straightforward to prove the properties of orthogonal vector space, yet the extension of the cross cross-product to high- digeedimensional space is challenging. The behaviour of nigh-dimensiong space representation is difficult to control, resulting in our theory lacking sufficient interpretability in high-dimensional space. This highlights the importance of developing more sophisticated methods for high-dimensional data analysis in the future.
The Extraction of Unique Information. On the synthetic shortcuts dataset, our framework achieved significant performance, validating the positive role of unique information in preventing model shortcuts. However, within the entire framework, the core lies in identifying unique information, which is essentially task-dependent definition. Our experiment indicates that pre-trained models (such as CLIP (52)) typically yield higher unique information benefits due to their extensive generic representations. Non-pre-trained models (such as VSE++ (23) struggle to discern unique information. tending to shortcuts and losing vital unique information. In this study, we introduced an additional unique information decoder capture unique information, which incorporated an extra gradient branch, suggesting exploration space remains for single-stream unique information extraction. This also proposes future research: how to more effectively extract unique information from a single stream to reduce computational complexity while maintaining model performance.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims presented in the abstract and introduction consistent with the contributions and scope detailed in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Appendix E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

Justification: See Section
2 and Appendix D
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix B. and Appendix B.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
Justification:
See the "supplementarrzzip" file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (kttps://nnps.cm
Policy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7
htte//CoooSSSiiieeeeee oa wwweiisssioooiiiiieeie ooic for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 3 and Appendix B.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Section 3.2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
28

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix B.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: See Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This paper have state which version of the asset is used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssithcooee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
30

---

Answer:
[NA]
Justification:
Paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "vjV7XhAkih", "reviewer": "Reviewer_KzFg", "summary": "A new Multimodal Contrastive Learning method named QUEST is proposed to deal with the fine-grained alignment problem between different modal. Both quaternion contrastive objectives and orthogonal constraints are proposed to extract sufficient unique information. The quaternion vector spaces are designed to simultaneously optimize shared and unique information. Experiments conduct the superior performance in multimodal contrastive learning benchmarks.", "questions": "Why the value of Eqn4 should be optimized to maximum value? Can you explain with an example in multimodal contrastive learning?", "limitations": "The experiments conducted in text and vision modal. Experiments on more modal data should be conducted to show the generalization ability.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed method utilizes quadruple embedding to constrain unique information from different views in a plane space which avoid the degeneration of the unique decoder.\nA self-penalization mechanism is proposed to penalize hard negative samples by dynamically re-weight the distribution of negative samples. And theoretical analysis is provided to show how this penalization effectively improves the extraction of unique information.\nExperiments conduct the superior performance in multimodal contrastive learning benchmarks.", "weaknesses": "1) There are some typo, for example: line 218 MS-COCO-Cpation\n2) The position of Z_i^s and Z_i^u of Modal M_i should be exchanged in Figure3. Also, Z_i_s and Z_i^u of Modal M_j should be Z_j^s and Z_j^u in Figure3.\n3) Please check the position of + symbol in Eqn5.\n4) Why the value of Eqn4 should be optimized to maximum value? Can you explain with an example in multimodal contrastive learning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer KzFg,\n\nWe sincerely appreciate your thorough review and insightful comments. Please find our responses below.\n\n------\n\n**W1-3: ”Presentations/Grammar/Typos”** \n\n**A1:** We apologize for the some typos in this paper and we have fixed it. We have re-examined Equation 5 and refined them. The notation $\\mathbf z_j^{\\mathbf n+}$ now correctly refers to the positive samples from modality $j$, and $\\mathbf Z_{jk}^{\\mathbf n-}$ refers to the negative samples from modality $j$. We appreciate your careful review and guidance.\n\n**W4: “Maximize the Equation 4 value”**\n\n**A2:** We apologize for any confusion caused by our oversight. In fact, we intend to maximize the absolute value of Equation 4 (Appendix.B3 for more details). We will correct this in the new version.\n\nFirst, we investigate why InfoNCE falls into shortcuts. We provide a simplified explanation in Equation 6: $-\\frac{\\partial\\mathcal L_{\\mathrm{InfoNCE}}}{\\partial Z_{a}} = \\frac{1}{\\tau}(Z_{b}^{+} - \\sum_{i=0}^{N}\\beta_{i}Z_{bi})$. The term $\\frac{1}{\\tau}Z_b^+$ brings positive samples closer together, maximizing .  $||Z_{a}||\\cdot||Z_{b}^+\\sin \\alpha$.However, in multi-view scenarios, $Z_{b}^{+}$ is sampled from $k$ positive samples as $Z_{b1}^{+}, Z_{b2}^{+}, \\ldots, Z_{bk}^{+}$. Under the guidance of the gradient, the final representation tends to capture the shared information among all positive samples, leading to shortcuts.\n\nTo capture unique information, we disentangle features into shared and unique representation. We apply strict constraints to bring the shared representations from different modalities closer together, while applying weaker constraints to keep the unique representations in the same plane (Intuitively, the unique information between different views is unrelated, we will not pull them closer) as shown in Figure 1(b). Maximizing the absolute value of Equation 4 is equivalent to optimizing the quaternion vectors $(\\mathbf{Z}_i^\\mathbf{s},\\mathbf{Z}_i^\\mathbf{u},\\mathbf{Z}_j^\\mathbf{s},\\mathbf{Z}_j^\\mathbf{u})$ to be in the same plane.\n\n------\n\n**L1: ”Generalization ability for more modality”**  \n**A3:**  We conducted additional experiments on common modalities including images, text, and audio.  Specifically, we performed image-audio experiments using the FMA and GTZAN datasets, as shown in Table 1. For text-audio experiments, we utilized the CLOTHO and AUDIOCAPS datasets, with results presented in Table 2. All of our models outperformed the baseline. Almost all of our models outperformed the baseline. For simplicity, we employed the almost simplest decoder structure (linear layers) and did not implement modality fusion or any cross-modal interactions. We believe that enhancing the architecture, strengthening cross-modal interactions, employing a larger batch size and incorporating pre-training would yield even higher performance. \n\n**Table 1: The results on the GTZAN and FMA datasets.**\n\n| Method  | Datasets | R@1       | R@5       | R@10      | R@1       | R@5       | R@10      |\n| ------- | -------- | --------- | --------- | --------- | --------- | --------- | --------- |\n|         | FMA      |   |       **i2a**     |           |   |       **a2i**     |           |\n| InfoNCE |          | 15.87     | 28.62     | 35.87     | 12.50     | 25.50     | 29.12     |\n| QUEST   |          | **17.83** | **30.87** | **38.50** | **13.50** | **26.52** | **30.62** |\n|         | GTZAN    |   |      **i2a**      |           | |      **a2i**        |           |\n| InfoNCE |          | 34.01     | 84.73     | 94.41     | 32.48     | 78.68     | 90.86     |\n| QUEST   |          | **41.62** | **88.83** | **97.65** | **35.53** | **82.23** | **93.40** |\n\n**Table 2: The results on the CLOTHO and AUDIOCAPS datasets.**\n\n| Method  | Datasets  | R@1       | R@5       | R@10      | R@1       | R@5       | R@10      |\n| ------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n|         | CLOTHO    |       |         **t2a**      |           |         |      **a2t**      |           |\n| InfoNCE |           | 20.16     | 51.30     | 66.56     | 20.06     | 52.66     | 68.23     |\n| QUEST   |           | **21.10** | **52.45** | **68.86** | **22.36** | **54.23** | **70.42** |\n|         | AUDIOCAPS |           |      **i2a**      |           |         |       **a2i**      |           |\n| InfoNCE |           | 4.59      | 17.79     | 26.22     | 5.45      | 15.78     | 22.48     |\n| QUEST   |           | **5.16**  | **18.08** | **27.17** | **6.02**  | **15.98** | **24.78** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RAStMtLS0A", "reviewer": "Reviewer_9n5E", "summary": "The paper focuses on developing a new multi-modal representation learning approach where the extraction and integration of both shared and unique information across multimodal data is the focus. The method aims to pull shared representations closer while aligning the unique representations with the shared representation on a common plane.\n\nKey components: an encoder, a shared decoder, and a unique decoder. Contrastive loss constrains learning of shared information. The proposed framework seeks to mitigate shortcut learning.", "questions": "What does it mean by shortcut learning in this case? There can be large variations in learned shortcuts.\n\nBetter to discuss more recent multimodal shortcut learning papers such as Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion, ICML 2024.", "limitations": "Perhaps, the proposed method may be too complicated for some problems.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The proposed idea is novel and makes sense.\n\nTechnical details are sufficient to understand the main idea of the paper.\n\nThe proposed design and the architecture makes sense.", "weaknesses": "Lack of theoretical analysis.\n\nIs there any theoretical justification/demonstration or proof that the encoder information can be disentangled to shared and unique representations for a given distribution w.r.t a task or in a task-agnostic manner? I would like to hear author perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 9n5E,\n\nWe sincerely thank you for taking the time to review our paper. Our responses to your comments are provided here:\n\n---\n\n**W1 “Theoretical justification that the encoder information can be disentangled to shared and unique representations”**:\n\n**A1:** To the best of our knowledge, common methods for disentangling shared and unique representations involve the use of estimators. The shared representation is learned by maximizing the cross-mutual information estimation while minimizing the mutual information between the shared and unique representations. In our paper, we minimize the SIC to maximize the lower bound of mutual information among shared representations. Similarly, our proposed self-penalization method utilizes self-supervised signals to tighten the lower bound, $I(Z_i, Z_j) \\geq H^{\\tilde{P}}(Z_j | Z_i) - H(Z_j | Z_i) + \\log N - \\widetilde{\\mathcal{L}}_{\\text{P-UIC}}$ (see Appendix D.3 for more details).\n\nTypically, mutual information is minimized by minimizing its upper bound, such as through adversarial objectives[1] or CLUB-like estimators[2,3]. However, these approaches are not equivalent to directly minimizing mutual information, despite many recent works striving to tighten the lower bound. These methods perform well in supervised tasks because task-relevant information can be well-defined (e.g., classification) and may not be suitable for pre training tasks. However, we focus more on how to preserve features in self-supervised learning. We conducted extensive experiments on the shortcuts dataset to confirm this: using only InfoNCE to fine-tune pre-trained models (CLIP, ResNet) on a new dataset results in the loss of a significant amount of original features, even though these features still exist in the shortcuts dataset (see Table 1).\n\nIn the image-text self-supervised pre-training phase, we loosely define task-relevant information as $\\mathcal{T} =\\bigcup_{n=1}^N\\{(x_{i1},...x_{ik})\\cap(x_{j1},...x_{jm})\\}_n$, where images and texts have a many-to-many relationship. Multiple texts can describe the same image (or video), with different texts holding both shared and unique information related to the image. Therefore, minimizing unique information between texts and images during the pre-training phase may not be appropriate. Hence, we choose UIC, which does not pull closer or push away the unique information of different modalities, but rather optimizes them within a plane to retain information that may be beneficial for downstream tasks (e.g., classification, segmentation).\n\n**Q1-2: “Shortcut learning in this case and discuss more recent multimodal shortcut learning papers”**\n\n**A2:** Shortcut learning, in the context of deep learning, refers to a phenomenon where neural networks exploit simple but suboptimal decision rules that work for most examples in the training set but fail to generalize to more challenging test examples [4]. This occurs when models learn to solve a task using features or heuristics that are correlated with the target in the training data but are not causally related to the task at hand [5]. In our case, under the setting of image-text retrieval, we synthesize images with MNIST image patches on them and append corresponding numbers to their associated captions. In this case, the model can easily get better performance by capturing MNIST shortcuts in the image and the corresponding numbers in the text under optimization of $\\mathcal{L}_{\\text{InfoNCE}}$, rather than focusing on more complex representational information present in both the image and text. And in downstream tasks when there is more complex unique representation in the data, the model fails and cannot complete the task well. More examples of synthesized shortcuts can be found in the submitted rebuttal PDF file.\n\nBesides, we are more than delighted to discuss with some recent work on multimodal shortcuts learning. Below are some brief discussion:  [6] uses the QUAG method to reveal that current VideoQA models often exploit dataset shortcuts rather than learning true multimodal representations. [7] proposes a novel backdoor attack method for multimodal-guided visual grasping systems leveraging shortcut learning and multimodal information. [8] provides comprehensive strategies for detecting and mitigating shortcut learning in VQA. Theses studies highlights the critical need to addressing shortcut learning in multimodal systems.\n\n**L1: “The proposed method may be too complicated for some problems”**\n\n**A3:** For some tasks, especially for tasks on some traditional datasets. With the rise of the pre-training era, we believe that for general models, capturing as much unique information as possible during the training phase may lead to better performance in downstream tasks. At the same time, we used an almost simplest version to experiment on more modalities (only introducing a small number of parameters, see Table 1 and Table 2), and this method can be plug-and-play to various architectures (ResNet, ViT, etc.). We believe that combining more advanced methods can further improve model performance.\n\n[1] Learning Disentangled Representations via Mutual Information Estimation. ECCV2020\n\n[2] CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information. ICML2022.\n\n[3] Factorized contrastive learning: Going beyond multi-view redundancy. NeurIPS 2023.\n\n[4] Shortcut learning in deep neural networks.\" Nature Machine Intelligence 2020.\n\n[5] Can contrastive learning avoid shortcut solutions?. NeurIPS 2021.\n\n[6] Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion. ICML2024.\n\n[7] Shortcut-enhanced Multimodal Backdoor Attack in Vision-guided Robot Grasping.\" Authorea Preprints 2024.\n\n[8] Shortcut Learning in Visual Question Answering.  2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "15cW1Brcpj", "reviewer": "Reviewer_zJ9C", "summary": "This paper proposes the Quadruple Multimodal Contrastive Learning Framework (QUEST) to capture shared and unique task-relevant information in the process of contrastive learning, enabling models to capture more unique information in downstream tasks to achieve better performance. Specifically, this paper introduces a quadruple embedding space and optimizes shared and unique information within it simultaneously. Additionally, this paper adopts a self-penalization mechanism using shared information to guide the optimization of unique information. This paper evaluates the QUEST method on popular datasets (Flickr30k and COCO). On public benchmarks and synthetic datasets, the method shows significant performance improvements.", "questions": "How can features be decomposed into shared features and unique features? What is the effectiveness of this method in practice?", "limitations": "The behavior of the cross product operation in low dimensions may be unpredictable in higher dimensions.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Compared to the latest state-of-the-art methods, a significant performance improvement was achieved (an average of 97.95% on the CLIP model), with enhancements observed across multiple mainstream datasets (flickr30k and coco) and models (both CNN-based model and Transformer-based model).\n\n2. The idea of decomposing features into task-relevant unique features and task-relevant shared features, leveraging quaternion vector spaces, is very novel and effective according to the ablation study. Great work!\n\n3.  The motivation of the paper is very logical, clear, and easy to understand.\n\n4.  The paper demonstrates very high performance on both popular public datasets and the shortcuts dataset.", "weaknesses": "1. In the shared Decoder of Equation 1 in Section 2.3, the parameters on both sides of the shared decoder equation are inconsistent: the left side has $\\Phi_i$, while the right side has $\\Psi_i$. This is strange. Can you tell me why it needs to be written like this if necessary?\n\n2. In the 3.3 ablation study section, what does QUEST-SIC and QUEST-UIC mean? There was no explanation before. Do QUEST-SIC and QUEST-UIC correspond to $\\mathcal{L}_{\\text{SIC}}$ and $\\mathcal{L}_{\\text{UIC}} in Table 2 respectively?\n\n3. What does $Z_j^n+$ in equation 5 mean? Can you explain why the \"+\" sign is placed to the right of the symbol \"Z_j^n\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer zJ9C,\n\nWe sincerely thank you for taking the time to review our paper. Our responses to your comments are provided here:\n\n---\n\n**W1: \"In the shared Decoder of Equation 1.\"**\n\n**A1:** Thank you for your valuable review comments. There is indeed a typo in Equation 1. For shared information, the input data $\\mathbf X_i$ is encoded by an encoder parameterized by $\\Theta_i$, denoted as $\\mathcal F_{\\mathcal M_i}$, and then further processed by a shared decoder parameterized by $\\Psi_i$, denoted as $\\mathcal G^{s}_{\\mathcal M_i}$, to obtain the representation $\\mathbf Z ^s_i$.\n\n**W2: \"What does QUEST-SIC and QUEST-UIC mean?\"**\n\n**A2:** In the section of ablation study, QUEST-SIC means only use shared information constraint for training, similarly, QUEST-UIC refers to only using unique information constraint during training. In order to be consistent with the experimental table, QUEST-SIC and QUEST-UIC will be called $\\mathcal L_{\\text{SIC}}$ and $\\mathcal L_{\\text{UIC}}$ respectively.\n\n**W3: \"What does $Z_j^n +$ in equation 5 mean? Can you explain why the \"+\" sign is placed to the right of the symbol \"$Z_j^n$\"?\"**\n\n**A3:** Thank you for your correction. Sorry for the small mistake in writing the latex formula. The \"n\" and \"+\" should be placed together on the upper right corner of \"z\", that is, it should be written as $Z_j^{n+}$. This symbol represents the representation of a positive sample in the quaternion embedding space.\n\n**Q1: \"How can features be decomposed into shared features and unique features? What is the effectiveness of this method in practice?\"**\n\n**A4:** Please see reviewer 9n5E A1. For pre-trained models, it is imperative to retain as much potentially beneficial information for downstream tasks as possible. However, achieving this through contrastive learning in multi-view scenarios presents significant challenges, particularly in cases of many-to-many relationships between images and text. For instance, a single image can often be described by multiple captions, all of which serve as positive samples for that image. Traditional contrastive learning methods tend to prioritize the shared information among captions (as discussed in Section 2.3), potentially leading to the loss of unique information in the encoder. This information loss can be detrimental to downstream tasks. Moreover, the unique information contained in each caption is of  importance. We have conducted extensive experiments on MSCOCO and Flickr30k datasets to substantiate this claim.\n\n**L1: \"The behavior of the cross product operation in higher dimensions.\"**\n\n**A5:** You've raised a valuable point. The generalization of the product can be achieved through multiple approaches, leveraging both the orientation and metric structure analogous to the conventional three-dimensional cross product. In an $n$-dimensional space, it is feasible to compute the product of $n - 1$ vectors, resulting in a vector orthogonal to all input vectors. However, when constrained to non-trivial binary operations yielding vector outputs, the existence of such a product is limited to spaces of three and seven dimensions. Empirically, the properties of high-dimensional cross products may be different from those in low dimensions, so we are discussing the cross product in a limited finite embedding space. Although we have experimentally validated the effectiveness of this method, we are also eager to provide an explanatory approach. For each pair $(A_i, B_i)$, we calculate the cross product in 3-dimensional space: $A_i \\times B_i = \\mathbf{K}_i A_i B_i$, where $\\mathbf{K}_i$ is a zero-diagonal matrix. When extending to higher dimensions, $\\mathbf{K}$ can be considered as a sparse projection matrix with fixed parameters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers, we would like to sincerely thank all the reviewers for taking the time to read our paper and provide valuable feedback. We are delighted that reviewer KzFg (zJ9C) acknowledged the superior performance, innovation (9n5E, zJ9C), and reasonable motivation (9n5E, zJ9C) of our approach. Additionally, we appreciate all the reviewers' comments, which we have taken into consideration during the rebuttal period and made the following revisions:

1. According to reviewer KzFg's comments, we conducted experiments on additional modalities. To simplify, we did not employ any multimodal techniques (e.g., cross-modal interaction) and only added linear layers. Table 1 demonstrates that Quest(ours) achieved superior performance in the image-audio modality compared to the baseline model. Table 2 shows that Quest outperformed the baseline model in the text-audio modality. The experimental results indicate Quest's generalizability across more modality data.
2. In response to reviewers KzFg and 9n5E, we added examples of shortcuts in multimodal contrastive learning in Figure 1. Handwritten digits were simultaneously added to both text and image samples (with minimal impact on the original meaning of the images). However, training on these datasets led to a significant drop in model performance due to the tendency to learn easy features. Our model effectively addresses this issue. Even without adding these handwritten digits, an image is often described by multiple captions that usually hold different meanings. This is common in the real world and underscores the significance of our approach.
3. We appreciate reviewers KzFg and zJ9C for their suggestions regarding our phrasing and terminology. In our rebuttal, we provided explanations and committed to correcting all typos and terminology.

Please note that we have added tables and figures in the attached pdf to support our responses to the reviewers KzFg, 9n5E, and zJ9C.

Meta-review: This paper introduces the Quadruple Multimodal Contrastive Learning Framework (QUEST), which is designed to extract both shared and unique task-relevant information across multiple modalities during contrastive learning. The framework is novel in its use of quaternion vector spaces to simultaneously optimize shared and unique information, thereby addressing the issue of "shortcut learning" that often arises in multimodal tasks. The method is empirically validated on several multimodal datasets, including popular public benchmarks such as Flickr30k and MSCOCO, where it demonstrates significant improvements in performance.

Given the significant strengths of the work, the thoughtful and comprehensive rebuttal, and the consensus among reviewers for acceptance, I recommend that this paper be accepted as a poster (as some reviewers yet noted a lack of rigorous theoretical analysis, particularly regarding the disentanglement of shared and unique information). The authors are encouraged to incorporate the suggestions for further theoretical exploration and to refine the presentation in the camera-ready version.
Decision: Accept (poster)