QUEST: Quadruple Multimodal Contrastive Learning with Constraints and Self-Penalization

Authors: qi  song
tianxiang  gong
shiqi  gao
haoyi  zhou
jianxin  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 17:06:34

Abstract: Multimodal contrastive learning (MCL) has recently demonstrated significant success across various tasks. However, the existing MCL treats all negative samples equally and ignores the potential semantic association with positive samples, which limits the model's ability to achieve fine-grained alignment. In multi-view scenarios, MCL tends to prioritize shared information while neglecting modality-specific unique information across different views, leading to feature suppression and suboptimal performance in downstream tasks. To address these limitations, we propose a novel contrastive framework name *QUEST: Quadruple Multimodal Contrastive Learning with Constraints and Self-Penalization*. In the QUEST framework, we propose quaternion contrastive objectives and orthogonal constraints to extract sufficient unique information. Meanwhile, a shared information-guided penalization is introduced to ensure that shared information does not excessively influence the optimization of unique information. Our method leverages quaternion vector spaces to simultaneously optimize shared and unique information. Experiments on multiple datasets show that our method achieves superior performance in multimodal contrastive learning benchmarks. On public benchmark, our approach achieves state-of-the-art performance, and on synthetic shortcut datasets, we outperform existing baseline methods by an average of 97.95\% on the CLIP model.

Main paper: 
 introduction: 
 Multimodal Contrastive Learning (MCL) has demonstrated robust representation capabilities and generalizability and effectively transferring to various downstream tasks (e.g. cross-modal retrieval [40, 39, 33] , image captioning [37, 72, 73] ). However, simply applying contrastive learning in multimodal scenarios presents significant challenges. In particular, contrastive learning treats all negative samples equally, ignoring the potential semantic relationships between negative samples and the anchor. Besides, current contrastive learning methods focus on maximizing mutual information between two views [68, 70] while ignoring unique information [41] . In multi-view scenarios, the assumption that modalities share substantial task-related information often does not hold, especially in complex datasets with minimal inter-modal overlap. Meanwhile, recent studies [56, 75] indicate that contrastive learning often neglects significant portions of input information, leading to feature suppression [1, 8] and shortcut learning [25, 57] , where models minimize loss through the simplest path (e.g. shared information [3] ), sacrificing deeper learning. These issues are prevalent in multimodal [45, 54, 32] and multi-view tasks [76, 82, 50, 43] . Recent approaches focus on pre- We build the quaternion embedding space, which aligns shared and unique representations from different modalities through the application of constraints and self-penalization. The L SIC narrows the gap between shared representations, while L P-UIC pulls the plane spanned by intra-modality shared and unique representations closer. Furthermore, Orthogonalization loss L cos is employed to constrain the area. serving more unique information, including reconstruction regularization [77, 4] , implicit feature modification [58] , and factorized representation [41] , among others. However, these methods either overly introduce noise which may harm downstream tasks, or rely on certain assumptions (e.g., augmentation [41, 65, 45] ). Additionally, we find that these methods do not explicitly distinguish unique information and still optimize using contrastive learning, making it difficult to avoid the model learning shortcuts [68, 29] . This raises the question: can we explicitly extract both task-related unique and shared information without introducing too much noise?To this end, we proposes a novel contrastive framework called QUEST: Quadruple Multimodal Contrastive Learning with Constraints and Self-Penalization, designed to enhance the extraction and integration of both shared and unique information across multimodal data. Our primary motivation is to develop a mechanism that effectively captures unique information through a novel quaternion multimodal embedding space, as illustrated in Figure 1(b) . This embedding space aims to pull shared representations closer while aligning the unique representations with the shared representation on a common plane. We achieve this by leveraging the properties of the normal vector from the cross-product to diversified unique representation. Consequently, our approach aligns commonalities across modalities while preserving the distinctive unique features of each modality. Specifically, we first split a network into three components: an encoder, a shared decoder, and a unique decoder. The encoder learns general features with little bias toward specific tasks, while the shared decoder and unique decoder learn agreement and discriminative information, respectively. We build contrastive loss to constrain learning of shared information. To avoid the unique decoder degenerating into the shared decoder, we propose novel contrastive objectives and orthogonal constraints to optimize the quaternion vector space. Finally, self-penalization is used to prevent shared information from overly affecting quaternion vector space optimization. Our framework seeks to mitigate shortcut learning, offering a more nuanced, task-related learning paradigm. Our main contributions can be summarized as follows:• We develop a novel framework to efficiently extract shared and unique information across multimodal data. To avoid the degeneration of the unique decoder, we propose an algorithm that utilizes quadruple embedding to constrain unique information from different views in a plane space. • We consider that traditional CL overly relies on shared information due to data bias, causing failures with negative samples containing shared information related to the positive sample. Meanwhile, to prevent shared information from dominating the extraction of unique information, we introduce a self-penalization mechanism to dynamically reweight the distribution of negative samples, which penalizes hard negative samples. We provide theoretical analysis to show how this penalization effectively improves the extraction of unique information. • We achieve state-of-the-art on popular datasets (e.g. MS-COCO [9] and Flickr30k [80] ) compared to the baseline, demonstrating the general effectiveness of QUEST. Additionally, experiment results on synthetic shortcut datasets outperform baselines 97.95% on average for CLIP, verifying the efficacy of QUEST.2 Methods 
problem formulation: 
 For different modalities {M i } K i=1 , given one modality, denoted as M i , along with its corresponding set of views {x j i } Ni j=1 , N i ≥ 1, for one modality M 1 encoder parameterized by Θ 1 , represented as F M1 (•; Θ 1 ), and another modality M 2 encoder parameterized by Θ 2 , denoted as F M2 (•; Θ 2 ). These encoders process the sample from modal M 1 and M 2 , for those modalities with multiple views, like modal one M 1 and each of its views through their respective encoder, resulting in corresponding general representations H j M1 = F M1 (x j 1 ; Θ 1 ) and H j M2 = F M2 (x j 2 ; Θ 2 ). However, as illustrated in Figure 2 , the InfoNCE loss maximizes task-related features shared across all modalities during training (i.e. I(X A ; X B ; Y )), while simultaneously suppressing the unique task-related features of each individual modality(i.e. I(X A ; Y |X B ) and I(X B ; Y |X A )). This process ultimately results in the loss of unique information. Therefore, the general representations are then separated into task-related shared and unique features through different decoders, i.e., the representations of different modalities H j M1 , H j M2 are inputted separately into different decoders G Mi (•) parameterized by Φ i . For the complete notations, refer to Appendix C, Table 6 . X A Modality MA Y Task T X B Modality MB I(X A ; Y |X B ) I(X B ; Y |X A ) I(X A ; X B ; Y ) X A Modality MA Y Task T X B Modality MB I(X A ; Y |X B ) I(X B ; Y |X A ) I(X A ; X B ; Y )Figure 3 : Framework of QUEST. The unique decoder is utilized to extra view-specific unique information and this process is guided by the proposed constraints and penalization.In multi-view scenarios, the relationships between different modalities are many-to-many (e.g., for image retrieval, multiple captions can refer to the same image). Within a single modality, different views contain task-related unique and shared information. Additionally, task-related shared information may also exist among negative samples (as indicated by the red shading in Figure 3 ). Therefore, optimizing solely for shared information while ignoring unique information is suboptimal.To address the challenge of overlooking unique information inherent to different perspectives in multimodal scenarios, we introduce the effective framework called QUEST: Quadruple Multimodal Contrastive Learning with Constraints and Self-Penalization. This framework extends existing contrastive learning methods by incorporating a fourpartite architecture specifically designed to enhance the capture and integration of distinctive modalspecific features. The overall architecture is shown in Figure 3 . According to [84, 62, 34] , from the perspective of neural network architecture, shallow layers learn low-level and general features while deeper layers learn task-biased high-level semantic features. Therefore, we define the encoder output as H, which contains shared S and unique U information related to the task, H ⊇ S ∪ U . Consequently, we shared shallow layers for general representation to reduce computational cost and optimize the shared decoder and unique decoder for S and U , respectively.For each modality, the input data X i (sampled from views) undergoes transformation by a modalityspecific encoder F Mi (•), producing an intermediary general representation denoted as H Mi . Consequently, we introduce two decoders: a shared information decoder G s Mi (•) and a unique information decoder G u Mi (•). These decoders are tasked with disentangling the shared and unique components from the representation H Mi , respectively. Let Θ i represent the parameters of a shared encoder, while Φ i and Ψ i symbolize the decoders for shared and unique information, respectively. The representation can be formulated as follows:EQUATIONShared Information Constraint. In multimodal and multi-view scenarios, data from different modalities and views often encapsulate shared information vital for model training. We conduct a shared Information Constraint (SIC) to maximize the lower bound of MI between representations from different views to encourage the shared decoder to learn agreement related to the task. This constraint is optimized by computing the InfoNCE loss between (Z s i and Z s j ), defined as,EQUATIONUnique Information Constraint. In contrast to shared information, unique information is modalityspecific and task-related, providing essential insights for downstream tasks. To preserve this, we introduce a Unique Information Constraint (UIC), extracting the unique information that exists within different views, which are unrelated to each other yet relevant to the task. Relying solely on Shared Information Constraint (SIC) is insufficient to preserve this information [41, 45, 3] . Strict constraints, such as directly enforcing consistency of distributions across diverse views, may lead to the Unique Information Constraint converging to the SIC, particularly in scenarios with identical input representations, loss functions, and network structures. To address this issue, we implement a less stringent constraint, aiming to maximize the similarity between the spaces formed by unique and shared information across different modalities. Firstly, we derive the representation space of normal vectors for shared and unique embedding spaces through cross-product calculations,EQUATIONIn the newly projected space, our objectives aim to maximize the alignment of unique representation from different modalities within the plane spanned by the shared representation, the magnitude can be formulated as:EQUATIONwhere sin α and sin β represent the sine similarity between the shared and unique representation across different modalities, and cos γ represents the cosine similarity of the normal vector. We maximize sin α and sin β via orthogonalized cosine loss L cos and contrastive loss to maximize cos γ, the unique information constraint can be formulated as:EQUATIONBy pulling positive samples closer and pushing negative samples away, we constrain the shared and unique representations of different modalities to coexist within the same spatial plane as much as practicable. Concurrently, we employ L cos to maintain the diversity of unique information and ensure that Z s i = Z u i . It is worth noting that we do not impose any constraints on the unique representations of different modalities, as these may be inherently unrelated.Comparison. Applying standard InfoNCE to extract unique information leads to suboptimal generalization, as demonstrated in Section 3.3. To address this, we propose UIC with indirect vectors Z n , where the cross-product operation fundamentally alters gradient propagation patterns compared to pair-wised InfoNCE. Specifically, given [70, 83] , the gradient with respect to anchor embedding Z a as follows,L InfoNCE = -log exp(Za•Z + b /τ ) N i=0 exp(Za•Zi/τ )EQUATIONwhereβ i = exp(Za•Zi/τ ) N i=0 exp(Za•Zi/τ ) and Z + b is random sampled from positive set {Z 1 b , ...Z j b }.Under the assumption that different views hold both shared and unique information, we haveI(Z j b ; Y ) = I(Z j b ; Z a ; Y ) + I(Z j b ; Y |Z a ), where I(Z j b ; Z a ; Y ) represents shared information between two views and I(Z j b ; Y |Z a ) represents unique information for jth view. With sufficient training iterations, the unique information tends towards noise as shared information dominates the accumulated gradient (first term in Eq. ( 6)). This is consistent with the conclusion of MI [41] . Assume that the features obtained from the encoder consist of shared features S which correspond to the anchor and unique features U , represented as Z j b = (S ∪ U j ). Traditional contrastive learning defines an additive model Z j b = (S + U j ) whereas Z j b = (S × U j ) in our model. Intuitively, there existsζ satisfes ζ = Z a • (S × U 1 ) = ... = Z a • (S × U j ).Therefore, UIC is a weaker constraint that ensures quaternion vectors between different views lie on the same plane as much as possible. If we use both SIC and UIC simultaneously, SIC will pull the shared representations of different views closer, while UIC will ensure that the unique representations of different views lie on the same plane as much as possible, rather than measuring their cosine similarity, as unique information is uncorrelated. We conducted extensive experiments to verify this (Section 3.3 for more details).Contrastive learning fundamentally operates by optimizing vector representations to minimize distances between positive pairs while maximizing distances between negative pairs in the embedding space. However, a critical limitation inherent in conventional approaches stems from their undifferentiated treatment of all samples within a batch B as negative examples. This indiscriminate categorization may inadvertently cause the model to overlook potential semantic relationships (as illustrated by the red shading in Figure 3 ), despite their shared semantic content (e.g., different image captions containing identical substrings in image-text retrieval tasks). Such misclassification significantly impairs the model's representation capacity. We refer to these cases as "hard negative samples. While existing methods attempt to address this issue through clustering-based approaches, their two-stage nature limits practical adoption. Leveraging our dual-branch architecture, we propose a more effective solution that directly utilizes the output of the shared decoder as a supervision signal for the penalty term.Considering our objectives is to optimize the shared information decoder through L SIC and unique information decoder through L UIC , and the shared information also affects the optimization of unique information as shown in Eq. ( 4), we attempt to use the intra-model shared information similar to penalization to guide the optimization of unique information. Unlike soft label [49, 17, 16, 47, 60, 24] which aim to mitigate the strict constraints of one-hot labels, preventing overconfidence by retaining more potential positive samples, our method aims to impose stricter constraints to suppress shared information in the process of learning unique information. Specifically, the more shared information between the anchor and all the negative samples, the greater the encouragement in learning unique information between the anchor and the positive sample. Formally, for shared representation Z s i and Z s j , the weighted similarity matrix can be formulated as:EQUATIONwhere S = Z s i Z s j T is the similarity matrix, I is identity matrix, λ ∈ R is learnable weight parameter. Next, the weighted similarity matrix P ∈ R N ×N is utilized as penalization to supervise the optimization of unique information satisfied P ij,i =j ∝ s(z s i , z s j ), the penalized UIC can be defined as:EQUATIONGradient Analysis. For simplicity, we set λ = m k=0 exp(P k • s(Z n i , Z n jk )/τ ) and ignore the second term, we reformulate Eq. ( 8) as:EQUATIONThe gradient can be calculated as (Appendix D.2 for details):EQUATIONwheres + = s(Z n i , Z n + j ) and s k = s(Z n i , Z n - jk ).Intuitively, P represents the belief mass based on shared information, and the larger P k indicates more shared information between hard positive samples, which also influences the optimization of the unique decoder as in Eq. ( 4). When P k = 0 (i.e, s(Z i , Z jk ) ≈ 0), it will degenerate to the original InfoNCE loss. For hard negatives samples hold shared information where s(Z s i , Z s j ) > 0, we increase the gradient using a penalty term which ensures that even if Z s i is relatively large in Eq. ( 4), the lower loss reinforce other terms to constrain the overall value. From mutual information perspectives (Appendix D.3 for details), we haveEQUATIONWhen all negative samples k satisfy s(Z i , Z jk ) ≈ 0, we obtainH P (Z j |Z i ) = H(Z j |Z i ).Subsequently, as the shared information between hard positive samples increases, H P (Z j |Z i ) correspondingly increases, thereby elevating both the lower bound of mutual information and the confidence level, which consequently facilitates the learning of unique information.In summary, we apply (1) SIC (Eq. 2) to keep shared information relevance between different modalities, (2) UIC (Eq. 5) to build quadruple embedding space by maximizing the alignment of normal vector spanned by shared representation and unique representation, and the shared information is jointly optimized by SIC and UIC, (3) Self-penalization (Eq. 7) to amplify the effect of false positive samples in unique information optimization. The overall objective can be formulated as:EQUATION)3 Experiment 
experiment setup: 
 Baselines and Setup. Shortcut learning refers to the process in deep learning model training where the model completes tasks (such as classification, retrieval, etc.) by learning simple and discriminatory features while ignoring the semantic and more complex features of the data. This can result in poor model performance on downstream tasks. Latent target decoding (LTD) [4] ,and implicit feature modification (IFM) [58] are two methods that mitigate shortcut learning by reducing feature suppression. LTD reconstructs the caption representations in the latent space of a Sentence-BERT model, allowing the encoder to mitigate feature suppression via correct mapping. IFM perturbs discriminatory features through encoders and removes part of these features to avoid learning shortcuts, which is implemented as a dual loss combined with the InfoNCE loss. We provide source code of our paper. 2 .Image Caption Retrieval(ICR) retrieves the most relevant sample in another modality by using a sample of one modality as a query. In this task, there are two retrieval modes: text-to-image (t2i) and image-to-text (i2t). We evaluated our method in the ICR task using CLIP [52] and VSE++ [23] models on Flickr30k [80] and MS-COCO [42] datasets.Bleeker et al. [3] proposed synthetic shortcuts for the vision-language framework. This allows us to evaluate whether vision language (VL) models capture easy-to-learn discriminatory features or task-related information. We add MNIST Images to the top of pictures and appending corresponding numbers at the end of their respective captions. This controlled approach preserves the original information from both modalities and increases explicit mutual information between them.Evaluation Metrics. To evaluate the model's performance on the Flickr30k and MS-COCO-Caption datasets, the Recall@K (i.e. R@1, R@5, R@10, which refers to the proportion of instances where the correct answer appears among the top K returned results out of all instances) and recall sum (RSUM) were selected as evaluation metrics for both i2t and t2i retrieval.Implementation Details. We select ViT-B/32 as the visual backbone for CLIP and resnet152 for the VSE++ model when we evaluate them on the caption retrieval task using Flickr30k and MS-COCO dataset. We fine-tuned the pre-trained CLIP on downstream tasks and trained VSE++ from scratch with our method.Alternatives of Unique Decoder. The key of multi-view assumption lies in the extraction of unique information, current methods achieve this by employing a single-layer MLP, ensuring orthogonality with shared information or through data augmentation and factorized loss. In our framework, a single-layer MLP is used for the ResNet backbone, while a two-layer Transformer is implemented for the Transformer backbone. Detailed methodologies are provided in the Appendix B.2.QUEST vs. Vanilla InfoNCE. QUEST outperforms the vanilla InfoNCE, as shown in Table 1 .On the Flickr30k test set, QUEST yields R@1 improvements of (2.4, 1.5) for CLIP and (1.1, 3.4) for VSE++ in i2t and t2i tasks.. The corresponding RSUM metrics increase by 3.2 and 12.1. On MS-COCO, QUEST achieves R@1 gains of (1.6, 0.9) for CLIP and (3.1, 3.2) for VSE++ in i2t and t2i tasks, with corresponding RSUM improvements of 8.1 and 17.3. Additionaly, QUEST exhibits faster convergence to the optimal solution.Experiment on Synthetic Shortcuts. To assess the effectiveness of our proposed QUEST mitigating feature suppression in Contrastive Learning, we use synthetic shortcuts [3] by injecting easy-to-learn and discriminatory shared information into the image-text training dataset. We then evaluate the model's performance on downstream tasks with and without these synthetic shortcuts to determine if the presence of shortcuts causes the suppression of other task-related information and an over-reliance on shortcut features. Our baselines' results are consistent with [3] .As shown in Table 1 , adding shortcuts leads to performance degradation across all models to some degree, indicating that the models have not learned sufficient shared and unique information. However, our method outperforms LTD and IFM, indicating it captures task-related information more effectively in downstream tasks (evaluation without shortcuts).CLIP Performance Enhancement. Our method significantly enhances the CLIP model's performance on the Flickr30k and MS-COCO datasets. Compared to InfoNCE, we observe substantial R@1 improvements in both i2t and t2i tasks, with RSUM increases of 91.6 and 240 on Flickr30k and MS-COCO, respectively. Our approach also outperforms previous SOTA methods, surpassing L InfoNCE+IFM on Flickr30k and showing sinificant performance improvements against L InfoNCE on MS-COCO. Comparative Analysis of VSE++ Improvements. VSE++ also shows improvements on both datasets, with increases in R@1 for i2t and t2i tasks compared to L InfoNCE . On MS-COCO, VSE++ outperforms L InfoNCE+IFM , enhancing R@1 and rsum metrics. However, L InfoNCE+LTD achieves better performance on Flickr30k, excelling in text modality reconstruction and enhancement. This discrepancy may be attributed to VSE++'s GRU text encoder, which appears to be less effective in capturing modality-independent information on smaller datasets like Flickr30k .Flickr30k MS-COCO Method sc i2t t2i RSUM i2t t2i RSUM R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@We also evaluate the effects of two constraints (shared information constraint L SIC and unique information constraint L UIC ) and self-penalization on the performance of our proposed QUEST method. These variants, denoted as L SIC , L UIC , L P-UIC , are evaluated through modality-specific decoder objectives in our ablation studies. The evaluation is conducted on the ICR task, comparing the performance differences between CLIP and VSE++ models on the Flickr30k and MS-COCO datasets.Decoder Configuration and Objective Function Assignment. As shown in Table 2 , both L SIC and L UIC individually exhibit performance drops compared to the full QUEST. CLIP and VSE++ perform poorly when only using unique information constraints, achieving 55.1 and 40.7 on COCO and 80.8 and 47.8 on Flickr30k, respectively, suggesting that capturing unique information alone is insufficient for improvement. Furthermore, the results demonstrate that self-penalization enhances the performance of QUEST, preventing overconfidence and retaining more potential positive samples. CLIP and VSE++ have better performance when using L P-UIC instead of L UIC . In addition, our QUEST method outperforms the approach utilizing L SIC+UIC . The ablation study aligns with our theoretical framework, confirming that simultaneously capturing shared and unique information in multi-view contrastive learning, along with the application of self-penalization, leads to significant improvements in downstream tasks.Unique Decoder Degeneration. Our architecture includes a unique decoder module. To evaluate its impact on the task, we replaced this unique decoder with a shared decoder to determine if performance improvements were due to parameter changes. As shown in Table 2 , using dual shared decoders (L SIC+SIC ) did not enhance performance and even resulted in a decline compared to a single shared decoder (L SIC ). We refer to this decline as the degradation of the unique decoder.We conducted extensive experiments on three modalities: image, text and audio. For image-audio evaluations, we leveraged the FMA [19, 20] and GTZAN [69] datasets, while text-audio experiments were evaluated on CLOTHO [21] and AUDIOCAPS [36] datasets. As shown in Figure 4 , our pro- LSIC LSIC LUIC LP-UIC R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 InfoNCE R@5 InfoNCE R@10 InfoNCE rsum QUEST (ours) R@1 QUEST (ours) R@5 QUEST (ours) R@10 QUEST (ours) rsum posed methods outperformed the baseline. For simplicity, we employed the almost simplest decoder structure (linear layers) and did not implement modality fusion or any cross-modal interactions.We believe that enhancing the architecture, strengthening cross-modal interactions, employing a larger batch size and incorporating pre-training would yield even higher performance.On Flickr30k, L InfoNCE over-privileges shared information, particularly for synthetic shortcut data, while L QUEST emphasizes task-relevant semantic information. In the text-to-image retrieval case, L InfoNCE leverages modality-shared information like "gold" and "bicycle" for retrieval. When shared information is explicitly injected across modalities through shortcuts, L InfoNCE pays less attention to the semantic information in queries.4 Related Work 
multimodal contrastive learning: 
 In the field of multimodal contrastive learning, substantial progress has been achieved in harnessing the synergistic effects of multimodal data [38, 46, 11, 74] . Multimodal contrastive learning mainly concentrates on developing methods to semantically align data from diverse modalities through contrastive learning techniques, including optimizing contrastive objectives to enhance agreement between paired data [52, 33, 79, 15] , improving the selection of positive and negative sample [85, 35, 64] , optimize training mechanism [28, 12] and designing innovative network architectures A dark-haired woman in a green and white apron sits amid her out door display of flowers for sale o n a city street. 9 9 4 5 1 8A woman is sitting by her dried f lower display at an outside mark et. 9 9 4 5 1 8A woman sells flowers and incen se at the market. 9 9 4 5 1 8A lady is sitting down tending to her stand. 9 9 4 5 1 8A woman in an apron shopping a t a market. 9 9 4 5 1 8Top 5In the middle of a shopping cente r, there is a man who is painted g old with a gold bicycle.A man dressed in gold with his g old bike stops to look at the city.A man is wearing a gold outfit w hile standing with his gold bike. A man painted in gold with cycle stands in the street.A man in a gold coat has a gold b icycle in the city. [7, 6, 26, 10]. Nevertheless, our work is orthogonal to most of the aforementioned studies. Our shared branch approach can be integrated with existing training methods such as MoCo [28] and SimCLR [7] .Shortcut Learning refers to the tendency of deep neural networks to exploit simple but potentially unreliable features (i.e., "shortcuts") in data for decision-making, rather than learning more complex but reliable features [25] . This phenomenon can lead to poor model performance on out-of-distribution data and is particularly common in multimodal retrieval [3] and VQA tasks [55, 18] . Robinson et al. [58] proposed strategically adjusting the feature distribution of positive and negative sample pairs to achieve implicit feature modification in contrastive learning, guiding models to learn more robust feature representations. Sanchez et al. [59] propose maximizing mutual information to capture data attributes in shared and exclusive representations, while minimizing it between them to enforce disentanglement. LTD [4] introduces an additional decoder to reconstruct input text descriptions in the latent space of a universal sentence encoder, preventing image and text encoders from suppressing predictive features. More recently, some works strive to enhance the estimation of mutual information through the utilization of stricter bounds [30, 41, 13, 51] or the introduction of regularization constraints [45] , consequently preserving unique information more effectively. 
conclusion: 
 We introduce QUEST, a framework utilizing specialized decoders to extract both unique and shared information via shared information-guided constraints and self-penalization. This study addresses the challenges of imbalanced negative samples and task-related unique feature suppression in Multimodal Contrastive Learning. Our method optimizes shared and unique representations simultaneously, outperforming state-of-the-art methods in preserving unique information and enhancing contrastive learning. Unlike traditional approaches that employ direct dot products to minimize distances between positive samples, QUEST leverages quaternions for the indirect optimization of unique and shared information. However, the application of cross products in high-dimensional spaces is limited, complicating the control of high-dimensional representations and reducing theoretical interpretability.For further discussion on these limitations, see the appendix (Appendix E). Clotho is a diverse audio captioning dataset comprising 4981 audio samples (15-30 seconds each) from Freesound, paired with 24,905 crowdsourced captions (8-20 words each). Designed to facilitate general audio content description using free text, Clotho emphasizes perceptual diversity by providing multiple captions per audio and excluding visual or contextual cues during annotation. The dataset's post-processing, including removal of unique words and speech transcription, enhances its suitability for developing and evaluating audio captioning systems. 
 
 Appendix: AudioCaps is a seminal dataset for audio captioning, comprising 46,000 audio clips from AudioSet with human-authored descriptions. This large-scale corpus has become the benchmark for evaluating audio captioning models, catalyzing advancements in audio representation and multimodal learning. AudioCaps has facilitated the development of innovative architectures such as the Audio Captioning Transformer and retrieval-augmented models, significantly contributing to audio-language research. Its impact extends beyond captioning, influencing broader studies in audio-visual integration and inspiring more comprehensive datasets in the field. For the VSE++ model, we utilized a training regime consisting of 30 epochs with a batch size of 128.The optimization process was driven by the Adam optimizer with a learning rate set at 2 × 10 -4 . Notably, no warmup steps were employed, and the learning rate was adjusted using a stepLR scheduler. This configuration was consistently applied across both the Flickr30k and MS-COCO datasets. The visual encoder for VSE++ was based on ResNet152, while the text encoding was handled by a GRU, resulting in a total parameter count of 67 million.Moreover, the CLIP model was trained over five epochs with a larger batch size of 256. Optimization was performed using the AdamW optimizer with a significantly lower learning rate of 2 × 10 -5 . In contrast to VSE++, CLIP incorporated 100 warmup steps and utilized a cosine annealing scheduler for learning rate adjustment. This setup was also uniformly applied to both datasets. The visual encoder for CLIP was a ViT-B/32, and text encoding was managed by a Transformer, leading to a substantially larger parameter count of 152 million.Unique Decoder Implementation Details We used CLIP(ViT/B-32 backbone) and VSE++(resnet152 backbone); we chose a single-layer MLP for vse++ and two-layer Transformers for CLIP. Both of them use nine layers as a unique start layer and 0 for no unique start layer. We choose the hyperparameters alpha_t as 0.08 on most experiments and set positive_sample to false.The L UIC algorithm enhances contrastive loss in representation learning through a novel embedding space involving quadruplets. This approach primarily comprises two stages: the computation of a similarity map from quadruple embeddings and the subsequent calculation of cross-entropy loss based on this map. The method leverages shared and unique embeddings to construct expressive feature representations, further augmented by three-dimensional vectors generated via cross-products, thereby intensifying the discriminative power of embeddings.As shown in 1, the algorithm firstly processes four embedding vectors:x shared , x unique , y shared , and y unique , representing shared and unique features across two data sets. Padding may be applied to ensure dimensional consistency (Divisible by 3) across embeddings. The GETSIMMAP function, shown in 2 then calculates the similarity map between x shared and y shared , reflecting the similarity between shared feature embeddings. This computation involves normalizing the embeddings, obtaining a preliminary similarity matrix via dot products, and adjusting the similarity values through exponential weighting and diagonal normalization.Upon obtaining the similarity map, the algorithm transforms x shared and x unique (along with their y counterparts) into three-dimensional vectors, or triplet embeddings, via cross products. This geometric transformation aims to further enhance the distinctiveness of embeddings. The triplet embeddings are then normalized to ensure numerical stability during dot product calculations. Ultimately, the algorithm calculates the dot product of x triplet and y triplet , scales the result, and applies element-wise multiplication with the similarity map to produce the final logits. These logits, after scaling and absolute value adjustments, are used to compute the cross-entropy loss.We also use orthogonal cosine embedding loss 3 to quantify the similarity between pairs of input vectors, facilitating the training of models on tasks that distinguish between similar and dissimilar data points. The objective of this loss computation is to minimize the discrepancy between predicted logits and actual labels, thereby optimizing the representational capacity of the embeddings.The synthetic shortcuts experiment enables a quantitative analysis of the model's reliance on shortcuts when they are present and its ability to capture shared and task-relevant unique information. The results demonstrate that our proposed method effectively mitigates the feature suppression phenomenon, contributing to improved performance on downstream tasks compared to previous approaches.To evaluate the generalization capability of our proposed approach on more modalities, we conducted comprehensive experiments across various modalities, including visual, textual, and acoustic domains. Specifically, we performed image-audio retrieval experiments utilizing the FMA (Free Music Archive) and GTZAN datasets, with quantitative results presented in Table 4 . For text-audio retrieval evaluations, we leveraged the CLOTHO and AUDIOCAPS datasets, with comparative performance metrics detailed in Table 5 .The empirical results demonstrate that our proposed models consistently outperformed the baseline approaches across all experimental configurations. For simplicity, we employed the almost simplest Algorithm 1 L UIC loss calculation Require: Latent representation x shared , x unique , y shared , y unique . Ensure: L UIC 1: function CALCULATE L UIC (x shared , x unique , y shared , y unique )B, C ← shape of x shared 3:if padding is enabled in config then x shared ← pad x shared with pad_size 6:x unique ← pad x unique with pad_size participated_dims ← mini_heads × 313:x shared ← reshape x shared [:, : participated_dims] to (B, -x unique ← reshape x unique [:, : participated_dims] to (B, -y shared ← reshape y shared [:, : participated_dims] to (B, -y unique ← reshape y unique [:, : participated_dims] to (B, -x uic ← cross product of x shared and x unique along dimension 2, then reshape to (B, -y uic ← cross product of y shared and y unique along dimension 2, then reshape to (B, -x uic ← normalize x uic along the last dimension L QUAD ← CE loss of logits with labels + CE loss of logits T with labels 27:L cos ← GETCOSLOSS(y shared , y unique ) + GETCOSLOSS(x shared , x unique )28:L UIC ← L QUAD + L cos 29:return L UIC 30: end function Algorithm 2 Calculate similarity map Require: latent representations x, y Ensure: similarity map sim_map 1: function GETSIMMAP(x, y)2:x ← normalize x along the last dimension product_sum ← (x unique , x shared ).sum(dim = 1)4:matnitude_square 1 ← (x unique , x unique ).sum(dim = 1) + 5:matnitude_square 2 ← (x shared , x shared ).sum(dim = 1) + 6: denominator ← √ mag_square1 × mag_square2 7:cos ← product_sum/denominator 8:zeros ← tensor of zeros with same shape as x unique along dimension 0 9:pos_loss ← 1 -cos 10:neg_loss ← clamp(|cos -0.0| , min = 0)11:target ← tensor of -1 with same shape as x unique along dimension 0 12:Initialize loss_pos and loss_neg with the same shape as target, pos_loss, and zeros 13:for each index i in target do14: if target[i] is 1 then 15: loss_pos[i] ← pos_loss[i]16: else 17:loss_pos[i] ← zeros[i] 18:end if19: if target[i] is -1 then 20: loss_neg[i] ← neg_loss[i] 21: else 22: loss_neg[i] ← zeros[i] 23:end ifend for 25:loss ← loss_pos + loss_neg 26:return mean of loss 27: end function Table 4 : Image audio retrieval results on FMA and GTZAN datasets.R@1 R@5 R@10 RSUM R@1 R@5 R@10 RSUM decoder structure (linear layers) and did not implement modality fusion or any cross-modal interactions. We believe that enhancing the architecture, strengthening cross-modal interactions, employing a larger batch size and incorporating pre-training would yield even higher performance. Table 5 : Text audio retrieval results on CLOTHO and AUDIOCAPS datasets.InfoNCER@1 R@5 R@10 RSUM R@1 R@5 R@10 RSUM InfoNCE CLOTHO 20. 16 j M = G M (H j M ; Φ) s(•, •)Scoring functionH(Z j |Z i )The conditional entropy of Z j given Z i H P (Z j |Z i ) Conditional entropy under the penalized term P D Analysis.In many machine learning tasks, it is often observed that different views simultaneously harbour both task-relevant shared information and unique information (e.g., image captioning [38, 40, 72] and referring expression segmentation [78, 44] ). To this end, we work with a dual-encoder architecture.Contrastive learning with multiple views obtains mutual information between different modalities by learning the similarity among different views. [70] et al. introduced a lower bound on mutual information, known as InfoNCE, which is based on the concept of Noise Contrastive Estimation (NCE); it compares the compatibility of different views by maximizing the mutual information of positive pairs and minimizing the mutual information of negative pairs, learning to extract the consistent representation across different modal. It can be defined as follows:EQUATIONIn this equation, x i and y i represent paired samples from the joint distribution of the two random variables under consideration, while y j represents samples from the marginal distribution of one of the variables. The function f (x, y) is a learnable function, often parameterized by a neural network, that aims to distinguish between the paired samples and the independently sampled ones.In the realm of vison-language contrastive learning, the InfoNCE loss function [70] enhances the similarity between positive sample pairs relative to negative ones. Mathematically, the InfoNCE loss is articulated as:EQUATIONwhere sim(x, y) calculates the similarity between samples x and y, typically computed through dot product or cosine similarity. y + denotes the positive sample corresponding to x, and {y i } K i=0 is a set including one positive and K negative samples. The parameter τ serves as a temperature coefficient, modulating the scale of similarity scores.InfoNCE loss prevails in many contrastive learning algorithms and performs well in applications. Multi-view redundancy [66, 2, 30, 63, 65, 67] assumes that there exists duplicated information across varied views or representations. Contrastive losses like InfoNCE loss tend to maximize the mutual features between different views while suppressing task-relevant features that may be used in downstream tasks in multi-view representation learning.Mutual information (MI), often denoted as I(X; Y ), is a fundamental concept in information theory that quantifies the statistical dependence between two random variables, X and Y . It measures the reduction in uncertainty about one variable when the value of the other is known and is defined as:EQUATIONwhere p(x, y) is the joint distribution, and p(x) and p(y) are the marginal distributions of X and Y .In machine learning, particularly in deep learning, MI is often used as an objective function or regularization component to promote or constrain the interdependence among variables. However, the precise quantification of MI is only attainable in limited instances, as it requires the closed form of the density function and the tractable logarithm density ratio between the joint and marginal distributions. In most machine learning applications, practitioners only have access to samples from the joint distribution, making the direct computation of MI infeasible.The InfoNCE bound has several desirable properties that make it an attractive choice for estimating mutual information. Firstly, it is a lower bound on the true mutual information, I(X; Y ) ≥ I NCE , which means that maximizing the InfoNCE bound leads to an increase in the true mutual information.Secondly, the bound is computationally tractable and can be efficiently optimized using standard gradient-based methods, such as stochastic gradient descent.As shown in Fig 2 , multimodal mutual information can be divided into shared information I(X A ; X B ; Y ) and task-relevant unique information (I(x A ; Y |X B ) and I(X B ; Y |X A )). Information useful for the task can be represented asEQUATIONFor the simplified loss function Eq. ( 9), we first calculate the partial derivative of Z s i :EQUATION)wheres + = s(Z n i , Z n + j ). Define s k = s(Z n i , Z n - jk), then we have:EQUATIONwhere λ = m k=0 exp(P • s(Z n i , Z n jk )/τ ). For energy function s(Z i , Z j ) = Ẑi • Ẑj = Z i • Z j /( Z i Z j ), the derived gradient on the vector Z i is shown as:EQUATIONLet Z i and Z i represent the compact vector representation from different modalities input; the classical mutual information between Z i and Z j can be defined as:EQUATIONAccording to [70] , the density ratio p(Zi,Zj ) p(Zi)p(Zj ) is expressed as exp(s(Z i , Z j )/τ ), where s(Z i , Z j ) is the similarity score and τ is the temperature parameter. The penalty term P is defined as follows:EQUATIONThe penalty term does not affect the numerator but amplifies the denominator in Eq. ( 8) thus the energy function in Eq. ( 8) can be reformulated as:exp(s(Z i , Z j )/τ ) • 1 P = exp(s(Z i , Z j )/τ -log P) ∝ p(Z j |Z i ) p(Z j )where P ∝ PNext, we derive the lower bound of MI,EQUATIONDimension restrictions. The limitations of the cross-product in the context of QupleInfoNCE's unique information extraction are theoretically constrained. When contrastive learning is applied to both unique and shared representations, it can lead to model degradation, and a trade-off choice is the cross-product, which is a weaker constraint. In low-dimensional space, it is straightforward to prove the properties of orthogonal vector space, yet the extension of the cross-product to high-dimensional space is challenging. The behaviour of high-dimensional space representation is difficult to control, resulting in our theory lacking sufficient interpretability in high-dimensional space. This highlights the importance of developing more sophisticated methods for high-dimensional data analysis in the future.The Extraction of Unique Information. On the synthetic shortcuts dataset, our framework achieved significant performance, validating the positive role of unique information in preventing model shortcuts. However, within the entire framework, the core lies in identifying unique information, which is essentially a task-dependent definition. Our experiment indicates that pre-trained models (such as CLIP [52] ) typically yield higher unique information benefits due to their extensive generic representations. Non-pre-trained models (such as VSE++ [23] ) struggle to discern unique information, tending to shortcuts and losing vital unique information. In this study, we introduced an additional unique information decoder to capture unique information, which incorporated an extra gradient branch, suggesting exploration space remains for single-stream unique information extraction. This also proposes future research: how to more effectively extract unique information from a single stream to reduce computational complexity while maintaining model performance.Answer Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.https://github.com/Vortexsong/QUEST 


Decision: This paper introduces the Quadruple Multimodal Contrastive Learning Framework (QUEST), which is designed to extract both shared and unique task-relevant information across multiple modalities during contrastive learning. The framework is novel in its use of quaternion vector spaces to simultaneously optimize shared and unique information, thereby addressing the issue of "shortcut learning" that often arises in multimodal tasks. The method is empirically validated on several multimodal datasets, including popular public benchmarks such as Flickr30k and MSCOCO, where it demonstrates significant improvements in performance.

Given the significant strengths of the work, the thoughtful and comprehensive rebuttal, and the consensus among reviewers for acceptance, I recommend that this paper be accepted as a poster (as some reviewers yet noted a lack of rigorous theoretical analysis, particularly regarding the disentanglement of shared and unique information). The authors are encouraged to incorporate the suggestions for further theoretical exploration and to refine the presentation in the camera-ready version.

Reviews: review: paper_summary: paper_summary: A new Multimodal Contrastive Learning method named QUEST is proposed to deal with the fine-grained alignment problem between different modal. Both quaternion contrastive objectives and orthogonal constraints are proposed to extract sufficient unique information. The quaternion vector spaces are designed to simultaneously optimize shared and unique information. Experiments conduct the superior performance in multimodal contrastive learning benchmarks.
strength_weakness: strength_weakness: 1) There are some typo, for example: line 218 MS-COCO-Cpation
2) The position of Z_i^s and Z_i^u of Modal M_i should be exchanged in Figure3. Also, Z_i_s and Z_i^u of Modal M_j should be Z_j^s and Z_j^u in Figure3.
3) Please check the position of + symbol in Eqn5.
4) Why the value of Eqn4 should be optimized to maximum value? Can you explain with an example in multimodal contrastive learning?
questions: questions: Why the value of Eqn4 should be optimized to maximum value? Can you explain with an example in multimodal contrastive learning?
limitations: limitations: The experiments conducted in text and vision modal. Experiments on more modal data should be conducted to show the generalization ability.
paper_summary: paper_summary: The paper focuses on developing a new multi-modal representation learning approach where the extraction and integration of both shared and unique information across multimodal data is the focus. The method aims to pull shared representations closer while aligning the unique representations with the shared representation on a common plane.

Key components: an encoder, a shared decoder, and a unique decoder. Contrastive loss constrains learning of shared information. The proposed framework seeks to mitigate shortcut learning.
strength_weakness: strength_weakness: Lack of theoretical analysis.

Is there any theoretical justification/demonstration or proof that the encoder information can be disentangled to shared and unique representations for a given distribution w.r.t a task or in a task-agnostic manner? I would like to hear author perspective.
questions: questions: What does it mean by shortcut learning in this case? There can be large variations in learned shortcuts.

Better to discuss more recent multimodal shortcut learning papers such as Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion, ICML 2024.
limitations: limitations: Perhaps, the proposed method may be too complicated for some problems.
paper_summary: paper_summary: This paper proposes the Quadruple Multimodal Contrastive Learning Framework (QUEST) to capture shared and unique task-relevant information in the process of contrastive learning, enabling models to capture more unique information in downstream tasks to achieve better performance. Specifically, this paper introduces a quadruple embedding space and optimizes shared and unique information within it simultaneously. Additionally, this paper adopts a self-penalization mechanism using shared information to guide the optimization of unique information. This paper evaluates the QUEST method on popular datasets (Flickr30k and COCO). On public benchmarks and synthetic datasets, the method shows significant performance improvements.
strength_weakness: strength_weakness: 1. In the shared Decoder of Equation 1 in Section 2.3, the parameters on both sides of the shared decoder equation are inconsistent: the left side has $\Phi_i$, while the right side has $\Psi_i$. This is strange. Can you tell me why it needs to be written like this if necessary?

2. In the 3.3 ablation study section, what does QUEST-SIC and QUEST-UIC mean? There was no explanation before. Do QUEST-SIC and QUEST-UIC correspond to $\mathcal{L}_{\text{SIC}}$ and $\mathcal{L}_{\text{UIC}} in Table 2 respectively?

3. What does $Z_j^n+$ in equation 5 mean? Can you explain why the "+" sign is placed to the right of the symbol "Z_j^n"?
questions: questions: How can features be decomposed into shared features and unique features? What is the effectiveness of this method in practice?
limitations: limitations: The behavior of the cross product operation in low dimensions may be unpredictable in higher dimensions.

Comments: comment: Dear Reviewer zJ9C,

Thank you very much for your valuable feedback. We will address the corresponding issues in the subsequent versions.
Dear Reviewer 9n5E,

We sincerely appreciate your recognition of our work. Thank you for your valuable feedback. We will incorporate more theoretical justification and a more comprehensive discussion of related work in subsequent versions of our paper.
Dear Reviewer KzFg,

We extend our sincere gratitude for your thorough review of our work. We humbly accept the issues you've raised and  we will address them  in future iterations of our manuscript.
Your insightful feedback is invaluable to us, and we are dedicated  to improving our work based on your suggestions.
Thank you very much for the author's response. Most of my doubts have been resolved, and I have raised my score.
Thanks for the responses. If possible, please include the discussion around theoretical justification in the paper and discuss related work. I have raised my rating to 7.
Thanks for the authors' responses, which are clear and comprehensive. I recommend the authors to modify the Eq.5 and add the explanation of w2 to the camera ready version.